A delay-aware medium access control method for an underwater acoustic network based on reinforcement learning and related equipment

Abstract

The application provides a delay-aware underwater acoustic network medium access control method based on reinforcement learning and related equipment, and belongs to the technical field of ocean information communication and underwater acoustic network protocol. The method comprises the following steps: dividing each transmission frame into multiple sending time slots according to the time slot length, obtaining the action space of each underwater sending node; obtaining the initial Q value corresponding to each sending time slot; reading the local Q table and selecting a sending time slot according to the Q value to obtain the selected sending time slot; waiting for an ACK confirmation packet from the sink node before the ACK waiting timeout time arrives to obtain the ACK receiving state; giving a positive reward or a negative punishment to the selected sending time slot; updating the Q value of the corresponding sending time slot in the local Q table; if the transmission fails, entering the backoff state, and through multiple rounds of iteration, different underwater sending nodes gradually select different sending time slots to obtain a distributed time slot self-organization result. The application can improve the data transmission stability of the underwater acoustic network.

Description

Technical Field

[0001] This application relates to the field of marine information communication and underwater acoustic network protocol technology, and in particular to a time-delay-aware underwater acoustic network media access control method and related equipment based on reinforcement learning. Background Technology

[0002] In related technologies, the traditional ALOHA protocol is prone to problems such as high collision rate, many retransmissions and low channel utilization in underwater acoustic long propagation delay scenarios.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a reinforcement learning-based delay-aware underwater acoustic network media access control method and related equipment, which can improve the data transmission stability of the underwater acoustic network.

[0005] To achieve the above objectives, one aspect of this application proposes a delay-aware underwater acoustic network media access control method based on reinforcement learning, the method comprising the following steps: The maximum one-way propagation delay is calculated based on the underwater acoustic propagation speed in the underwater acoustic channel and the maximum communication distance between the underwater transmitting node and the sinking node. The time slot length and ACK waiting timeout are then set based on the maximum one-way propagation delay. The continuous communication time is divided into several transmission frames, and each transmission frame is divided into multiple transmission time slots according to the time slot length, so as to obtain the action space of each underwater transmission node; the action space corresponds to all the selectable transmission time slots within a transmission frame; Each of the underwater transmitting nodes independently maintains a local Q table to obtain the initial Q value corresponding to each transmitting time slot; the Q value is used to represent the historical transmission effect of the transmitting time slot. When the underwater transmitting node has data to be transmitted, it reads the local Q table and selects a transmission time slot according to the Q value to obtain the selected transmission time slot. Send data packets within the selected transmission time slot and start an ACK waiting timer. Wait for an ACK confirmation packet from the aggregation node before the ACK waiting timeout period expires to obtain the ACK reception status. Based on the ACK reception status, determine whether the transmission was successful or failed, obtain the result of the success or failure of the transmission, and give a positive reward or a negative penalty to the selected transmission time slot; The Q value corresponding to the transmission slot in the local Q table is updated according to the positive reward or negative penalty to obtain the updated local Q table; the updated local Q table is used for the next transmission slot selection; If transmission fails, the system enters a backoff state. The backoff time is calculated based on the time slot length. After the backoff ends, the system returns to the step of obtaining the selected transmission time slot and uses the updated local Q table as the read object. Through multiple iterations, different underwater transmission nodes gradually select different transmission time slots to obtain a distributed time slot self-organization result.

[0006] In some embodiments, calculating the maximum one-way propagation delay based on the underwater acoustic propagation speed in the underwater acoustic channel and the maximum communication distance between the underwater transmitting node and the sinking node, and setting the time slot length and ACK waiting timeout based on the maximum one-way propagation delay, includes: The underwater acoustic propagation speed in the underwater acoustic channel and the maximum communication distance between the underwater transmitting node and the sinking node are obtained. The underwater transmitting node includes an underwater sensor node, an underwater communication node, or an underwater unmanned platform equipped with underwater acoustic communication equipment, which is used to collect underwater environmental data and send data packets to the sinking node. The sinking node includes an underwater gateway, a surface buoy node, or a data collection node, which is used to receive the data packets uploaded by the underwater transmitting node and return an ACK confirmation packet after correct reception. Divide the maximum communication distance by the underwater acoustic propagation speed to obtain the maximum one-way propagation delay; Set the time slot length according to the maximum one-way propagation delay; Set the ACK waiting timeout period according to the maximum one-way propagation delay, so that the ACK waiting timeout period is equal to the sum of the data packet sending time, the ACK confirmation packet sending time, twice the maximum one-way propagation delay, and the guard interval.

[0007] In some embodiments, dividing the continuous communication time into several transmission frames, and dividing each transmission frame into multiple transmission time slots according to the time slot length to obtain the action space of each underwater transmitting node includes: The continuous communication time between the underwater transmitting nodes is divided into multiple transmission frames of equal length, and each transmission frame is a transmission cycle. Based on the time slot length, each transmission frame is uniformly divided into multiple equal-length transmission time slots in the time domain; the length of each transmission time slot is equal to the time slot length, and adjacent transmission time slots are separated by time slot boundaries; The underwater transmitting node's action space is formed by dividing all the transmission time slots within each transmission frame; each transmission time slot in the action space corresponds to an optional action in the reinforcement learning process; when the underwater transmitting node needs to transmit data, it selects a transmission time slot from the action space as the action for this data transmission.

[0008] In some embodiments, the step of maintaining a local Q table independently for each of the underwater transmitting nodes to obtain the initial Q value corresponding to each transmitting time slot includes: Each of the underwater transmitting nodes independently establishes and maintains a local Q-table in its local memory; the local Q-table adopts a one-dimensional table structure; each entry in the local Q-table corresponds to a transmitting time slot, and each entry is used to store the Q value corresponding to the transmitting time slot; During the protocol initialization phase, the Q values ​​of all entries in the local Q table are set to the same initial value so that each transmission time slot has the same probability of being selected in the initial state. The Q value is used to represent the historical transmission effect of the corresponding transmission time slot. The higher the Q value, the higher the historical transmission success rate of the transmission time slot. The lower the Q value, the higher the historical transmission failure rate of the transmission time slot. The local Q table is stored locally on the underwater transmission node.

[0009] In some embodiments, when the underwater transmitting node has data to be transmitted, reading the local Q table and selecting a transmission time slot based on the Q value to obtain the selected transmission time slot includes: When there are data packets to be sent in the data buffer of the underwater transmitting node, the underwater transmitting node enters the transmission time slot selection process, and the underwater transmitting node reads the Q value corresponding to all the transmission time slots stored in the local Q table; Based on the Q values ​​of each of the transmitted time slots, a selection strategy is adopted to select one of the transmitted time slots from the action space as the selected transmitted time slot; the selection strategy is either a maximum Q value strategy or a Softmax probability selection strategy; the maximum Q value strategy selects the transmitted time slot with the largest current Q value as the selected transmitted time slot; the Softmax probability selection strategy calculates the probability of each transmitted time slot being selected based on the Q value of each transmitted time slot, and the higher the Q value, the greater the probability of the transmitted time slot being selected.

[0010] In some embodiments, the step of sending data packets within the selected transmission slot and starting an ACK waiting timer, waiting for an ACK acknowledgment packet from the aggregation node before the ACK waiting timeout period expires, and obtaining the ACK reception status includes: During the selected transmission time slot, the underwater transmitting node transmits data packets through the underwater acoustic channel; the data packets include a transmitting node identifier, a receiving node identifier, a data packet sequence number, a data payload, and verification information; After the data packet is sent, the underwater transmitting node starts an ACK waiting timer and sets the timeout of the ACK waiting timer to the ACK waiting timeout period. The underwater transmitting node continuously listens to the underwater acoustic channel before the ACK waiting timeout period expires to wait for an ACK confirmation packet from the aggregation node; the ACK confirmation packet includes the target transmitting node identifier and the sequence number of the confirmed data packet; If the ACK confirmation packet is received before the ACK waiting timeout period expires, and the target sending node identifier in the ACK confirmation packet is consistent with the sending node identifier in the data packet, and the sequence number of the confirmed data packet is consistent with the sequence number of the data packet, then the ACK reception status is obtained; If a valid ACK acknowledgment packet is not received by the time the ACK waiting timeout period expires, then the ACK reception status is ACK timeout.

[0011] In some embodiments, determining whether the transmission was successful or failed based on the ACK reception status, obtaining a success or failure result, and applying a positive reward or negative penalty to the selected transmission slot includes: The underwater transmitting node is used to obtain the ACK reception status. If the ACK reception status is "ACK received", the transmission is determined to be successful, and a successful transmission result is obtained. A positive reward is given to the selected transmission time slot. The positive reward is a positive value. The positive reward is used to increase the Q value of the selected transmission time slot. If the ACK reception status is ACK timeout, the transmission is determined to have failed, a transmission failure judgment result is obtained, and a negative penalty is applied to the selected transmission time slot; the negative penalty is a negative value or less than the positive reward value; the negative penalty is used to reduce the Q value of the selected transmission time slot.

[0012] In some embodiments, updating the Q value in the local Q table corresponding to the transmission slot according to the positive reward or negative penalty to obtain the updated local Q table includes: The underwater transmitting node is used to obtain the positive reward or negative penalty, and to obtain the selected transmitting time slot; The Q-value update formula in the Q-learning algorithm is used to update the entries in the local Q-table corresponding to the selected transmission slot, resulting in an updated local Q-table. The Q-value update formula is: the new Q-value equals the difference between one and the learning rate multiplied by the old Q-value, plus the learning rate multiplied by the reward value obtained in this transmission. The old Q-value is the Q-value corresponding to the selected transmission slot before the update; the new Q-value is the Q-value corresponding to the selected transmission slot after the update. The learning rate is a preset value between 0 and 1, used to control the weight ratio of the old Q-value and the reward value in the update. When the reward value is a positive reward, the new Q-value is greater than the old Q-value; when the reward value is a negative penalty, the new Q-value is less than the old Q-value.

[0013] In some embodiments, if transmission fails, a backoff state is entered, the backoff time is calculated based on the time slot length, and after the backoff ends, the step of obtaining the selected transmission time slot is returned, and the updated local Q table is used as the read object. Through multiple rounds of iteration, different underwater transmitting nodes gradually select different transmission time slots to obtain a distributed time slot self-organization result, including: When the judgment result indicates transmission failure, the underwater transmitting node enters a backoff state. In the backoff state, the underwater transmitting node calculates the backoff time based on the time slot length; the backoff time is equal to a random integer multiplied by the time slot length; the random integer is selected between zero and the backoff window size; The underwater transmitting node suspends transmission during the backoff time and exits the backoff state after the backoff time ends. The underwater transmitting node returns to the step of obtaining the selected transmission time slot, and uses the updated local Q table as the reading object to reselect the transmission time slot for data transmission. Through multiple rounds of feedback on successful and failed transmissions, each underwater transmitting node independently updates its own local Q table during multiple iterations, so that the Q value of successful time slots increases and the Q value of conflicting time slots decreases, thus obtaining a distributed time slot self-organization result.

[0014] To achieve the above objectives, another aspect of this application proposes a time-delay-aware underwater acoustic network media access control system based on reinforcement learning, for implementing the method described above, wherein the system includes multiple underwater transmitting nodes and at least one aggregation node. Each of the underwater transmitting nodes includes: The time slot division unit is used to calculate the maximum one-way propagation delay based on the underwater acoustic propagation speed in the underwater acoustic channel and the maximum communication distance between the underwater transmitting node and the sinking node, and to set the time slot length and ACK waiting timeout based on the maximum one-way propagation delay; and to divide the continuous communication time into several transmission frames, and to divide each transmission frame into multiple transmission time slots according to the time slot length, so as to obtain the optional action space of each underwater transmitting node; The Q-table storage unit is used to independently maintain a local Q-table and store the Q-values ​​corresponding to each transmission time slot; the Q-values ​​are used to represent the historical transmission effect of the transmission time slot. The time slot selection unit is used to read the local Q table and select a transmission time slot according to the Q value when there is data to be transmitted, so as to obtain the selected transmission time slot. The underwater acoustic transmission unit is used to transmit data packets within the selected transmission time slot; The ACK receiving unit is used to wait for an ACK acknowledgment packet from the aggregation node before the ACK waiting timeout period expires, and to obtain the ACK receiving status. The timeout determination unit is used to determine whether the transmission was successful or failed based on the ACK reception status, and to obtain the determination result of whether the transmission was successful or failed. The Q-value update unit is used to update the Q-value corresponding to the transmission time slot in the local Q-table according to the positive reward or negative penalty, so as to obtain the updated local Q-table; The backoff and retransmission unit is used to enter the backoff state when the transmission fails. The backoff time is calculated based on the time slot length. After the backoff ends, the time slot selection unit is retried. The aggregation node includes: A data receiving unit is used to receive the data packets sent by the underwater transmitting node; A data decoding unit is used to determine whether the data packet has been correctly received; An ACK feedback unit is used to return an ACK confirmation packet to the corresponding underwater transmitting node after the data packet has been correctly received.

[0015] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0016] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0017] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0018] The embodiments of this application include at least the following beneficial effects: This application provides a method, system, electronic device, storage medium, and program product for media access control of a time-delay-aware underwater acoustic network based on reinforcement learning. The method includes: dividing each transmission frame into multiple transmission time slots according to the time slot length to obtain the action space of each underwater transmitting node; obtaining the initial Q value corresponding to each transmission time slot; reading the local Q table and selecting a transmission time slot according to the Q value to obtain the selected transmission time slot; waiting for an ACK confirmation packet from the sink node before the ACK waiting timeout period arrives to obtain the ACK reception status; giving positive rewards or negative penalties to the selected transmission time slot; updating the Q value of the corresponding transmission time slot in the local Q table; and entering a backoff state if transmission fails, and through multiple rounds of iteration, gradually enabling different underwater transmitting nodes to select different transmission time slots to obtain a distributed time slot self-organization result. This application can improve the data transmission stability of underwater acoustic networks. Attached Figure Description

[0019] Figure 1 This is a flowchart of the delay-aware underwater acoustic network media access control method based on reinforcement learning provided in the embodiments of this application; Figure 2 This is a schematic diagram of a hidden terminal and an exposed terminal provided in the embodiments of this application; Figure 3(a) is a schematic diagram of a three-dimensional underwater random topology in an applicable network provided in the embodiments of this application; Figure 3(b) is a schematic diagram of a regular ring topology in an applicable network provided in the embodiments of this application; Figure 4 This is a time slot partitioning diagram provided in an embodiment of this application; Figure 5(a) is a first schematic diagram of the allocation results provided in an embodiment of this application; Figure 5(b) is a second schematic diagram of the allocation results provided in the embodiments of this application; Figure 5(c) is a third schematic diagram of the allocation results provided in the embodiments of this application; Figure 5(d) is a fourth schematic diagram of the allocation results provided in the embodiments of this application; Figure 6(a) is a schematic diagram comparing the PDR of ALOHA, DA-ALOHA-Q and adaptive backoff ALOHA under different topologies provided in the embodiments of this application; Figure 6(b) is a schematic diagram comparing the throughput of ALOHA, DA-ALOHA-Q and adaptive backoff ALOHA under different topologies provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0023] 1) Underwater acoustic network: refers to an underwater communication network that uses sound waves as the main information transmission medium. It is usually composed of multiple underwater sensing nodes, communication nodes or aggregation nodes, and can be used in scenarios such as marine environmental monitoring, underwater resource exploration, underwater target detection, and underwater data acquisition.

[0024] 2) Underwater acoustic channel: refers to the communication medium and transmission environment through which underwater acoustic signals propagate. Underwater acoustic channels are characterized by slow propagation speed, large propagation delay, small available bandwidth, strong environmental noise, and significant multipath effects, making them important factors affecting underwater communication performance.

[0025] 3) Media Access Control Protocol (MAC): This is a protocol in the network protocol stack used to control how multiple nodes share the same communication channel. Its main function is to coordinate the timing of data transmission by nodes, reducing packet collisions caused by multiple nodes transmitting simultaneously.

[0026] 4) Underwater Acoustic Network MAC Protocol: This refers to a media access control protocol specifically designed for underwater acoustic communication environments. Due to the characteristics of underwater acoustic communication, such as long propagation delay and low bandwidth, traditional wireless network MAC protocols are difficult to directly apply to underwater environments. Therefore, improvements are needed to adapt to the characteristics of underwater acoustic channels.

[0027] 5) Propagation delay: refers to the time required for a signal to travel from the transmitting node to the receiving node. In underwater environments, the speed of sound propagation is approximately 1500 m / s, which is much lower than the speed of radio signal propagation on land. Therefore, the propagation delay in underwater acoustic networks is usually quite large.

[0028] 6) Long propagation delay: Due to the slow speed of underwater sound waves, even short distances between nodes will experience significant signal propagation delays. Long propagation delays cause nodes to lag in assessing channel conditions, thereby increasing the probability of hidden terminals, exposed terminals, and data packet collisions.

[0029] 7) Time slot: refers to a number of fixed time slices divided into a transmission cycle. Sending nodes can transmit data within selected time slots to reduce collisions caused by completely random transmission. In this invention, time slots are also optional actions for nodes when making reinforcement learning decisions.

[0030] 8) Frame: Refers to a complete transmission cycle consisting of multiple time slots. Each frame contains several data transmission time slots that nodes can choose from. Each node selects one of the time slots to send data based on its own learning results.

[0031] 9) Packet collision: This refers to a situation where data packets sent by two or more nodes overlap in time at the receiving end or during transmission, causing the receiving node to be unable to decode the data packets correctly. Packet collisions can lead to retransmissions, reduced throughput, and wasted energy.

[0032] 10) ACK Acknowledgment Packet: ACK is short for Acknowledgment, indicating confirmation. When a receiving node successfully receives a data packet from a sending node, it returns an ACK acknowledgment packet to the sending node to notify it that the data transmission was successful.

[0033] 11) Timeout event: This refers to a situation where the sending node does not receive an ACK confirmation packet from the receiving node within a preset waiting time after sending a data packet, and considers the transmission to have failed. A timeout event usually indicates that a data packet collision, loss, or ACK return failure.

[0034] 12) Reinforcement learning: This refers to a machine learning method that learns optimal behavioral strategies through interaction between an agent and its environment. The agent selects actions based on its current state and receives rewards or penalties based on environmental feedback, thereby continuously adjusting its subsequent action selection strategy.

[0035] 13) Q-Learning: A typical reinforcement learning algorithm. This algorithm evaluates the long-term benefits of different actions by maintaining Q-values ​​and continuously updates the Q-values ​​based on reward feedback, enabling the agent to gradually learn better action selection strategies.

[0036] 14) Q-table: In the Q-Learning algorithm, this refers to the data table used to store the values ​​of different actions. In this invention, each underwater transmitting node independently maintains a local Q-table, and each element in the Q-table corresponds to the value of an optional transmission slot.

[0037] 15) Q-value: Refers to the value of the actions recorded in the Q table. In this invention, the Q-value is used to represent the transmission effect of a certain transmission time slot for the current node. The higher the Q-value, the better the historical transmission success effect of that time slot, and the greater the probability of it being selected again in the future.

[0038] 16) Action Space: Refers to the set of all actions that an agent can choose during reinforcement learning. In this invention, the action space corresponds to all available transmission slots within a frame; a node selecting a slot indicates that it has selected an action.

[0039] 17) Reward function: refers to the feedback mechanism used in reinforcement learning to evaluate the quality of actions. In this invention, a node is given a positive reward when it receives an ACK, increasing the Q value of the current time slot; a node is given a negative penalty when it times out, decreasing the Q value of the current time slot.

[0040] 18) Delay Awareness: This refers to the consideration of the propagation delay characteristics of the underwater acoustic channel during the design and operation of the protocol. In this invention, delay awareness is mainly reflected in the consideration of underwater acoustic propagation delay in the time slot length setting, ACK waiting time setting, and timeout judgment process.

[0041] 19) Distributed access: This refers to a system where each transmitting node makes its own transmission decisions based on information it maintains, without relying on a central controller or unified scheduling node. In this invention, each underwater node autonomously selects its transmission time slot using its local Q-table and transmission feedback.

[0042] 20) Adaptive learning: This refers to a node's ability to dynamically adjust its transmission strategy based on actual feedback during network operation. In this invention, the node updates its Q table based on ACK success feedback and timeout failure feedback, thereby gradually avoiding time slots prone to collisions.

[0043] 21) Backoff mechanism: This refers to a mechanism where, after a transmission failure, a node does not immediately retransmit but waits for a random or predetermined period of time before attempting to retransmit. The backoff mechanism can reduce continuous collisions caused by multiple nodes repeatedly competing for the channel at the same time.

[0044] 22) DA-ALOHA-Q: This is an abbreviation for Delay-Aware ALOHA with Q-Learning, which can be expressed in Chinese as "Delay-Aware ALOHA Protocol Based on Q-Learning". In this invention, this term is used to represent a medium access control method for underwater acoustic networks that combines underwater acoustic propagation delay awareness, time slot allocation, ACK / timeout feedback, and Q-Learning learning mechanisms.

[0045] 23) Packet delivery rate: refers to the ratio between the number of packets successfully received by the receiving node and the number of packets sent by the sending node, and is used to measure the reliability of network transmission.

[0046] 24) Throughput: refers to the amount of effective data successfully received by a receiving node per unit time, used to measure the effective utilization of network channels.

[0047] 25) NS-3: A discrete event network simulation platform that can be used to build and verify network protocol models. In this invention, NS-3 can be used to perform simulation experiments to verify the proposed underwater acoustic network MAC protocol.

[0048] 26) Aqua-Sim-NG: An underwater acoustic network simulation extension module in the NS-3 platform, which can be used to simulate underwater acoustic propagation, underwater node communication, and underwater acoustic network protocol behavior. In this invention, this module can be used to verify the feasibility and performance of the proposed protocol in an underwater acoustic network scenario.

[0049] 27) ALOHA: A random access media access control protocol.

[0050] 28) TDMA: Time Division Multiple Access is a channel access protocol based on time division.

[0051] In related technologies, the traditional ALOHA protocol adopts a random access method of "send as soon as data is available." While its protocol structure is simple and deployment is convenient, its transmission decisions are highly random. Nodes do not check the transmission status of other nodes before sending data, nor do they adjust their subsequent transmission strategies based on historical transmission results. In underwater acoustic networks, the propagation speed of sound waves is much lower than that of electromagnetic waves, resulting in significant propagation delays between nodes. When a node begins sending data, other nodes may not be able to detect this transmission behavior in time, thus continuing to send data at the wrong time, causing multiple data packets to collide at the receiving end. Therefore, the traditional ALOHA protocol is prone to high collision rates, numerous retransmissions, and low channel utilization in underwater acoustic long propagation delay scenarios. The main drawback of this type of protocol is that nodes only perform random sending and random backoff, lacking the ability to learn collision patterns and proactively avoid transmission moments prone to collisions.

[0052] In view of this, this application provides a method and related equipment for delay-aware underwater acoustic network media access control based on reinforcement learning. This application mainly solves the following technical problems: 1. Solve the problem of high collision rate of traditional random access protocols in underwater acoustic long propagation delay environments.

[0053] 2. Solve the problem that fixed time slot protocols cannot adaptively adjust the transmission time slot based on the transmission results.

[0054] 3. Solve the problem of high control packet overhead and long waiting time in the RTS / CTS handshake protocol.

[0055] 4. Solve the problems of TDMA-type protocols relying on centralized scheduling and strict synchronization, and lacking dynamic adaptability.

[0056] 5. Address the issues of complex state space and high computational and communication overhead in existing intelligent MAC protocols.

[0057] 6. Solve the problem that underwater acoustic network nodes have difficulty autonomously forming low-collision access strategies when global information is lacking.

[0058] The purpose of this invention is to provide a delay-aware underwater acoustic network media access control method and system based on delay-aware reinforcement learning (a delay-aware underwater acoustic network media access control method and related equipment based on reinforcement learning), enabling underwater nodes to autonomously adjust their transmission time slot selection strategy based on their own data transmission feedback without the need for central scheduling, extensive control packet handshakes, or acquisition of global network status, thereby reducing the data packet collision rate in the underwater acoustic network and improving channel utilization and data transmission stability.

[0059] Specifically, this invention models the transmission slot selection process in an underwater acoustic network as a Q-Learning reinforcement learning process, enabling each underwater transmitting node to independently maintain its local Q-table. Each node uses multiple selectable slots within a transmission cycle as its action space, updating the selected slots with rewards or penalties based on successful ACK feedback and timeout failure feedback. Through multiple rounds of data transmission and feedback learning, the node can gradually increase the probability of selecting successful slots and decrease the probability of selecting conflicting slots, thereby achieving distributed adaptive channel access.

[0060] Compared to the traditional ALOHA protocol, the purpose of this invention is not to allow nodes to blindly send data randomly, but to enable nodes to learn optimal transmission time slots based on historical transmission results; compared to fixed-time-slot ALOHA, the purpose of this invention is to enable nodes to dynamically adjust their capabilities; compared to the RTS / CTS handshake mechanism, the purpose of this invention is to reduce control packet interaction and long latency waiting overhead; compared to the TDMA scheduling protocol, the purpose of this invention is to avoid dependence on centralized scheduling and strict synchronization; compared to complex intelligent MAC protocols, the purpose of this invention is to adopt a lightweight Q-table learning method to reduce the computational and storage burden on underwater nodes.

[0061] Therefore, this invention aims to develop a low-overhead, distributed, and adaptively learning-based MAC access method for underwater acoustic networks, enabling underwater acoustic nodes to gradually avoid conflicting time slots in environments with long propagation delays and multi-node contention, thereby improving the data transmission reliability and protocol engineering deployment feasibility of the underwater acoustic network. This application focuses on solving the problems of packet collisions under conditions of long propagation delays in underwater acoustic channels, concurrent node contention, and topology uncertainty.

[0062] Figure 1 This is an optional flowchart of the reinforcement learning-based delay-aware underwater acoustic network media access control method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S108.

[0063] Step S101: Calculate the maximum one-way propagation delay based on the underwater acoustic propagation speed in the underwater acoustic channel and the maximum communication distance between the underwater transmitting node and the sinking node, and set the time slot length and ACK waiting timeout based on the maximum one-way propagation delay. Step S102: Divide the continuous communication time into several transmission frames, and divide each transmission frame into multiple transmission time slots according to the time slot length to obtain the action space of each underwater transmission node; the action space corresponds to all selectable transmission time slots within a transmission frame; Step S103: Each underwater transmitting node independently maintains a local Q table to obtain the initial Q value corresponding to each transmitting time slot; the Q value is used to represent the historical transmission effect of the transmitting time slot. Step S104: When the underwater transmitting node has data to be transmitted, it reads the local Q table and selects a transmission time slot according to the Q value to obtain the selected transmission time slot; Step S105: Send a data packet within the selected transmission time slot and start an ACK waiting timer. Wait for an ACK confirmation packet from the aggregation node before the ACK waiting timeout period expires to obtain the ACK reception status. Step S106: Based on the ACK reception status, determine whether the transmission was successful or failed, obtain the result of the success or failure of the transmission, and give a positive reward or a negative penalty to the selected transmission time slot. Step S107: Update the Q value of the corresponding transmission slot in the local Q table according to the positive reward or negative penalty to obtain the updated local Q table; the updated local Q table is used for the next transmission slot selection. Step S108: If transmission fails, the system enters a backoff state. The backoff time is calculated based on the time slot length. After the backoff ends, the system returns to the step of obtaining the selected transmission time slot and uses the updated local Q table as the read object. Through multiple iterations, different underwater transmission nodes gradually select different transmission time slots to obtain the distributed time slot self-organization result.

[0064] Steps S101 to S108 of this embodiment incorporate the long propagation delay characteristics of the underwater acoustic channel into time slot partitioning and timeout judgment. The selection of transmission time slots is modeled as a reinforcement learning process centered on a local Q-table. Each underwater transmitting node does not require central scheduling or extensive control packet interaction; it autonomously learns and memorizes the historical transmission effects of each time slot by relying on ACK confirmations or timeout events after data packet transmission as reward signals. Upon success, the Q-value of the corresponding time slot is increased to increase its probability of being selected subsequently; upon failure, the Q-value is decreased and random backoff based on time slot length is triggered. After multiple iterations, the local Q-tables of different nodes gradually differentiate, causing each node to tend to select different transmission time slots. This achieves distributed time slot self-organization in an environment of long underwater acoustic propagation delay and multi-node competition, effectively reducing data packet collision rate, improving channel utilization and data transmission stability, while maintaining the engineering advantages of low protocol overhead, no need for strict synchronization, and ease of deployment in underwater acoustic network nodes.

[0065] In some embodiments, step S101 may include, but is not limited to, steps S111 to S114: Step S111: Obtain the underwater acoustic propagation speed in the underwater acoustic channel, and the maximum communication distance between the underwater transmitting node and the sinking node; the underwater transmitting node includes an underwater sensor node, an underwater communication node, or an underwater unmanned platform equipped with underwater acoustic communication equipment, used to collect underwater environmental data and send data packets to the sinking node; the sinking node includes an underwater gateway, a surface buoy node, or a data collection node, used to receive data packets uploaded by the underwater transmitting node and return an ACK confirmation packet after correct reception; Step S112: Divide the maximum communication distance by the underwater acoustic propagation speed to obtain the maximum one-way propagation delay; Step S113: Set the time slot length according to the maximum one-way propagation delay; Step S114: Set the ACK waiting timeout period according to the maximum one-way propagation delay, so that the ACK waiting timeout period is equal to the sum of the data packet sending time, the ACK confirmation packet sending time, twice the maximum one-way propagation delay, and the protection interval.

[0066] In steps S111 to S114 of some embodiments, the physical characteristics of the underwater acoustic channel are quantified into specific protocol parameters, enabling subsequent time slot allocation and timeout judgment to adapt to the long propagation delay environment of underwater acoustics and avoiding channel state misjudgment and data packet collisions caused by propagation delay. For example, in a scenario where an underwater transmitting node and a sinking node are 5000 meters apart, the sound wave propagation speed is 1500 meters per second, and the maximum one-way propagation delay is approximately 3.33 seconds. When setting the ACK waiting timeout based on this delay, the data packet sending time, the ACK confirmation packet sending time, twice the maximum one-way propagation delay, and the guard interval need to be added together, totaling approximately 7.5 seconds. If the millisecond-level timeout setting common in terrestrial wireless networks is used instead of considering the underwater acoustic propagation delay, the node will determine a timeout before the ACK is returned, resulting in a large number of unnecessary retransmissions and a sharp drop in throughput. Through parameter settings, the protocol can correctly judge the success or failure of transmission in the long propagation delay environment of underwater acoustics.

[0067] In some embodiments, step S102 may include, but is not limited to, steps S201 to S203: Step S201: Divide the continuous communication time between underwater transmitting nodes into multiple transmission frames of equal length, with each transmission frame serving as a transmission cycle; Step S202: Based on the time slot length, each transmission frame is uniformly divided into multiple transmission time slots of equal length in the time domain; the length of each transmission time slot is equal to the time slot length, and adjacent transmission time slots are separated by time slot boundaries; Step S203: The action space of the underwater transmitting node is formed by dividing all the transmission time slots within each transmission frame; each transmission time slot in the action space corresponds to an optional action in the reinforcement learning process; when the underwater transmitting node needs to transmit data, it selects a transmission time slot from the action space as the action for this data transmission.

[0068] In steps S201 to S203 of some embodiments, time resources are discretized into a finite number of selectable transmission opportunities, and these transmission opportunities are used as "actions" in reinforcement learning. This transforms the originally continuous and difficult-to-determine channel access problem into a selection problem within a finite action space, reducing decision complexity. For example, if each transmission frame is set to a length of 40 seconds and a time slot length of 4 seconds, then each transmission frame can be divided into 10 transmission time slots. These 10 time slots constitute the action space of the node, with each time slot corresponding to one selectable action. In a network with 10 underwater transmission nodes, each node only needs to select one transmission from the 10 time slots in each transmission frame. If time slots are not divided, nodes can transmit at any time, resulting in a very high probability of collision. After dividing time slots, collisions only occur when multiple nodes simultaneously select the same time slot. The probability of collision is reduced from an infinite number of possibilities in continuous time to a finite combination of finite time slots, which provides a feasible decision space for subsequent reinforcement learning.

[0069] In some embodiments, step S103 may include, but is not limited to, steps S301 to S302: Step S301: Each underwater transmitting node independently establishes and maintains a local Q table in its local memory; the local Q table adopts a one-dimensional table structure; each entry in the local Q table corresponds to a transmitting time slot, and each entry is used to store the Q value corresponding to the transmitting time slot; Step S302: During the protocol initialization phase, the Q values ​​of all entries in the local Q table are set to the same initial value so that each transmission time slot has the same probability of being selected in the initial state. The Q value is used to represent the historical transmission effect of the corresponding transmission time slot. The higher the Q value, the higher the historical transmission success rate of the transmission time slot. The lower the Q value, the higher the historical transmission failure rate of the transmission time slot. The local Q table is stored locally on the underwater transmission node.

[0070] In steps S301 to S302 of some embodiments, each node independently memorizes the historical transmission performance of each time slot, storing the experience in its local Q-table. This eliminates the need for inter-node information exchange or reliance on a central node, achieving a distributed learning mechanism. For example, in a system with 10 time slots, each underwater transmitting node establishes a one-dimensional Q-table in its local memory, containing 10 entries, each corresponding to a transmission time slot. Initially, all Q values ​​are set to 0, and each time slot has the same probability of being selected in the initial state. Node A and Node B each maintain their own Q-table without interference. Node A's Q-table records the success rate of its transmission in each time slot, while Node B's Q-table records the success rate of its transmission in each time slot. After a period of learning, Node A discovers that time slot 3 is frequently successful, and its Q value gradually increases; Node B discovers that time slot 3 frequently conflicts, and its Q value gradually decreases, switching to time slot 7. The entire process requires no exchange of Q-table information between nodes, nor does it require a central node for unified scheduling, achieving distributed learning.

[0071] In some embodiments, step S104 may include, but is not limited to, steps S401 to S402: Step S401: When there are data packets to be sent in the data buffer of the underwater transmitting node, the underwater transmitting node enters the transmission time slot selection process. The underwater transmitting node reads the Q values ​​corresponding to all transmission time slots stored in the local Q table. Step S402: Based on the Q values ​​of each transmission time slot read, a selection strategy is adopted to select a transmission time slot from the action space as the selected transmission time slot; the selection strategy is either the maximum Q value strategy or the Softmax probability selection strategy; the maximum Q value strategy selects the transmission time slot with the largest current Q value as the selected transmission time slot; the Softmax probability selection strategy calculates the probability of each transmission time slot being selected based on the Q value of each transmission time slot, and the higher the Q value, the greater the probability of the transmission time slot being selected.

[0072] In steps S401 to S402 of some embodiments, historical learning results are transformed into current decision-making behavior, enabling nodes to prioritize time slots with high historical transmission success rates, while achieving a balance between utilization and exploration through different selection strategies. This application provides two time slot selection strategies. The first is the maximum Q-value strategy, where nodes always select the time slot with the highest current Q-value for transmission. This strategy is suitable for scenarios with stable network load and fixed node locations, and can quickly converge to the optimal time slot. The second is the Softmax probability selection strategy, where nodes calculate the selection probability based on the Q-value of each time slot. Time slots with higher Q-values ​​have a greater probability of being selected, but time slots with lower Q-values ​​still retain a certain probability of being selected. For example, in node A's Q-table, time slot 1 has a Q-value of 10, time slot 2 has 8, and time slot 3 has 2. Using the Softmax strategy, the selection probability of time slot 1 is approximately 50%, time slot 2 is approximately 30%, and time slot 3 is approximately 5%. Although time slot 3 has a poor historical performance, it still has a 5% probability of being selected. If the network topology changes and node A moves to a new location, time slot 3 may become the optimal time slot. The Softmax strategy allows the node to explore this change and relearn, while the maximum Q-value strategy will lock in time slot 1 and will not adapt to environmental changes.

[0073] In some embodiments, step S105 may include, but is not limited to, steps S501 to S505: In step S501, within the selected transmission time slot, the underwater transmitting node transmits data packets through the underwater acoustic channel; the data packets include the transmitting node identifier, the receiving node identifier, the data packet sequence number, the data payload, and the verification information. Step S502: After the data packet is sent, the underwater transmitting node starts the ACK waiting timer and sets the timeout of the ACK waiting timer to the ACK waiting timeout period. Step S503: The underwater transmitting node continuously listens to the underwater acoustic channel before the ACK waiting timeout period expires to wait for the ACK confirmation packet from the aggregation node; the ACK confirmation packet includes the target transmitting node identifier and the sequence number of the confirmed data packet; Step S504: If an ACK confirmation packet is received before the ACK waiting timeout period expires, and the target sending node identifier in the ACK confirmation packet is consistent with the sending node identifier in the data packet, and the sequence number of the confirmed data packet is consistent with the sequence number of the data packet, then the ACK reception status of having received the ACK is obtained. Step S505: If a valid ACK confirmation packet is not received by the time the ACK waiting timeout period expires, the ACK reception status of ACK timeout is obtained.

[0074] In steps S501 to S505 of some embodiments, the long underwater acoustic propagation delay is incorporated into the timeout judgment to avoid misjudgment caused by propagation delay; simultaneously, the content verification of the ACK confirmation packet ensures that the reward signal corresponds to the data packet sent this time, avoiding mismatch. This application embodiment provides the field structure of the data packet and the ACK confirmation packet, as well as the judgment logic for the ACK reception status. The data packet contains fields such as the sending node identifier and the data packet sequence number, and the ACK confirmation packet contains the target sending node identifier and the sequence number of the confirmed data packet. Node A sends a data packet with sequence number 5, and then sends a data packet with sequence number 6. If the ACK confirmation returned by the sink node is for the data packet with sequence number 5, but Node A mistakenly believes it is a successful feedback for sequence number 6, it will incorrectly give a positive reward to time slot 6, while in reality, time slot 6 may have severe conflicts. Through sequence number verification, Node A can correctly match the ACK with the corresponding sending time slot, ensuring the accuracy of the Q value update. Furthermore, at a distance of 5000 meters, the one-way propagation delay is approximately 3.33 seconds, and the round-trip delay is approximately 6.66 seconds. If the ACK waiting timeout is set to 7.5 seconds, the node has sufficient time to receive the ACK; if it is set to 1 second, the node will never receive the ACK, and all transmissions will be considered failed. By correctly setting the timeout and performing content verification, the accuracy of the reward signal is ensured.

[0075] In some embodiments, step S106 may include, but is not limited to, steps S601 to S602: Step S601: Use the underwater transmitting node to obtain the ACK reception status. If the ACK reception status is "ACK received", the transmission is determined to be successful. Obtain the judgment result of successful transmission and give a positive reward to the selected transmission time slot. The positive reward is a positive value. The positive reward is used to increase the Q value of the selected transmission time slot. Step S602: If the ACK reception status is ACK timeout, the transmission is determined to have failed, the transmission failure judgment result is obtained, and a negative penalty is applied to the selected transmission time slot; the negative penalty is a negative value or less than the positive reward value; the negative penalty is used to reduce the Q value of the selected transmission time slot.

[0076] In steps S601 to S602 of some embodiments, the existing ACK confirmation mechanism of the MAC layer is transformed into a reward signal for reinforcement learning, without the need for additional control packet overhead. Successful time slots are reinforced with positive rewards, and conflicting time slots are weakened with negative penalties, forming a clear learning orientation. This application provides specific values ​​for rewards and penalties and their effects. When transmission is successful, the node gives a positive reward to the selected time slot, for example, adding 1, increasing the Q value of the time slot and increasing the probability of it being selected subsequently. When transmission fails, the node gives a negative penalty to the selected time slot, for example, subtracting 1, decreasing the Q value of the time slot and decreasing the probability of it being selected subsequently. For example, if node A sends data in time slot 3 and successfully receives an ACK, the Q value of time slot 3 increases from 5 to 5.1, and the probability of it being selected in the next decision increases slightly. Conversely, if transmission fails in time slot 5, the Q value of time slot 5 decreases from 5 to 4.9, and the probability of it being selected in the next decision decreases slightly. After 100 transmissions, the Q value of the successful time slot will be significantly higher than the Q value of the conflicting time slot, and the node will naturally tend to select the successful time slot. If the failure penalty is set to 0 instead of a negative number, the Q value of the failure slot remains unchanged, and the node will still choose the failure slot with a certain probability, which will greatly reduce the learning convergence speed.

[0077] In some embodiments, step S107 may include, but is not limited to, steps S701 to S702: Step S701: Use the underwater transmitting node to obtain positive rewards or negative penalties, and obtain the selected transmitting time slot; Step S702: Using the Q-value update formula in the Q-learning algorithm, update the entries in the local Q-table corresponding to the selected transmission slot to obtain the updated local Q-table. The Q-value update formula is: the new Q-value equals the difference between one and the learning rate multiplied by the old Q-value, plus the learning rate multiplied by the reward value obtained in this transmission. The old Q-value is the Q-value corresponding to the selected transmission slot before the update; the new Q-value is the Q-value corresponding to the selected transmission slot after the update. The learning rate is a preset value between 0 and 1, used to control the weight ratio of the old Q-value and the reward value in the update. When the reward value is a positive reward, the new Q-value is greater than the old Q-value; when the reward value is a negative penalty, the new Q-value is less than the old Q-value.

[0078] In steps S701 to S702 of some embodiments, the weighted update formula in the Q-learning algorithm enables the Q-value to smoothly reflect the combined effect of long-term historical experience and the latest feedback, avoiding excessive influence of a single abnormal result on the Q-value. This application provides a Q-value update formula where the new Q-value equals the difference between the old and current Q-values ​​multiplied by the learning rate, plus the learning rate multiplied by the reward value obtained in this transmission. The learning rate is a value between 0 and 1, used to control the weight ratio of the old Q-value and the current reward value in the update. When the learning rate is small, such as 0.1, historical experience accounts for 90%, and new feedback only accounts for 10%, resulting in smooth Q-value changes, strong anti-interference ability, and suitability for scenarios with relatively stable channels. When the learning rate is large, such as 0.5 or 0.9, new feedback has a larger weight, resulting in a fast Q-value response, suitable for scenarios with dynamically changing network topology or channel conditions. For example, in a stable deep-sea environment, nodes can use a smaller learning rate to avoid the impact of occasional channel interference (such as the instantaneous attenuation caused by a school of fish passing by) on the Q value. In dynamic scenarios where nodes move or new nodes are added, nodes can use a larger learning rate to quickly adapt to the new optimal time slot distribution.

[0079] In some embodiments, step S108 may include, but is not limited to, steps S801 to S804: Step S801: When the judgment result is transmission failure, the underwater transmitting node enters the backoff state. In step S802, under the backoff state, the underwater transmitting node calculates the backoff time based on the time slot length; the backoff time is equal to a random integer multiplied by the time slot length; the random integer is selected between zero and the backoff window size; Step S803: Use the underwater transmitting node to suspend transmission during the backoff time and wait for the backoff time to end before exiting the backoff state; Step S804 involves obtaining the selected transmission time slot through the return of the underwater transmitting node, and using the updated local Q table as the reading object to reselect the transmission time slot for data transmission. Through multiple rounds of feedback on successful and failed transmissions, each underwater transmitting node independently updates its local Q table during multiple iterations to increase the Q value of successful time slots and decrease the Q value of conflicting time slots, thus obtaining the distributed time slot self-organization result.

[0080] In steps S801 to S804 of some embodiments, the backoff mechanism is combined with Q-table learning. Upon failure, the Q-value of the conflicting time slot is reduced through negative penalties, and random backoff prevents multiple nodes from repeatedly colliding within the same time slot. Through multiple iterations, the Q-tables of each node naturally differentiate, achieving distributed time slot self-organization without central scheduling. This application provides the backoff time and the effects of multiple iterations. The backoff time is equal to a random integer multiplied by the time slot length, with the random integer uniformly selected between zero and the backoff window size. For example, if the time slot length is 4 seconds and the backoff window is 5, the backoff time could be a random value of 0 seconds, 4 seconds, 8 seconds, 12 seconds, or 16 seconds. This random backoff mechanism can prevent multiple failed nodes from retrying at the same time, causing continuous collisions. More importantly, the backoff is combined with Q-table learning: after a node fails, it reduces the Q-value of the conflicting time slot and temporarily relinquishes the channel through random backoff, giving other nodes a chance to transmit. After multiple iterations, for example, in a system with three nodes and 10 time slots, initially all Q values ​​are 0, and each node randomly selects a time slot, resulting in frequent collisions. After about 50 iterations, node A finds that time slot 3 often succeeds, and its Q value increases; node B finds that time slot 3 has severe collisions, and its Q value decreases, then finds that time slot 7 has a higher success rate; node C stabilizes in time slot 5. After about 100 iterations, the three nodes stabilize in time slots 3, 7, and 5 respectively, without collisions, forming an approximately orthogonal time slot distribution. The entire process does not require a central node to allocate time slots, nor does it require any information exchange between nodes. It learns through each node's own ACK feedback, resulting in a distributed time slot self-organization.

[0081] This application also provides a time-delay-aware underwater acoustic network media access control system based on reinforcement learning, for implementing the method described above. The system includes multiple underwater transmitting nodes and at least one aggregation node. Each underwater transmitting node includes: The time slot division unit is used to calculate the maximum one-way propagation delay based on the underwater acoustic propagation speed in the underwater acoustic channel and the maximum communication distance between the underwater transmitting node and the sinking node, and to set the time slot length and ACK waiting timeout based on the maximum one-way propagation delay; and to divide the continuous communication time into several transmission frames, and to divide each transmission frame into multiple transmission time slots according to the time slot length, so as to obtain the optional action space of each underwater transmitting node. The Q-table storage unit is used to independently maintain a local Q-table, storing the Q-values ​​corresponding to each transmission time slot; the Q-values ​​are used to represent the historical transmission effect of the transmission time slot. The time slot selection unit is used to read the local Q table and select a transmission time slot according to the Q value when there is data to be transmitted, thus obtaining the selected transmission time slot. An underwater acoustic transmitting unit is used to transmit data packets within a selected transmission time slot; The ACK receiving unit is used to wait for an ACK acknowledgment packet from the aggregation node before the ACK waiting timeout period expires, and to obtain the ACK receiving status. The timeout judgment unit is used to determine whether the transmission was successful or failed based on the ACK reception status, and obtain the judgment result of whether the transmission was successful or failed. The Q-value update unit is used to update the Q-value of the corresponding transmission slot in the local Q-table according to the positive reward or negative penalty, so as to obtain the updated local Q-table; The backoff and retransmission unit is used to enter the backoff state when transmission fails. The backoff time is calculated based on the time slot length. After the backoff ends, the time slot selection unit is retried. The aggregation nodes include: The data receiving unit is used to receive data packets sent by the underwater transmitting node; The data decoding unit is used to determine whether the data packet has been received correctly. The ACK feedback unit is used to return an ACK confirmation packet to the corresponding underwater transmitting node after the data packet has been correctly received.

[0082] As an optional implementation, this invention proposes a delay-aware underwater acoustic network media access control method and related equipment based on reinforcement learning. This method addresses the problems of large propagation delays, strong node concurrency contention, and high packet collision probability in underwater acoustic networks by introducing Q-Learning, or Q-learning algorithm, from reinforcement learning into the underwater acoustic network media access control layer. Each underwater transmitting node selects multiple transmission slots within a transmission cycle as optional actions and maintains a Q-table locally. The node updates the Q-value of the selected slots based on ACK confirmation packets or timeout failure events after data transmission, thereby gradually learning suitable transmission slots and reducing packet collisions caused by multiple nodes competing for the channel at the same time.

[0083] Among them, MAC stands for Medium Access Control; ACK stands for Acknowledgement; Q-Learning is the Q-learning algorithm; Q-Table is the Q-table; and DA-ALOHA-Q stands for Delay-Aware ALOHA with Q-Learning, which can be described as "Delay-Aware ALOHA Protocol Based on Q-Learning".

[0084] (I) Overall technical concept of the present invention: Figure 2This invention addresses the problem of long propagation delay in underwater acoustic networks. In underwater acoustic networks, the speed of sound propagation is approximately 1500 m / s, far lower than the propagation speed of radio signals on land. Therefore, when a node determines the channel is idle and prepares to send data, data packets already sent by other remote nodes may still be in transit and have not yet reached that node or the aggregation node. If the node continues to send data at this time, multiple data packets may overlap at the receiving end, leading to data packet collisions.

[0085] Traditional ALOHA protocols use a "send data as soon as it's available" approach, leaving nodes with little memory or learning ability for historical collisions. While traditional time-slotted ALOHA protocols divide time into fixed time slots, nodes may still choose the same time slot for extended periods, leading to persistent collisions. Although RTS / CTS-based handshake mechanisms can reduce some collisions, they introduce significant control packet overhead and latency in long-delay underwater acoustic channels. While TDMA protocols can avoid collisions through fixed time slot allocation, they typically require centralized scheduling and strict synchronization, making deployment complex.

[0086] To address the aforementioned problems, the basic idea of ​​this invention is to eliminate the central node's pre-allocation of time slots to all sending nodes and to avoid reliance on frequent handshake control packets. Instead, each sending node autonomously learns which time slot is more suitable for transmission based on its own success or failure results. Specifically, each node treats each transmission time slot as a selectable action, updating its local Q-table using ACK success feedback and timeout failure feedback, increasing the Q-value of successful time slots and decreasing the Q-value of failed time slots. After multiple rounds of transmission and feedback, each node gradually develops different time slot selection preferences, thereby reducing mutual conflicts.

[0087] (II) Composition of the underwater acoustic network system: Referring to Figures 3(a) and 3(b), the composition of the underwater acoustic network system to which this invention applies is illustrated. This invention is applicable to underwater acoustic communication networks consisting of multiple underwater transmitting nodes, one or more underwater converging nodes, and underwater acoustic channels. Transmitting nodes can be underwater sensor nodes, underwater communication nodes, or underwater unmanned platforms equipped with underwater acoustic communication devices; converging nodes can be underwater gateways, surface buoy nodes, or data collection nodes.

[0088] In a typical implementation, the system includes a sink node and multiple transmitting nodes. The sink node is responsible for receiving data packets uploaded by each transmitting node and returning an ACK confirmation packet upon successful reception. The transmitting nodes are responsible for collecting underwater environmental data and selecting transmission time slots according to their local Q-tables to send data packets to the sink node. An underwater acoustic channel is used to carry acoustic communication signals between the nodes.

[0089] Each transmitting node includes at least the following functional units: 1. Data Acquisition Unit: Used to collect underwater environmental data, such as temperature, pressure, salinity, water flow, target detection information, etc.

[0090] 2. Data buffer unit: Used to store data packets waiting to be sent.

[0091] 3. Time Slot Division Unit: Used to divide the transmission period into multiple transmission time slots according to protocol parameters.

[0092] 4. Q-table storage unit: used to store the Q value corresponding to each time slot.

[0093] 5. Time slot selection unit: used to select the transmission time slot of this data packet according to the Q table.

[0094] 6. Underwater acoustic transmission unit: Used to transmit data packets within a selected time slot.

[0095] 7. ACK receiving unit: Used to receive acknowledgment packets returned by the aggregation node.

[0096] 8. Timeout Judgment Unit: Used to determine whether a node has received an ACK within the specified time.

[0097] 9. Q-value update unit: Used to update the Q table based on ACK or timeout results.

[0098] 10. Backoff and retransmission unit: Used to perform backoff after transmission failure and re-enter the transmission decision process.

[0099] A convergence node includes at least the following functional units: 1. Data receiving unit: Used to receive data packets uploaded by the sending node.

[0100] 2. Data decoding unit: used to determine whether the data packet has been received correctly.

[0101] 3. ACK Feedback Unit: Used to return an ACK confirmation packet to the node that successfully sent the data.

[0102] (III) Transmission Frame and Time Slot Allocation Method: refer to Figure 4 This invention illustrates the frame structure and time slot structure. The invention divides continuous communication time into several transmission frames, and each transmission frame is further divided into multiple selectable transmission time slots. When a transmitting node needs to send data, it does not send it directly at any arbitrary time, but rather selects one time slot within a transmission frame to send it.

[0103] A transmission frame can be represented as:

[0104] in, Represents a transmission frame. Indicates the first One transmission slot, This indicates the number of time slots within each transmission frame.

[0105] In this invention, a time slot is not only a time window for nodes to send data, but also an action in reinforcement learning. That is, when a node selects a time slot to send data, it is equivalent to selecting an action during the reinforcement learning process. Through this design, the medium access control problem in underwater acoustic networks can be transformed into a learning problem of "how a node selects the optimal transmission time slot from multiple candidate time slots".

[0106] The time slot length needs to be set in conjunction with the propagation delay of the underwater acoustic channel. Let the maximum communication distance between nodes be... The speed of sound propagation in water is The maximum one-way propagation delay can then be expressed as:

[0107] in, This represents the maximum one-way propagation delay.

[0108] The data packet transmission time can be expressed as:

[0109] in, Indicates the time the data packet was sent. Indicates the length of the data packet. This indicates the transmission rate.

[0110] The ACK confirmation packet sending time can be expressed as:

[0111] in, Indicates the time when the ACK confirmation packet was sent. Indicates the length of the ACK packet.

[0112] The timeout for a node to wait for an ACK after sending data can be set as follows:

[0113] in, This indicates the timeout period for the ACK request. This indicates the guard interval. The guard interval is used to mitigate the effects of underwater acoustic propagation delay fluctuations, node processing delays, and channel variations.

[0114] Through the above design, the present invention incorporates underwater acoustic propagation delay into the time slot division and timeout judgment process, enabling the protocol to adapt to underwater long propagation delay environments.

[0115] (iv) Method for creating a local Q table: Each underwater transmitting node independently maintains a local Q-table. Since this invention treats transmission slots as optional node actions, the Q-table can employ a one-dimensional table structure. For the... For each sending node, its Q table can be represented as:

[0116] in, Indicates the first The node is selected. The action value corresponding to the transmission in each time slot.

[0117] The Q value represents the historical transmission performance of a given time slot for the current node. If a time slot successfully transmits multiple times and receives ACKs, the Q value for that time slot will gradually increase; conversely, if a time slot fails to transmit multiple times and times out, the Q value for that time slot will gradually decrease. A higher Q value indicates that the time slot is more suitable for the node to send data; a lower Q value indicates that the time slot is more prone to collisions or transmission failures.

[0118] During the protocol initialization phase, the Q values ​​for each time slot can be set to the same initial value. For example, they can all be initialized to 0, or they can be initialized to the same positive number. After initialization, each time slot has the same selection opportunity in the initial state. As nodes continuously send data and receive feedback, the Q values ​​of different time slots will gradually differentiate, thus forming the node's selection preference for each time slot.

[0119] The Q table is stored locally only at the sending node and does not need to be maintained by the aggregation node or exchanged between nodes. Therefore, this invention can operate without central scheduling and without global network state information, making it suitable for low-bandwidth, long-latency, and resource-constrained underwater acoustic sensor networks.

[0120] (v) Transmission time slot selection method: When a transmitting node has data packets to be sent in its data buffer, the node enters the transmission time slot selection process. The node first reads its local Q table, and then selects a transmission time slot based on the current Q value.

[0121] In one implementation, the node can select the time slot with the largest current Q value as the transmission time slot:

[0122] in, This indicates the sending action selected by the node, i.e., the selected sending time slot.

[0123] In another implementation, nodes can employ a Softmax probability selection strategy. This strategy does not always select the time slot with the highest Q-value, but rather calculates the selection probability based on the Q-values ​​of each time slot. The probability of a time slot being selected can be expressed as:

[0124] in, Indicates the first The probability of a time slot being selected This represents the temperature parameter. Time slots with higher Q values ​​have a greater probability of being selected; time slots with lower Q values ​​still retain a certain probability of being selected.

[0125] The Softmax strategy allows nodes to strike a balance between "exploitation" and "exploration." Exploitation means nodes prioritize time slots with historically good transmission performance; exploration means nodes retain the opportunity to try other time slots. When network load changes, nodes join or leave, or node locations change, the exploration mechanism helps nodes rediscover more suitable transmission time slots, improving the protocol's dynamic adaptability.

[0126] (vi) Data transmission and ACK feedback methods: After selecting a transmission time slot, the node schedules the data packets to be transmitted to the corresponding time slot. When the system time reaches the start of that time slot, the node transmits the data packets through the underwater acoustic channel.

[0127] A data packet can include at least the sending node identifier, receiving node identifier, packet sequence number, data payload, and checksum information. The sending node identifier is used by the aggregation node to identify the data source; the packet sequence number is used to confirm the specific data packet corresponding to the ACK; and the checksum information is used to determine whether the data packet was received correctly.

[0128] After sending a data packet, the sending node starts an ACK waiting timer. If the sink node correctly receives the data packet, it generates an ACK confirmation packet and returns it to the sending node. The ACK confirmation packet includes at least the target sending node identifier, the sequence number of the acknowledged data packet, and checksum information. Upon receiving the ACK, the sending node determines whether the ACK corresponds to the currently sent data packet based on the node identifier and data packet sequence number in the ACK.

[0129] If data packets sent by multiple nodes collide at the aggregation node, or if data packets cannot be correctly decoded due to channel interference, the aggregation node will not return an ACK. If the sending node does not receive a correct ACK within the ACK waiting time, it determines that the transmission has failed and triggers a timeout.

[0130] (vii) ACK / Timeout Reward Feedback Method: This invention utilizes ACK confirmation packets and timeout events as reward feedback for Q-Learning. After a node sends data, if it receives an ACK within a specified time, the transmission is considered successful, and a positive reward is given to the selected time slot; if the node does not receive an ACK within the specified time, the transmission is considered a failure, and a negative penalty is given to the selected time slot.

[0131] The reward function can be expressed as:

[0132] in, The reward for success is usually a positive value; This indicates the penalty for failure, and is usually a negative or small value.

[0133] For example, it can be set as:

[0134] The reward function can be further extended according to actual application needs. For example, a higher reward can be given for successful transmission if the transmission latency is low; a lower reward can be given if success is achieved after multiple retransmissions; and a greater penalty can be given if nodes time out consecutively. The specific value of the above reward function can be adjusted according to the scale of the underwater acoustic network, node load, and protocol parameters.

[0135] The advantage of this feedback mechanism is that ACK and timeout events are information that can be obtained during reliable transmission at the MAC layer. This invention can convert them into reinforcement learning feedback signals without adding additional complex control packets, thereby reducing protocol overhead.

[0136] (viii) Q table update method: The node first selects a transmission slot according to the Q table, and then sends data in the selected slot; after sending, it waits for ACK; if an ACK is received, a successful update is performed; if a timeout occurs, a failed update is performed and a backoff is initiated; after the backoff is completed, a new slot is selected.

[0137] The sending node updates the Q value corresponding to the currently selected time slot based on the reward feedback. In a simplified implementation, the Q value update formula is:

[0138] in, Indicates the number before the update The Q value of each time slot, This represents the updated Q value. Indicates the learning rate. This indicates the reward received during the current transmission.

[0139] When a node successfully sends data and receives an ACK in a certain time slot, a reward is given. A positive Q value increases the Q value corresponding to that time slot, increasing the probability of it being selected later. When a node fails to transmit in a time slot and a timeout occurs, the reward... If the value is negative or small, the Q value corresponding to that time slot decreases, and the probability of it being selected in the future decreases.

[0140] (ix) Timeout backoff and retransmission methods: If the sending node does not receive a correct ACK within the ACK waiting time, the node determines that the transmission has failed. At this time, the node first updates the Q value of the current transmission slot negatively, and then enters the backoff state.

[0141] In the random backoff implementation, the backoff time can be expressed as:

[0142] in, Indicates the time of retreat. Indicates the size of the backoff window. Indicates from 0 to Select an integer randomly from the range. Indicates the time slot length.

[0143] After the backoff process ends, the node re-enters the transmission decision-making process. Since the Q value of the time slot that just failed has been reduced, the probability of the node continuing to select the conflicting time slot will decrease, thereby reducing the probability of consecutive conflicts.

[0144] The combination of the backoff mechanism and the Q-table update mechanism can prevent nodes from repeatedly occupying the same conflict time slot in a short period of time, thereby improving the stability of the protocol.

[0145] (x) Complete workflow of the agreement: The complete workflow of this invention is as follows: 1. Deploy multiple underwater transmitting nodes and at least one aggregation node in the underwater acoustic network; 2. Set the time slot length, frame length, and ACK waiting time based on the underwater acoustic propagation speed, maximum node communication distance, data packet length, and transmission rate; 3. Divide the continuous communication time into several transmission frames, and divide each transmission frame into multiple transmission time slots; 4. Each transmitting node initializes its local Q table, giving each time slot an initial Q value; 5. After the sending node generates a data packet, it puts the data packet into the buffer queue; 6. The transmitting node reads the local Q table and selects a transmission slot based on the maximum Q value strategy and the Softmax strategy; 7. The node waits for the selected time slot to arrive and sends data packets within that time slot; 8. After sending a data packet, the node starts an ACK waiting timer; 9. If the aggregation node correctly receives the data packet, it returns an ACK confirmation packet to the corresponding sending node; 10. If the sending node receives an ACK within the specified time, the transmission is considered successful, a positive reward is given to the selected time slot, and the Q table is updated; 11. If the sending node does not receive an ACK within the specified time, it determines that the transmission has failed, applies a negative penalty to the selected time slot, and updates the Q table; 12. A node that fails to transmit enters a backoff state, and after the backoff process ends, it re-enters the time slot selection process; 13. Each node repeats the above process to gradually form a transmission time slot selection strategy that suits itself; 14. After multiple rounds of learning, different nodes tend to select different high-value time slots, thereby reducing packet collisions and improving the transmission stability of underwater acoustic networks.

[0146] (xi) Distributed time slot self-organizing effect: Figures 5(a), 5(b), 5(c), and 5(d) illustrate the learning results and slot self-organization effect of this invention. The Q-value heatmaps in the figures represent the selection preferences of different nodes for different slots. The higher the Q-value of a node in a certain slot, the higher the historical success rate of data transmission by that node in that slot. The slot allocation results are used to represent the transmission slots that multiple nodes ultimately tend to choose after learning.

[0147] Figures 5(a), 5(b), 5(c), and 5(d) illustrate that in this invention, nodes do not need a central node to uniformly allocate time slots, nor do they need to exchange Q tables with each other. Nodes only need to update their local Q tables based on whether they have received an ACK to gradually form a relatively stable time slot selection result. When multiple nodes select the same time slot and a conflict occurs, the corresponding time slot will be penalized, reducing the probability of the node selecting that time slot subsequently; when a node successfully sends data in a time slot, that time slot receives a positive reward, increasing the probability of subsequent selection.

[0148] After multiple rounds of learning, different nodes will gradually avoid time slots that are prone to conflict with each other, thus forming an approximately orthogonal time slot distribution. This process embodies the distributed adaptive feature of this invention.

[0149] (xii) The differences between this invention and traditional technical solutions: Figures 6(a) and 6(b) illustrate the performance differences of this invention compared to traditional ALOHA, adaptive backoff ALOHA, and other protocols. PDR represents packet delivery rate, and Throughput represents throughput.

[0150] The difference between this invention and the traditional ALOHA protocol is that nodes in the traditional ALOHA protocol send data directly when there is data available and randomly back off after failure, lacking the ability to remember conflicting time slots; this invention records the historical transmission effects of different time slots through a Q table, enabling nodes to gradually avoid time slots with more conflicts.

[0151] The difference between this invention and the slotted ALOHA protocol is that slotted ALOHA only restricts nodes to send at slot boundaries, but does not provide a mechanism to dynamically adjust slot selection based on transmission feedback; this invention uses slots as reinforcement learning actions and dynamically updates the Q value through ACK / timeout feedback.

[0152] The difference between this invention and RTS / CTS handshake protocols is that handshake protocols require the exchange of control packets before data transmission, while this invention mainly uses ACK and timeout events as learning feedback, without the need to introduce a large number of reservation control packets. Therefore, it is more suitable for long propagation delay and low bandwidth underwater acoustic channels.

[0153] The difference between this invention and the TDMA scheduling protocol is that TDMA usually requires the central node to pre-allocate fixed time slots and requires strict synchronization; this invention does not rely on central scheduling, and each node can gradually form a low-conflict transmission time slot selection result through local learning.

[0154] (xiii) The technical closed loop of the present invention: This invention forms a complete technical closed loop of "underwater acoustic delay sensing parameter setting - transmission frame and time slot division - node local Q table establishment - transmission time slot selection - data packet transmission - ACK / timeout feedback - Q table update - backoff retransmission - distributed time slot self-organization".

[0155] First, considering the slow propagation speed and long propagation delay of underwater acoustic channels, the time slot length and ACK waiting time are set. Second, each transmission frame is divided into multiple selectable time slots, which are used as the action space for Q-Learning. Then, each transmitting node independently maintains a Q-table and selects a transmission time slot based on the current Q value. After transmission, the node determines whether the transmission was successful or failed based on the ACK or timeout event and updates the Q-table accordingly. If transmission fails, the node enters a backoff state, and after the backoff ends, it performs time slot selection again. After multiple iterations, each node gradually develops a transmission time slot selection strategy suitable for itself, thereby reducing the probability of collisions.

[0156] Therefore, this invention is not simply a simulation analysis method, but a distributed channel access control method that can be deployed in the MAC layer of underwater communication nodes. Its core technical features are: incorporating underwater acoustic network propagation delay into time slot and timeout design, using transmission time slots as Q-Learning actions, and using ACK and timeout events as learning feedback, with each transmitting node independently maintaining a Q-table to complete distributed adaptive access. This technical solution can improve the reliability and stability of multi-node data transmission in underwater acoustic networks without relying on centralized scheduling and a large number of control packet handshakes.

[0157] The advantages of this invention are as follows: Compared to the closest related underwater acoustic network MAC protocols based on ALOHA, slotted ALOHA, or improved random access, this invention does not simply rely on random transmission, fixed time slots, or random backoff to avoid collisions. Instead, it models the transmission time slot selection process as a Q-Learning reinforcement learning process. Each underwater transmitting node can dynamically update its local Q-table based on whether it has received an ACK confirmation packet and whether a timeout failure has occurred, thereby gradually increasing the probability of selecting a successful time slot and reducing the probability of selecting a conflicting time slot. Therefore, this invention enables nodes to have adaptive learning capabilities for long underwater acoustic propagation delays and multi-node competition environments, reducing the problems of repeated collisions and blind retransmissions in traditional random access protocols.

[0158] Meanwhile, compared with related technologies based on RTS / CTS handshake or TDMA centralized scheduling, this invention does not require a large number of control packets to reserve channels, nor does it rely on a central node to uniformly allocate time slots or strict global synchronization. Each transmitting node can complete distributed time slot selection solely based on its local Q table and ACK / timeout feedback, offering advantages such as low control overhead, low implementation complexity, and suitability for resource-constrained underwater node deployment. Through the technical process of "delay-aware time slot allocation—local Q table learning—ACK / timeout feedback update—backoff retransmission," this invention can improve channel access stability, reduce packet collision rate, and enhance the reliability and engineering applicability of underwater acoustic network data transmission in long-delay underwater acoustic environments.

[0159] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0160] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0161] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0162] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0163] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0164] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0165] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0166] This application provides a reinforcement learning-based delay-aware underwater acoustic network media access control method, system, electronic device, storage medium, and program product. This application uses selectable time slots within a transmission frame as the action space for Q-Learning. Each node independently maintains its local Q-table and updates the selected time slots with rewards or penalties based on ACK success feedback and timeout failure feedback. Nodes do not require a central controller to uniformly allocate time slots, nor do they need to exchange complex neighbor state information. Instead, they gradually learn suitable transmission time slots based on local transmission feedback. In this way, while maintaining the advantages of ALOHA-type protocols in terms of simple structure and easy deployment, nodes possess adaptive collision avoidance capabilities, thereby reducing the packet collision rate in long-propagation-delay underwater acoustic environments.

[0167] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0168] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0171] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0172] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0174] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0176] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0177] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for media access control in a time-delay-aware underwater acoustic network based on reinforcement learning, characterized in that, The method includes the following steps: The maximum one-way propagation delay is calculated based on the underwater acoustic propagation speed in the underwater acoustic channel and the maximum communication distance between the underwater transmitting node and the sinking node. The time slot length and ACK waiting timeout are then set based on the maximum one-way propagation delay. The continuous communication time is divided into several transmission frames, and each transmission frame is divided into multiple transmission time slots according to the time slot length, so as to obtain the action space of each underwater transmission node; the action space corresponds to all the selectable transmission time slots within a transmission frame; Each of the underwater transmitting nodes independently maintains a local Q table to obtain the initial Q value corresponding to each transmitting time slot; the Q value is used to represent the historical transmission effect of the transmitting time slot. When the underwater transmitting node has data to be transmitted, it reads the local Q table and selects a transmission time slot according to the Q value to obtain the selected transmission time slot. Send data packets within the selected transmission time slot and start an ACK waiting timer. Wait for an ACK confirmation packet from the aggregation node before the ACK waiting timeout period expires to obtain the ACK reception status. Based on the ACK reception status, determine whether the transmission was successful or failed, obtain the result of the success or failure of the transmission, and give a positive reward or a negative penalty to the selected transmission time slot; The Q value corresponding to the transmission slot in the local Q table is updated according to the positive reward or negative penalty to obtain the updated local Q table; the updated local Q table is used for the next transmission slot selection; If transmission fails, the system enters a backoff state. The backoff time is calculated based on the time slot length. After the backoff ends, the system returns to the step of obtaining the selected transmission time slot and uses the updated local Q table as the read object. Through multiple iterations, different underwater transmission nodes gradually select different transmission time slots to obtain a distributed time slot self-organization result.

2. The method according to claim 1, characterized in that, The step of calculating the maximum one-way propagation delay based on the underwater acoustic propagation speed in the underwater acoustic channel and the maximum communication distance between the underwater transmitting node and the sinking node, and setting the time slot length and ACK waiting timeout based on the maximum one-way propagation delay, includes: The underwater acoustic propagation speed in the underwater acoustic channel and the maximum communication distance between the underwater transmitting node and the sinking node are obtained. The underwater transmitting node includes an underwater sensor node, an underwater communication node, or an underwater unmanned platform equipped with underwater acoustic communication equipment, which is used to collect underwater environmental data and send data packets to the sinking node. The sinking node includes an underwater gateway, a surface buoy node, or a data collection node, which is used to receive the data packets uploaded by the underwater transmitting node and return an ACK confirmation packet after correct reception. Divide the maximum communication distance by the underwater acoustic propagation speed to obtain the maximum one-way propagation delay; Set the time slot length according to the maximum one-way propagation delay; Set the ACK waiting timeout period according to the maximum one-way propagation delay, so that the ACK waiting timeout period is equal to the sum of the data packet sending time, the ACK confirmation packet sending time, twice the maximum one-way propagation delay, and the guard interval.

3. The method according to claim 1, characterized in that, The process of dividing the continuous communication time into several transmission frames, and further dividing each transmission frame into multiple transmission time slots according to the time slot length, to obtain the action space of each underwater transmission node includes: The continuous communication time between the underwater transmitting nodes is divided into multiple transmission frames of equal length, and each transmission frame is a transmission cycle. Based on the time slot length, each transmission frame is uniformly divided into multiple equal-length transmission time slots in the time domain; the length of each transmission time slot is equal to the time slot length, and adjacent transmission time slots are separated by time slot boundaries; The underwater transmitting node's action space is formed by dividing all the transmission time slots within each transmission frame; each transmission time slot in the action space corresponds to an optional action in the reinforcement learning process; when the underwater transmitting node needs to transmit data, it selects a transmission time slot from the action space as the action for this data transmission.

4. The method according to claim 1, characterized in that, The step of maintaining a local Q table independently for each of the underwater transmitting nodes to obtain the initial Q value corresponding to each transmitting time slot includes: Each of the underwater transmitting nodes independently establishes and maintains a local Q-table in its local memory; the local Q-table adopts a one-dimensional table structure; each entry in the local Q-table corresponds to a transmitting time slot, and each entry is used to store the Q value corresponding to the transmitting time slot; During the protocol initialization phase, the Q values ​​of all entries in the local Q table are set to the same initial value so that each transmission time slot has the same probability of being selected in the initial state. The Q value is used to represent the historical transmission effect of the corresponding transmission time slot. The higher the Q value, the higher the historical transmission success rate of the transmission time slot. The lower the Q value, the higher the historical transmission failure rate of the transmission time slot. The local Q table is stored locally on the underwater transmission node.

5. The method according to claim 1, characterized in that, When the underwater transmitting node has data to transmit, it reads the local Q table and selects a transmission time slot based on the Q value to obtain the selected transmission time slot, including: When there are data packets to be sent in the data buffer of the underwater transmitting node, the underwater transmitting node enters the transmission time slot selection process, and the underwater transmitting node reads the Q value corresponding to all the transmission time slots stored in the local Q table; Based on the Q values ​​of each of the transmitted time slots, a selection strategy is adopted to select one of the transmitted time slots from the action space as the selected transmitted time slot; the selection strategy is either a maximum Q value strategy or a Softmax probability selection strategy; the maximum Q value strategy selects the transmitted time slot with the largest current Q value as the selected transmitted time slot; the Softmax probability selection strategy calculates the probability of each transmitted time slot being selected based on the Q value of each transmitted time slot, and the higher the Q value, the greater the probability of the transmitted time slot being selected.

6. The method according to claim 1, characterized in that, The process of sending data packets within the selected transmission slot and starting an ACK waiting timer, waiting for an ACK acknowledgment packet from the aggregation node before the ACK waiting timeout period expires, and obtaining the ACK reception status includes: During the selected transmission time slot, the underwater transmitting node transmits data packets through the underwater acoustic channel; the data packets include a transmitting node identifier, a receiving node identifier, a data packet sequence number, a data payload, and verification information; After the data packet is sent, the underwater transmitting node starts an ACK waiting timer and sets the timeout of the ACK waiting timer to the ACK waiting timeout period. The underwater transmitting node continuously listens to the underwater acoustic channel before the ACK waiting timeout period expires to wait for an ACK confirmation packet from the aggregation node; the ACK confirmation packet includes the target transmitting node identifier and the sequence number of the confirmed data packet; If the ACK confirmation packet is received before the ACK waiting timeout period expires, and the target sending node identifier in the ACK confirmation packet is consistent with the sending node identifier in the data packet, and the sequence number of the confirmed data packet is consistent with the sequence number of the data packet, then the ACK reception status is obtained; If a valid ACK acknowledgment packet is not received by the time the ACK waiting timeout period expires, then the ACK reception status is ACK timeout.

7. The method according to claim 1, characterized in that, The step of determining whether the transmission was successful or failed based on the ACK reception status, obtaining the success or failure result, and giving a positive reward or negative penalty to the selected transmission slot includes: The underwater transmitting node is used to obtain the ACK reception status. If the ACK reception status is "ACK received", the transmission is determined to be successful, and a successful transmission result is obtained. A positive reward is given to the selected transmission time slot. The positive reward is a positive value. The positive reward is used to increase the Q value of the selected transmission time slot. If the ACK reception status is ACK timeout, the transmission is determined to have failed, a transmission failure judgment result is obtained, and a negative penalty is applied to the selected transmission time slot; the negative penalty is a negative value or less than the positive reward value; the negative penalty is used to reduce the Q value of the selected transmission time slot.

8. The method according to claim 1, characterized in that, The step of updating the Q value in the local Q table corresponding to the transmission slot according to the positive reward or negative penalty to obtain the updated local Q table includes: The underwater transmitting node is used to obtain the positive reward or negative penalty, and to obtain the selected transmitting time slot; The Q-value update formula in the Q-learning algorithm is used to update the entries in the local Q-table corresponding to the selected transmission slot, resulting in an updated local Q-table. The Q-value update formula is: the new Q-value equals the difference between one and the learning rate multiplied by the old Q-value, plus the learning rate multiplied by the reward value obtained in this transmission. The old Q-value is the Q-value corresponding to the selected transmission slot before the update; the new Q-value is the Q-value corresponding to the selected transmission slot after the update. The learning rate is a preset value between 0 and 1, used to control the weight ratio of the old Q-value and the reward value in the update. When the reward value is a positive reward, the new Q-value is greater than the old Q-value; when the reward value is a negative penalty, the new Q-value is less than the old Q-value.

9. The method according to claim 1, characterized in that, If transmission fails, a backoff state is entered. The backoff time is calculated based on the time slot length. After the backoff ends, the process returns to the step of obtaining the selected transmission time slot, and the updated local Q table is used as the read object. Through multiple rounds of iteration, different underwater transmitting nodes gradually select different transmission time slots to obtain a distributed time slot self-organization result, including: When the judgment result indicates transmission failure, the underwater transmitting node enters a backoff state. In the backoff state, the underwater transmitting node calculates the backoff time based on the time slot length; the backoff time is equal to a random integer multiplied by the time slot length; the random integer is selected between zero and the backoff window size; The underwater transmitting node suspends transmission during the backoff time and exits the backoff state after the backoff time ends. The underwater transmitting node returns to the step of obtaining the selected transmission time slot, and uses the updated local Q table as the reading object to reselect the transmission time slot for data transmission. Through multiple rounds of feedback on successful and failed transmissions, each underwater transmitting node independently updates its own local Q table during multiple iterations, so that the Q value of successful time slots increases and the Q value of conflicting time slots decreases, thus obtaining a distributed time slot self-organization result.

10. A time-delay-aware underwater acoustic network media access control system based on reinforcement learning, used to implement the method as described in any one of claims 1 to 9, characterized in that, The system includes multiple underwater transmitting nodes and at least one aggregation node; Each of the underwater transmitting nodes includes: The time slot division unit is used to calculate the maximum one-way propagation delay based on the underwater acoustic propagation speed in the underwater acoustic channel and the maximum communication distance between the underwater transmitting node and the sinking node, and to set the time slot length and ACK waiting timeout based on the maximum one-way propagation delay; and to divide the continuous communication time into several transmission frames, and to divide each transmission frame into multiple transmission time slots according to the time slot length, so as to obtain the optional action space of each underwater transmitting node; The Q-table storage unit is used to independently maintain a local Q-table and store the Q-values ​​corresponding to each transmission time slot; the Q-values ​​are used to represent the historical transmission effect of the transmission time slot. The time slot selection unit is used to read the local Q table and select a transmission time slot according to the Q value when there is data to be transmitted, so as to obtain the selected transmission time slot. The underwater acoustic transmission unit is used to transmit data packets within the selected transmission time slot; The ACK receiving unit is used to wait for an ACK acknowledgment packet from the aggregation node before the ACK waiting timeout period expires, and to obtain the ACK receiving status. The timeout determination unit is used to determine whether the transmission was successful or failed based on the ACK reception status, and to obtain the determination result of whether the transmission was successful or failed. The Q-value update unit is used to update the Q-value corresponding to the transmission time slot in the local Q-table according to the positive reward or negative penalty, so as to obtain the updated local Q-table; The backoff and retransmission unit is used to enter the backoff state when the transmission fails. The backoff time is calculated based on the time slot length. After the backoff ends, the time slot selection unit is retried. The aggregation node includes: A data receiving unit is used to receive the data packets sent by the underwater transmitting node; A data decoding unit is used to determine whether the data packet has been correctly received; An ACK feedback unit is used to return an ACK confirmation packet to the corresponding underwater transmitting node after the data packet has been correctly received.

Images