Underwater acoustic network medium access control method based on Q learning and accurate scheduling
By optimizing the medium access control of underwater acoustic networks using full-duplex operating mode and Q-learning algorithm, the problems of high collision rate and low throughput in underwater acoustic networks are solved, and the performance of underwater acoustic networks with high throughput and high channel utilization is achieved.
Patent Information
- Application Number
- CN202411277728.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-11-22
AI Technical Summary
Underwater acoustic networks suffer from high collision rates, low throughput, and low channel utilization. In particular, convergence is slow when the network is large, and the half-duplex mode used by sensor nodes limits the information transmission rate.
The underwater acoustic sensor node adopts a full-duplex working mode, concurrently schedules data transmission, and utilizes the Q-learning algorithm and precise scheduling to optimize media access control. The Q matrix describes the action priority of the node, and the Q value table is updated to optimize data transmission.
It achieves high throughput and high channel utilization network performance, reduces learning convergence time, and reduces channel space waste and collision probability.
Smart Images

Figure CN119011107A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an underwater acoustic network, and in particular to an underwater acoustic network medium access control method based on Q learning and precise scheduling. Background Art
[0002] The ocean occupies 71% of the earth's surface area. How to effectively utilize and develop the ocean has become the focus of research in various countries. As a key technology in the current marine military, underwater acoustic communication has become one of the important research hotspots.
[0003] As a major research focus of underwater acoustic communication, underwater acoustic sensor networks (hereinafter referred to as "hydroacoustic networks") are widely used in marine environment monitoring, seabed detection, disaster prevention, auxiliary navigation, distributed tactical surveillance, and mine reconnaissance. Hydroacoustic sensor nodes can collect data from the ocean in real time and transmit the data to the aggregation node instantly through the hydroacoustic network. Since the hydroacoustic sensor nodes are battery-powered, it is particularly important to improve the success rate of node data transmission and reduce transmission energy consumption. This is also the focus of the data link layer research in the hydroacoustic network protocol and the research focus of the hydroacoustic network media access control protocol.
[0004] In the research of underwater acoustic network media access control protocol, in response to the problems of low data throughput and low channel utilization in underwater acoustic networks, SHPark et al. (SHPark, PDMitchell and D.Grace, "Reinforcement Learning Based MAC Protocol (UW-ALOHA-Q) for Underwater Acoustic Sensor Networks," IEEE Access, vol. 7, pp. 165531-165542, 2019) adopted the ALOHA-Q protocol based on ground radio sensor networks and adopted asynchronous operation to eliminate the challenges related to underwater time synchronization. At the same time, the channel utilization was increased by reducing the number of time slots per frame, and collision-free scheduling was achieved by incorporating a new random backoff scheme. However, the protocol has the problem of slow convergence when the network scale is large. At present, the combination of underwater acoustic networks and reinforcement learning is mainly focused on the design of underwater routing protocols, and there are few schemes for designing underwater acoustic network media access control protocols using reinforcement learning. In addition, the sensor nodes used in underwater acoustic networks basically adopt half-duplex working mode, which limits the rate of underwater information transmission and has a certain impact on network throughput. Therefore, it is particularly important to combine reinforcement learning to optimize the design of underwater acoustic network media access control protocols. Summary of the invention
[0005] The purpose of the present invention is to address the problems of high collision rate, low throughput and low channel utilization during multi-access of underwater acoustic networks. By adopting an underwater acoustic sensor in full-duplex working mode, concurrent scheduling is first adopted to allow data to be transmitted simultaneously, and then the Q learning algorithm and precise scheduling are used to optimize the design, thereby providing a fast-converging underwater acoustic network medium access control method based on Q learning and precise scheduling to achieve network performance with high throughput and high channel utilization.
[0006] The present invention comprises the following steps:
[0007] 1) Parameter initialization:
[0008] Consider an underwater acoustic network, which includes M underwater acoustic sensor nodes (hereinafter referred to as "nodes") and a sink (hereinafter referred to as "sink"). Both nodes and sinks adopt full-duplex working mode. The nodes are responsible for sensing information from the marine environment, and the sink is responsible for collecting the information data sensed by the nodes.
[0009] The data collection process of the sink has N time slots. To ensure that each node has a time slot to send data to the sink, the number of time slots N can be equal to the number of nodes M. In the Q learning algorithm, a Q matrix of size M×(2N-1) is used to describe the time slot arrangement of the node accessing the sink for data transmission, where the row m (m=1,2,…,M) in the Q matrix represents the node sequence number, and the column n (n=1,2,…,N,N+1,…,2N-1) of the Q matrix represents the action of the node when performing data transmission. When n=N, it means that the node performs data transmission on time; when n=Nk, it means that the node performs data transmission k time slots in advance; when n=N+k, it means that the node delays k time slots. data transmission; therefore, Q(m, Nk) represents the Q value corresponding to the action of node m choosing to send data k time slots in advance, and Q(m, N+k) represents the Q value corresponding to the action of node m choosing to delay sending data k time slots; the larger the value of Q(m, n), the greater the priority of the mth node in choosing to send data |nN| time slots in advance / delay; that is, node m will give priority to the position with the largest Q value in the mth row of matrix Q as its action; if there are multiple largest Q values in the mth row, one of the largest values will be randomly selected as its action; in order to reduce the complexity of node calculation, the mth node only stores the Q value submatrix Q of the row 1×(2N-1) corresponding to itself m , where Q = [Q1; Q2; Q3; ...; Q M ]; Assume that the data frame format and length of each node are the same, the time length of the time slot is also the same, and the time length of the time slot is equal to the maximum propagation delay of underwater sound plus the data frame length; the initial Q value table is a matrix of M×(2N-1);
[0010] 2) When a node starts data transmission, all nodes send data to the sink at the same time; the sink records the transmission success of each time slot, which are success, conflict, and idle.
[0011] 3) After receiving the transmission status of each time slot, the destination will return a failed transmission signal NACK to one of the conflicting nodes in the next time slot of the conflicting time slot; the NACK signal contains the location information of the conflicting time slot;
[0012] 4) After all nodes have completed data transmission, the sink will broadcast a feedback signal ACK to all nodes; the feedback signal contains information about each time slot (success, conflict, and idle); the node updates the Q value table after receiving the feedback signal;
[0013] 5) Q-value table update mechanism design:
[0014] After receiving the feedback signal, the node will perform Q value sub-matrix Q m Update; according to the Q learning formula Q m (m,:)←(1-γ)·Q m (m,:)+γ·R m (m,:) Update the Q value table, where γ is the learning rate, which takes the value (0,1], and the submatrix R m is the size of the submatrix Q m The same reward sub-matrix; the values in the matrix are set as follows:
[0015] ① When node m receives both ACK and NACK signals: subtract the number of positions of the node's conflicting time slot from the number of positions of the idle time slot, and the difference between the two time slots is a time slots; at this time, the reward submatrix R m The element R in (m,:) m (m,N+K+a) is a positive value |Ψ| to ensure that Q m (m, N+K+a) value increases; K represents the number of time slots that this node delays when transmitting data compared to on-time transmission; the reward submatrix R m The other elements in (m,:) are 0;
[0016] ② When node m only receives the ACK signal: reward sub-matrix R m The element R in (m,:) m (m,N+K) is a positive value |Ψ| to ensure that Q m (m,N+K) value increases; reward submatrix R m The other elements in (m,:) are 0;
[0017] 6) Determine whether there are still nodes that receive NACK signals: If all nodes do not receive NACK signals, it means that the learning has ended and the transmission has reached a stable state; if there are still nodes that receive NACK signals, it means that there are still conflicts in the transmission process, and steps 2) to 6) need to be repeated;
[0018] 7) According to the final Q value table obtained through learning, let node m select the action corresponding to the maximum Q value to send data to the destination, and complete the task of allocating the use rights of the underwater acoustic channel medium resources.
[0019] The present invention has the following outstanding advantages:
[0020] 1) In view of the phenomenon that too many feedback signals will waste channel space and increase the probability of conflict when the scale of the underwater acoustic network is large, the present invention adopts a full-duplex working mode and returns the feedback signal after a round of transmission in all nodes, which can effectively save signal space and reduce conflicts.
[0021] 2) In view of the spatiotemporal uncertainty of sensor networks and the inherent low speed of underwater acoustic transmission, all nodes are first scheduled concurrently to transmit data simultaneously, and then precise scheduling is used to effectively avoid conflicts between data packets and improve channel utilization.
[0022] 3) The present invention adopts the Q learning algorithm and sets the actions in the Q learning algorithm to be sent in advance or delayed, which can effectively reduce the convergence time of learning and improve the overall network throughput of the underwater acoustic sensor network. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 This is a topology diagram of an underwater acoustic network according to the method for underwater acoustic network medium access control based on Q learning and precise scheduling of the present invention.
[0024] Figure 2 This is an example of Q value change of the underwater acoustic network media access control method based on Q learning and precise scheduling of the present invention.
[0025] Figure 3 This is a schematic diagram of the node transmission process of the underwater acoustic network medium access control method based on Q learning and precise scheduling of the present invention.
[0026] Figure 4 This is a flowchart of the node transmission process of the underwater acoustic network medium access control method based on Q learning and precise scheduling of the present invention.
[0027] Figure 5 This is a graph showing how the network throughput of the underwater acoustic network media access control method based on Q learning and precise scheduling changes with the number of transmission rounds.
[0028] Figure 6This is a graph showing how the channel utilization of the underwater acoustic network medium access control method based on Q learning and precise scheduling varies with the number of transmission rounds. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the following embodiments will further illustrate the present invention in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0030] The embodiment of the present invention includes the following steps:
[0031] 1) Consider an underwater acoustic topology network, such as Figure 1 As shown, it includes M = 20 sensor nodes (hereinafter referred to as "nodes") and a sink (hereinafter referred to as "sink"). Both the node and the sink adopt full-duplex working mode. The node is responsible for sensing information from the marine environment, and the sink is responsible for collecting the information data sensed by the node.
[0032] The data collection process of the sink has N = 20 time slots. To ensure that each node has a time slot to send data to the sink, the number of time slots can be equal to the number of nodes. In the Q learning algorithm, a Q matrix of size 20×(2×20-1) is used to describe the time slot arrangement of the node accessing the sink for data transmission, where the row m (m = 1, 2, ..., M) in the Q matrix represents the node sequence number, and the column n (n = 1, 2, ..., N, N + 1, ..., 2N-1) of the Q matrix represents the action of the node when performing data transmission. When n = N, it means that the node transmits data on time; when n = Nk, it means that the node transmits data k time slots in advance; when n = N + k, it means that the node delays k time slots for data transmission. Therefore, Q (m, Nk) represents the Q value corresponding to the action of node m choosing to send data k time slots in advance, and Q (m, N + k) represents the Q value corresponding to the action of node m choosing to delay k time slots for data transmission. The larger the value of Q(m, n), the higher the priority of the mth node to choose to send data in advance / delay |nN| time slots. That is, node m will give priority to the position with the largest Q value in the mth row of matrix Q as its action; if there are multiple largest Q values in the mth row, it will randomly select one of the largest values as its action. In order to reduce the complexity of node calculation, the mth node only stores the Q value submatrix Q of the row 1×(2N-1) corresponding to itself m , where Q = [Q1; Q2; Q3; ...; Q M ]. Assume that the data frame format and length of each node are the same, the time slot length is also the same, and the time slot length is equal to the maximum propagation delay of underwater sound plus the data frame length. The initial Q value table is a 20×(2×20-1) matrix.
[0033] 2) When a node starts data transmission, all nodes send data to the sink at the same time. The sink records the transmission success of each time slot: success, conflict, and idle.
[0034] In this embodiment, conflicts occur at the 10th, 13th, 14th, and 18th time slots, the 2nd, 5th, 7th, 8th, 15th, and 17th time slots are idle, and the other time slots are successfully transmitted.
[0035] 3) After receiving the transmission status of each time slot, the sink will return a transmission failure signal NACK to one of the conflicting nodes in the next time slot of the conflicting time slot. The NACK signal contains the location information of the conflicting time slot.
[0036] 4) After all nodes have completed data transmission, the sink will broadcast a feedback signal ACK to all nodes. The feedback signal contains information about each time slot (success, conflict, and idle). After receiving the feedback signal, the node updates the Q value table.
[0037] 5) Q-value table update mechanism design:
[0038] After receiving the feedback signal, the node will perform Q value sub-matrix Q m Update. According to the Q learning formula Q m (m,:)←(1-γ)·Q m (m,:)+γ·R m (m,:) Update the Q value table, where γ is the learning rate, set to 0.2, and the matrix R m is the size of the submatrix Q m The same reward sub-matrix. The specific settings of the values in the matrix are as follows:
[0039] ① When node m receives both an ACK signal and a NACK signal: subtract the number of positions of the node's conflicting time slot from the number of positions of the idle time slot, and the difference between the two time slots is a time slots. At this time, the reward submatrix R m The element R in (m,:) m (m,N+K+a) is a positive value |Ψ| to ensure that Q m (m,N+K+a) value increases. K represents the number of time slots that this node delays when transmitting data compared to the on-time transmission. The reward submatrix R m The other elements in (m,:) are 0;
[0040] ② When node m only receives the ACK signal: reward sub-matrix R m The element R in (m,:) m (m,N+K) is a positive value |Ψ| to ensure that Q m (m,N+K) value increases. Reward sub-matrix R mThe other elements in (m,:) are 0.
[0041] In this embodiment, nodes A and B collide at the 10th time slot, and the 2nd, 5th, 7th, 8th, 15th, and 17th time slots are idle. Node A receives both the NACK signal and the ACK signal from the destination, while node B only receives the ACK signal from the destination. Therefore, for node A, the difference between the idle time slot and the conflicting time slot is -8, -5, -3, -2, +5, and +7, respectively. Therefore, R in the reward submatrix R of node A is A (A,20-8), R A (A,20-5),R A (A,20-3),R A (A,20-2), R A (A,20+5), R A (A,20+7) is a positive value |Ψ|. For node B, since only the ACK signal is received and node B transmits data on time, the R in the reward submatrix R of node B is B (B, 20) is a positive value |Ψ|. In this embodiment, |Ψ| is set to 5. Similarly, the reward submatrix R of other nodes can also be derived from this. Then according to the Q learning formula Q m (m,:)←(1-γ)·Q m (m,:)+γ·R m (m,:) Update the Q value table, where γ is the learning rate, set to 0.2. Figure 2 The updating process of the sub-matrix of node A is described.
[0042] 6) Determine whether there are still nodes that receive NACK signals. If all nodes do not receive NACK signals, it means that the learning has ended and the transmission has reached a stable state; if there are still nodes that receive NACK signals, it means that there are still conflicts in the transmission process, and steps 2) to 6) need to be repeated.
[0043] 7) According to the final Q value table obtained through learning, let node m select the action corresponding to the maximum Q value to send data to the destination, and complete the task of allocating the use rights of the underwater acoustic channel medium resources.
[0044] Figure 3 This is a schematic diagram of the node transmission process of the underwater acoustic network medium access control method based on Q learning and precise scheduling described in the present invention, corresponding to the above steps 2), 3), 4) and 7). Figure 4 Q is a flowchart of the node transmission process of the underwater acoustic network medium access control method based on Q learning and precise scheduling described in the present invention, corresponding to the above steps 1), 5) and 6).
[0045] The feasibility of the method of the present invention is verified by computer simulation below.
[0046] 20 nodes are randomly distributed within the effective communication range of 10 km from the destination. The simulation parameters are set as follows: the underwater sound speed is 1500m / s, the transmission rate of the node and the destination is 2000bit / s, the data frame and the feedback signal format are the same, and the length is set to 256 bits. The time slot length is the same, set to 0.15s. The simulation time is 1h. The comparison algorithms are ALOHA and UW-ALOHA-Q.
[0047] The following is an analysis of the simulation results of the method described in the present invention.
[0048] 1) Network throughput comparison
[0049] Figure 5 This is a comparison chart of the network throughput of the underwater acoustic network medium access control method based on Q learning and precise scheduling according to the present invention as the transmission rounds change. Figure 5 It can be seen that the network throughput of the underwater acoustic network media access control method based on Q learning and precise scheduling of the present invention is significantly higher than that of the other two comparison algorithms. The present invention sets the action in the Q learning algorithm to be sent in advance or delayed, which effectively reduces the convergence time of learning and improves the overall network throughput of the underwater acoustic sensor network.
[0050] 2) Channel Utilization Comparison
[0051] Figure 6 This is a comparison chart of the channel utilization rate of the underwater acoustic network medium access control method based on Q learning and precise scheduling according to the transmission round. Figure 6 It can be seen that the channel utilization rate of the underwater acoustic network medium access control method based on Q learning and precise scheduling of the present invention is higher than that of the other two comparison algorithms in the early transmission process. With the increase of transmission rounds, the underwater acoustic network medium access control method based on Q learning and precise scheduling of the present invention can converge quickly, and in view of the spatial uncertainty of the sensor network, all nodes are allowed to adopt concurrent scheduling and transmit data at the same time, which can effectively avoid conflicts between data packets and improve channel utilization.
[0052] By comparing the above three simulation schemes, it can be seen that the underwater acoustic network medium access control method based on Q learning and precise scheduling has higher throughput and higher channel utilization. On the one hand, the present invention adopts a full-duplex working mode and returns the feedback signal after all nodes have completed a round of transmission, which can effectively save signal space and reduce conflicts. On the other hand, the designed precise scheduling method can utilize time and space uncertainty to reduce conflicts between nodes and improve channel utilization.
[0053] The above embodiments are only preferred embodiments of the present invention and cannot be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the present invention.
Claims
1. A method for underwater acoustic network medium access control based on Q learning and precise scheduling, characterized in that The following steps are involved: 1) Parameter initialization: Consider an underwater acoustic network, which contains M nodes and one sink. Both nodes and sinks use full-duplex working mode. The nodes are responsible for sensing information from the ocean environment, and the sinks are responsible for collecting the information data sensed by the nodes. The data collection process of the sink has N time slots. To ensure that each node has a time slot to send data to the sink, the number of time slots N is equal to the number of nodes M. In the Q learning algorithm, a Q matrix of size M×(2N-1) is used to describe the time slot arrangement of the node accessing the sink for data transmission, where the row m (m=1,2,…,M) in the Q matrix represents the node sequence number, and the column n (n=1,2,…,N,N+1,…,2N-1) of the Q matrix represents the action of the node when performing data transmission. When n=N, it means that the node performs data transmission on time; when n=Nk, it means that the node performs data transmission k time slots in advance; when n=N+k, it means that the node delays k time slots for data transmission. row data transmission; therefore, Q(m, Nk) represents the Q value corresponding to the action of node m choosing to send data k time slots in advance, and Q(m, N+k) represents the Q value corresponding to the action of node m choosing to delay sending data k time slots; the larger the value of Q(m, n), the greater the priority of the mth node in choosing to send data |nN| time slots in advance / delay; that is, node m will give priority to the position with the largest Q value in the mth row of matrix Q as its action; if there are multiple largest Q values in the mth row, one of the largest values will be randomly selected as its action; in order to reduce the complexity of node calculations, the mth node only stores the Q value submatrix Q of the row 1×(2N-1) corresponding to itself m , where Q = [Q1; Q2; Q3; ...; Q M ]; Assume that the data frame format and length of each node are the same, the time length of the time slot is also the same, and the time length of the time slot is equal to the maximum propagation delay of underwater sound plus the data frame length; the initial Q value table is a matrix of M×(2N-1); 2) When a node starts data transmission, all nodes send data to the sink at the same time; the sink records the transmission status of each time slot, which are success, conflict and idle. 3) After receiving the transmission status of each time slot, the destination will return a failed transmission signal NACK to one of the conflicting nodes in the next time slot of the conflicting time slot; the NACK signal contains the location information of the conflicting time slot; 4) After all nodes have completed data transmission, the sink will broadcast a feedback signal ACK to all nodes; the feedback signal contains information about each time slot, including success, conflict, or idle; the node updates the Q value table after receiving the feedback signal; 5) Q value table update mechanism design: After receiving the feedback signal, the node performs Q value sub-matrix Q m Updates; According to the Q-learning formula Q m (m,:)←(1-γ)·Q m (m,:)+γ·R m (m,:) Update the Q value table, where γ is the learning rate, which takes the value (0,1], and the submatrix R m is the size of the submatrix Q m The same reward sub-matrix; the values in the matrix are set as follows: ① When node m receives both ACK and NACK signals: subtract the number of positions of the node's conflicting time slot from the number of positions of the idle time slot, and the difference between the two time slots is a time slots; at this time, the reward submatrix R m The element R in (m,:) m (m,N+K+a) is a positive value |Ψ| to ensure that Q m (m, N+K+a) value increases; K represents the number of time slots that this node delays when transmitting data compared to on-time transmission; the reward submatrix R m The other elements in (m,:) are 0; ② When node m only receives the ACK signal: reward sub-matrix R m The element R in (m,:) m (m,N+K) is a positive value |Ψ| to ensure that Q m (m,N+K) value increases; reward submatrix R m The other elements in (m,:) are 0; 6) Determine whether there are still nodes that receive NACK signals: If all nodes do not receive NACK signals, it means that the learning has ended and the transmission has reached a stable state; if there are still nodes that receive NACK signals, it means that there are still conflicts in the transmission process, and steps 2) to 6) need to be repeated; 7) According to the final Q value table obtained through learning, let node m select the action corresponding to the maximum Q value to send data to the destination, and complete the task of allocating the use rights of the underwater acoustic channel medium resources.
Citation Information
Cited By
Low-cost underwater acoustic communication medium access control method and device based on reinforcement learning
CN120583539A
Underwater topology control and channel selection method based on hierarchical reinforcement learning
CN120729721A
