Underwater unmanned cluster formation time division-space division multiple access protocol based on Q learning

By adopting time division multiple access and space division multiple access protocols in AUV cluster formations and dynamically adjusting the beam angle in combination with Q learning technology, the throughput and robustness of the AUV cluster formation MAC protocol in the current technology in dynamic marine environments is solved. The problem of insufficient sex enables high throughput and high robustness multiple access.

CN119012360APending Publication Date: 2024-11-22XIAMEN UNIV +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411278982.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-12
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The MAC protocol of existing AUV cluster formations is difficult to make full use of the advantages of directional transceiver matrix, and optimizes the beam angle with the Q learning technology in reinforcement learning, resulting in insufficient network throughput and robustness in dynamic marine environments.

Method used

Time division multiple access and space division multiple access protocols are adopted, and the beam angle is dynamically adjusted in combination with Q learning technology. Directed transceiver base arrays are used to improve network throughput and robustness in the initialization and navigation stages of AUV cluster formation, and adapt to the dynamics of the marine environment. change.

Benefits of technology

It realizes the high throughput and high robust multi-access access protocol of AUV cluster formations, which can effectively adapt to topological changes in dynamic marine environments, and improves the stability and efficiency of the network.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119012360A_ABST
    Figure CN119012360A_ABST
Patent Text Reader

Abstract

The invention discloses an underwater unmanned cluster formation time division-space division multiple access protocol based on Q learning, and relates to underwater unmanned cluster formation. In an underwater unmanned cluster formation, each node is equipped with a directional transceiver array and a multi-path transceiver. In an initialization stage, a time division multiple access protocol is adopted, a master node broadcasts and schedules time slot information in an omnidirectional mode, and slave nodes broadcast feedback information in a specified time slot to complete network access. In the sailing stage, the slave node transmits data to the master node, a space division multiple access protocol is adopted, and the master node adjusts the beam direction according to the position information of the slave node in a directional mode to realize multi-point simultaneous communication. The Q learning technology is introduced, the beam angle is dynamically adjusted, it is ensured that space division multiple access adapts to dynamic changes of the marine environment, and the communication efficiency is effectively improved. And a multiple access scheme with high network throughput and high robustness is provided for the underwater unmanned cluster.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an underwater unmanned cluster formation, and in particular to a Q-learning-based underwater unmanned cluster formation time-space division multiple access protocol. Background Art

[0002] As autonomous underwater vehicle (AUV) technology matures, multiple AUVs collaborating in the form of clusters to perform tasks has become an inevitable development direction. Underwater unmanned cluster formations (hereinafter referred to as "AUV cluster formations") have the characteristics of strong mobility, large range of activities, and high operating efficiency. They can undertake complex marine resource exploration, reconnaissance and surveillance, and anti-submarine tasks. In underwater data collection tasks, the optimization design of the multiple access (MAC) protocol of the AUV cluster formation is crucial.

[0003] AUV nodes are equipped with hydroacoustic sensor nodes. In each working stage, cluster formation networking faces the characteristics of long propagation delay and spatiotemporal uncertainty of the ocean environment, which brings great challenges to the design of MAC protocol. M.Liu et al. proposed a scheduling-based adaptive broadcast multiple access (AB-MAC) protocol to improve the timeliness and reliability of information exchange between AUVs. In an incompletely centralized manner, the control node arranges the transmission of all nodes according to the updated network topology to avoid packet collision and minimize the frame length. T.Qiu et al. proposed an optimal broadcast scheduling algorithm for multi-AUV acoustic communication network, which uses the location information of AUV to adjust the broadcast order and schedule to achieve the shortest possible conflict-free broadcast cycle in the broadcast network under the current node distribution, and obtains a shorter broadcast cycle and higher channel utilization than the traditional time division multiple access protocol. However, the form of broadcast communication in a single time slot only allows the transmission of a single data packet, which is inefficient. M. Rahmati proposed a MAC protocol based on the probability space division multiple access (PSDMA) method, using directional transducers for multiple access, and optimizing the beam width of the transmitting antenna through a two-step estimation method (rough estimation and unscented Kalman filter) to achieve communication between AUVs, effectively improving the network throughput, but the complexity of beam width optimization is relatively high.

[0004] In summary, the current research on MAC protocols for AUV cluster formations rarely makes full use of the advantages of directional transceiver arrays and combines Q-learning technology in reinforcement learning to optimize the beam angle. It is still difficult to adapt to the dynamic ocean environment, and the network throughput and robustness need to be further improved. Summary of the invention

[0005] The purpose of the present invention is to use the advantages of directional transceiver arrays to meet the access needs of AUV clusters during the formation initialization and navigation stages, and respectively adopt time division multiple access and space division multiple access protocols to improve the throughput and robustness of the network; in addition, the present invention introduces Q learning technology to dynamically adjust the beam angle of the space division multiple access protocol to adapt to the dynamic changes of the marine environment; the present invention can provide a high-throughput, high-robustness dynamic MAC protocol suitable for AUV cluster formations.

[0006] The Q-learning-based underwater unmanned cluster formation time-division-space division multiple access protocol of the present invention comprises the following steps:

[0007] 1) AUV cluster formation scene setting:

[0008] Consider an AUV cluster formation, which includes a master node AUV (hereinafter referred to as the "master node") and N slave node AUVs (hereinafter referred to as the "slave nodes"); the maximum transmission range of each node is equal, represented by R; each slave node has an independent number n (n = 1, 2, ..., N); during the operation of the AUV cluster formation, the slave node is responsible for sensing information in the ocean environment and transmitting it to the master node; the master node is located at the center of the formation and is responsible for sending formation control instructions and collecting information data sensed by the slave nodes;

[0009] Each master node and slave node is equipped with a directional transceiver array and M transceivers, M ≥ N. The directional transmission array contains K beams (K = β·M, that is, each transceiver corresponds to β beams). A certain transceiver is denoted as transceiver m (m = 1, 2, ..., M), and its corresponding beams are denoted as beams (k-β+1), (k-β+2), (k-β+3), ..., k, where k = β·m. When the transceiver is working, in order to suppress interference from other directions, one of the beams will be activated as needed; the directional transmission array switches as needed. Omnidirectional working mode and directional working mode: Omnidirectional working mode can be used to collect external information and broadcast formation control information, which takes a long time and has a low throughput; directional working mode can effectively suppress interference in the direction outside the beam, can receive information from multiple nodes at the same time, and has a large throughput; the switching method of the working mode is shown in steps 2) and 3); in actual design, for convenience, M=N can be taken, that is, the number of transceivers of the master node is equal to the number of slave nodes, and the master node can realize the function of simultaneously transmitting and receiving different data to N slave nodes in a directional manner;

[0010] 2) MAC protocol in the initialization phase of AUV cluster formation:

[0011] In the initialization phase of the AUV cluster formation, since the master node is completely blind to external factors such as the position, distance, and ocean channel environment of the slave nodes, the time division multiple access (TDMA) protocol is required to obtain more accurate information at the expense of time, so as to facilitate the completion of the network access task of the slave nodes: the master node directional transceiver array adopts an omnidirectional working mode, calculates and broadcasts TDMA scheduling time slot information to each slave node; assuming that the nodes have completed time synchronization, for slave node n, its scheduling time slot starts at T n :

[0012] T n =n·T (1)

[0013] Among them, T n represents the start time of the scheduling time slot of slave node n, n represents the number of slave node n, and T represents the length of a single time slot; to ensure that each slave node can complete the task of accessing the network, the time slot length T can be expressed by the following formula:

[0014] T=R / v water +t packet (2)

[0015] Among them, v water is the speed of sound in water, t packet represents the time required to send a data packet, and R represents the maximum transmission range of each node; when the slave node n receives the scheduled time slot data packet from the master node, T n , and at T n Feedback information is broadcasted at all times; after receiving feedback information from each slave node, the master node obtains information such as the distance, position, and ocean channel environment of the slave node, and completes the initialization of the AUV cluster formation;

[0016] 3) MAC protocol for AUV cluster formation navigation phase:

[0017] During the navigation phase, the AUV cluster formation’s task is to collect underwater information data. During this navigation phase, data is transmitted from the slave node to the master node. The master node sends a message every T c The formation control information (including movement speed and direction) is broadcast once in omnidirectional mode to ensure that the cluster formation can maintain its formation. crepresents the time interval of the master node's omnidirectional mode broadcast; since the positions of the master and slave nodes are relatively fixed at this stage, the transmission of the slave nodes can adopt the space division multiple access (SDMA) protocol according to the relative positions of the nodes: the master node's directional transceiver array adopts a directional working mode, and according to the slave node position information obtained in step 2), adjusts each transceiver so that a beam corresponding to each transceiver points to a slave node, and activates the beam through the beam forming technology; since M = N, the slave node corresponding to transceiver m can be recorded as slave node m, and after one of the beams (k-β+1), (k-β+2), (k-β+3), ..., k of transceiver m is aligned with slave node m, the master node can complete information exchange with slave node m through the beam; since the master node has M transmitters in total, it can send different data to M slave nodes in different directions at the same time without wasting the waiting time of the slave nodes;

[0018] During this navigation phase, the master node will record the received signal-to-noise ratio (SNR) while receiving information sent by the slave node. DT is used to represent the signal detection threshold. It is believed that when SNR ≥ DT, the physical layer can be 100% error-free, that is, the receiving end can receive 100% of the data packets sent by the sending end. Due to dynamic changes in the ocean environment such as water currents, the position of the slave node may drift, resulting in deviation from the original beam coverage. It is necessary to adjust the beam to realign the slave node. For transceiver m, when slave node m is within the beam coverage, SNR ≥ DT. When SNR < DT, it means that slave node m deviates from the beam coverage, and the beam angle adjustment mechanism based on Q learning can be started:

[0019] 3.1) Q value table initialization: Generate a Q value table with A rows and B columns. The number of rows and columns of the Q value table satisfies the following formula:

[0020]

[0021] Among them, the value A is the number of rows in the Q value table, and it is also the maximum number of iterations. In order to prevent the algorithm from occupying too much storage space, the Q value table is set to a limited number of rows. When the Q value table is filled, the Q value table is cleared. If the topology structure changes drastically, A can take a larger value; the value B represents the number of beams corresponding to the transceiver m; the row a (a = 1, 2, ..., A) of the Q value table represents the number of time slots experienced after the node position leaves the beam coverage range (excluding the time slot occupied by the master node broadcasting the formation control information), and the column b (b = 1, 2, ..., A) of the Q value table represents the number of time slots experienced after the node position leaves the beam coverage range (excluding the time slot occupied by the master node broadcasting the formation control information). …,B) indicates adjusting the activation beam (k-β+b) of transceiver m; therefore, Q(a,b) indicates the Q value corresponding to the action of adjusting the activation beam (k-β+b) of transceiver m in the ath time slot; the larger the value of Q(a,b), the greater the priority of adjusting the activation beam (k-β+b) of transceiver m in the ath time slot; that is, in the ath time slot, transceiver m will preferentially select the beam with the largest Q value for activation; if multiple largest Q values ​​appear in the ath row, transceiver m randomly selects one of the beams for activation;

[0022] 3.2) Q value update mechanism: In the ath time slot, after the transceiver m activates the beam (k-β+b) according to the Q value at this time, the reward value r is determined according to the received signal-to-noise ratio SNR:

[0023]

[0024] Where r represents the reward value, SNR represents the received signal-to-noise ratio, and DT represents the signal detection threshold. If SNR ≥ DT, it means that the beam activated by transceiver m can normally receive information from slave node m, and a positive reward is given. Otherwise, it means that the beam range activated by transceiver m still cannot cover slave node m, and a negative reward is given. The Q value of the a+1th row is updated as follows:

[0025] Q(a+1,b)←(1-γ)Q(a,b)+γ·r (5)

[0026] Among them, γ is the learning rate, the value range is (0,1], r represents the reward value;

[0027] When initializing the Q value table, Q(1,b) is also initialized according to formulas (4) and (5) (Q(0,b)=0); when the Q value table is full, the beam corresponding to the maximum Q value is selected for activation, a round of update ends, the Q value table is cleared, and the beam angle adjustment mechanism based on Q learning ends; if SNR≥DT at this time, it means that the current beam range can cover the slave node m; if SNR<DT, the slave node m may leave the coverage range of all beams corresponding to the transceiver m, and the TDMA protocol in step 2) is used to re-initialize the AUV cluster formation;

[0028] 3.3) Q value update of multiple transceivers: In the present invention, each transceiver has an independent Q value table. If the received signal-to-noise ratio SNR of multiple transceivers is less than the signal detection threshold DT, the beam angle adjustment based on Q learning of these transceivers can be carried out simultaneously, and usually do not interfere with each other; but when a transceiver ends the beam angle adjustment mechanism based on Q learning, if the TDMA protocol in step 2) needs to be selected to re-initialize the AUV cluster formation, the beam angle adjustment of all other transceivers will also end, and the Q value table will be cleared.

[0029] The present invention has the following outstanding advantages:

[0030] 1) According to the different access requirements of the AUV cluster formation initialization phase and navigation phase, TDMA and SDMA protocols are used respectively to give full play to the advantages of directional transceiver array to obtain higher robustness and network throughput;

[0031] 2) Aiming at the dynamic topology problem of AUV networking during navigation, a beam angle adjustment mechanism based on Q learning is used to dynamically adjust the transceiver beam coverage, which can effectively adapt to the dynamically changing marine environment; during the beam adjustment of a single transceiver, the normal communication of other transceivers is not affected, thereby improving the working efficiency of the access protocol;

[0032] 3) When the AUV cluster formation topology changes significantly, the original transceiver beam range cannot cover the slave node, and the TDMA protocol is used to re-initialize the cluster formation to ensure the high robustness of the protocol. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a scenario diagram of the underwater unmanned cluster formation time-space division multiple access protocol based on Q learning of the present invention, including 1 master node, 4 slave nodes, 4 transceivers and 16 beams.

[0034] Figure 2 The present invention is a schematic diagram of single-channel transceiver communication of the underwater unmanned cluster formation time division-space division multiple access protocol based on Q learning.

[0035] Figure 3 This is a diagram of TDMA and SDMA time slot arrangement in the time division-space division multiple access protocol of the underwater unmanned cluster formation based on Q learning of the present invention.

[0036] Figure 4 This is a flow chart of the underwater unmanned cluster formation time-space division multiple access protocol based on Q learning of the present invention.

[0037] Figure 5 This is an initial Q value table of the underwater unmanned cluster formation time-space division multiple access protocol based on Q learning of the present invention.

[0038] Figure 6 The present invention is a schematic diagram of updating the Q value table in the time-space division multiple access protocol of the underwater unmanned cluster formation based on Q learning.

[0039] Figure 7 This is a comparison chart of the network throughput of the underwater unmanned cluster formation time-space division multiple access protocol based on Q learning of the present invention and the comparison scheme.

[0040] Figure 8 This is a comparison chart of the number of data packets transmitted in each time slot of the underwater unmanned cluster formation time division-space division multiple access protocol based on Q learning of the present invention and the comparison scheme. DETAILED DESCRIPTION

[0041] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] The embodiment of the present invention includes the following steps:

[0043] 1) AUV cluster formation scene setting:

[0044] Consider an AUV cluster formation, which includes a master node and 4 slave nodes; the maximum transmission range of each node is equal, set to 1.5km; each slave node has an independent number n (n = 1, 2, 3, 4). During the operation of the AUV cluster formation, the slave node is responsible for sensing information in the ocean environment and transmitting it to the master node; the master node is located at the center of the formation, responsible for sending formation control instructions and collecting information data sensed by the slave nodes. All slave nodes are within the communication range of the master node. The scene diagram is as follows Figure 1 As shown;

[0045] Each master node and slave node is equipped with a directional transceiver array and 4 transceivers. The directional transmission array contains 16 beams (each transceiver corresponds to β = 4 beams). For example, for transceiver m = 1, its corresponding beams are k = 1, k = 2, k = 3, k = 4, such as Figure 2 As shown; when the transceiver is working, in order to suppress interference in other directions, one of the beams will be activated as needed. The directional transmitting array can switch between the omnidirectional working mode and the directional working mode as needed: the omnidirectional working mode can be used to collect external information and broadcast formation control information, which takes a long time and has a low throughput; the directional working mode can effectively suppress interference in directions outside the beam, can receive information from multiple nodes at the same time, and has a high throughput; the working mode switching method is shown in step 2) and step 3). In the embodiment, M=N=4, the number of transceivers of the master node is equal to the number of slave nodes, and the master node is equipped with 4 transceivers, which can realize the function of the master node simultaneously transmitting and receiving different data to 4 slave nodes at most;

[0046] 2) MAC protocol in the initialization phase of AUV cluster formation:

[0047] In the initialization phase of the AUV cluster formation, since the master node is completely blind to external factors such as the position, distance, and ocean channel environment of the slave nodes, the time division multiple access (TDMA) protocol needs to be used to obtain more accurate information at the expense of time, so as to facilitate the completion of the slave node access network task: the master node directional transceiver array adopts omnidirectional working mode, calculates and broadcasts TDMA scheduling time slot information to each slave node. Assuming that the nodes have completed time synchronization, for slave node n, its scheduling time slot starts at T n :

[0048] T n =n·T (1)

[0049] Among them, T n represents the start time of the scheduling time slot of slave node n, n represents the number of slave node n, and T represents the length of a single time slot. To ensure that each slave node can complete the task of accessing the network, the time slot length T can be expressed by the following formula:

[0050] T=R / v water +t packet (2)

[0051] Among them, v water represents the speed of sound in water, set to 1500m / s, t packet It represents the time required to send a data packet, which is set to 0.1s. When the slave node n receives the scheduled time slot data packet from the master node, it can obtain T n , and at T n Broadcast feedback information at all times; for example, after receiving the scheduling time slot data packet from the master node, slave node 1 obtains the start time of its own scheduling time slot The feedback information is then broadcast at this time. After receiving the feedback information from each slave node, the master node obtains information such as the distance, direction, and ocean channel environment of the slave node, and completes the initialization of the AUV cluster formation; the time slot arrangement of the TDMA protocol is as follows Figure 3 As shown;

[0052] 3) MAC protocol for AUV cluster formation navigation phase:

[0053] During the navigation phase, the task of the AUV cluster formation is to collect underwater information data. This phase mainly involves transmitting data from the nodes to the master node. The master node transmits data every T c The formation control information (including movement speed and direction) is broadcast once in omnidirectional mode to ensure that the cluster formation can maintain its formation. cIndicates the time interval of the master node's omnidirectional mode broadcast. Since the positions of the master and slave nodes are relatively fixed at this stage, the transmission of the slave nodes can adopt the spatial division multiple access (SDMA) protocol according to the relative positions of the nodes: the master node directional transceiver array adopts a directional working mode, and according to the slave node position information obtained in step 2), adjusts each transceiver so that a beam corresponding to each transceiver points to a slave node, and activates the beam through beamforming technology to achieve efficient data transmission.

[0054] Assume that the slave node corresponding to transceiver 1 is slave node 1, and the relative positions of the two are as follows: Figure 2 As shown, after the transceiver 1 activates the beam 3 aimed at the slave node 1, the master node can complete the information exchange with the slave node 1 through the beam. Since the master node has 4 transmitters, it can send different data to 4 slave nodes in different directions at the same time without wasting the waiting time of the slave nodes. In the embodiment, it is assumed that T c =4T, making the SDMA frame length equal to the TDMA frame length, and the time slot arrangement is as follows Figure 3 shown.

[0055] During this stage, while receiving information sent by the slave node, the master node also records the received signal-to-noise ratio SNR. DT represents the signal detection threshold. It is considered that when SNR ≥ DT, the physical layer can be 100% error-free, that is, the receiving end can receive 100% of the data packets sent by the sending end. Due to dynamic changes in the ocean environment such as water currents, the location of the slave node may drift, resulting in deviation from the original beam coverage. It is necessary to adjust the beam to realign the slave node. For transceiver 1, when slave node 1 is within the beam coverage, SNR ≥ DT; when SNR < DT, it means that slave node 1 deviates from the beam coverage, and the beam angle adjustment mechanism based on Q learning can be activated. The process is as follows: Figure 4 As shown:

[0056] 3.1) Q value table initialization: Generate a Q value table with A rows and B columns. The number of rows and columns of the Q value table satisfies the following formula:

[0057]

[0058] Among them, the value of A is the number of rows of the Q - value table and also the maximum number of iterations. To prevent the algorithm from occupying too much storage space, the Q - value table is set to have a limited number of rows. When the Q - value table is filled, it is cleared. If the topological structure changes violently, a larger value can be taken for A; the value of B represents the number of beams corresponding to transceiver m. The row a (a = 1, 2, …, A) of the Q - value table represents the number of time slots experienced after the node position leaves the beam coverage range (excluding the time slot occupied by the master node broadcasting formation control information). The column b (b = 1, 2, …, B) of the Q - value table represents adjusting the active beam (k - β + b) of transceiver m. Therefore, Q(a, b) represents the Q - value corresponding to the action of adjusting the active beam (k - β + b) of transceiver m in the a - th time slot; the larger the value of Q(a, b), the higher the priority of adjusting the active beam (k - β + b) of transceiver m in the a - th time slot; that is, in the a - th time slot, transceiver m will preferentially select the beam with the largest Q - value to activate; if there are multiple maximum Q - values in the a - th row, transceiver m randomly selects one of the beams to activate; in the embodiment, since β = 4, a Q - value table with 4 rows and 4 columns can be generated. For transceiver 1, its initial Q - value table is as Figure 5 shown;

[0059] 3.2) Q - value update mechanism: In the a - th time slot, after transceiver m activates the beam (k - β + b) according to the Q - value at this time, it judges the value of the reward r according to the obtained received signal - to - noise ratio SNR:

[0060]

[0061] If SNR≥DT, it means that the beam activated by transceiver m at this time can normally receive information from slave node m, and a positive reward is given, indicating that the current beam is effective; on the contrary, if SNR < DT, it means that the beam range activated by transceiver m still cannot cover slave node m, and a negative reward is given, indicating that the beam needs to be adjusted. The Q - value update of the (a + 1) - th row is as follows:

[0062] Q(a + 1, b)←(1 - γ)Q(a, b)+γ·r (5)

[0063] Among them, γ is the learning rate, and its value range is (0, 1]. In the embodiment, the value is 0.4.

[0064] As Figure 6 shown, taking transceiver 1 as an example, assuming that slave node 1 leaves the coverage range of beam 3 and the beam angle needs to be adjusted:

[0065] a. Assume that node 1 moves to the coverage of beam 4: Initially, since beam 3 is activated and SNR < DT, Q(1,3) is initialized to γ·r = 0.4·(-1) = -0.4. At this time, Q(1,1) = Q(1,2) = Q(1,4) = 0 is the maximum Q value, and transceiver 1 randomly selects one of the beams to activate; assuming that beam 1 is selected for activation, the result is still SNR < DT, and Q(2,1) is updated to -0.4; then, beams 2 and 4 with the maximum Q value are randomly selected. Assuming that beam 4 is selected for activation, the result is SNR ≥ DT, and Q(3,4) is updated to 0.4, which is the maximum Q value of the current time slot (the third time slot); beam 4 is still selected for activation in the next time slot. , and SNR≥DT, continue to update Q(4,4), and so on; still get positive rewards because SNR≥DT, continue to select beam 4 with the largest Q value to activate, complete a round of updates, and clear the Q value table (when the Q value table completes a round of updates, if the beam is valid, there is no need to store this knowledge; if the beam is invalid, directly use TDMA to reinitialize, and there is still no need to store this knowledge, so clear the Q value table); at this time SNR≥DT, indicating that the current beam range of transceiver 1 can cover slave node 1.

[0066] b. Assume that the slave node 1 moves out of the coverage range of all beams corresponding to transceiver 1: Initially, Q(1,3) is initialized to γ·r=0.4·(-1)=-0.4. In the following three time slots, transceiver 1 selects beams 1, 2, and 4 for activation and obtains negative rewards. In the fourth time slot, since Q(1,1)=Q(1,2)=Q(1,3)=Q(1,4)=-0.4 are all maximum Q values, transceiver 1 randomly selects one of the beams for activation, completes a round of updates, and clears the Q value table. At this time, SNR<DT, it is considered that the slave node 1 leaves the coverage range of all beams corresponding to transceiver 1, and the TDMA protocol in step 2) is used to reinitialize the AUV cluster formation.

[0067] 3.3) Q value update of multiple transceivers: In the present invention, each transceiver has an independent Q value table. If SNR < DT occurs in multiple transceivers, the beam angle adjustment based on Q learning of these transceivers can be performed simultaneously, and usually do not interfere with each other; but when a transceiver ends the beam angle adjustment mechanism based on Q learning, if the TDMA protocol in step 2) needs to be selected to re-initialize the AUV cluster formation, the beam angle adjustment of all other transceivers will also end, and the Q value table will be cleared.

[0068] The feasibility of the method of the present invention is verified by computer simulation below.

[0069] According to step 1 of the embodiment, a simulation scenario is set, a single data packet size is set to 100 bits, and the comparison algorithm selects traditional TDMA. Starting from the initialization of the AUV cluster formation, a simulation of 100 time slots is performed, in which a situation where the topology changes due to the movement of slave nodes occurs in random time slots.

[0070] The following is an analysis of the simulation results of the method described in the present invention:

[0071] Figure 7 This is a comparison chart of the network throughput of the underwater unmanned cluster formation time-space division multiple access protocol based on Q learning of the present invention and the comparison scheme. Figure 7 It can be seen that the scheme of the present invention has significantly improved the network throughput by giving full play to the advantages of the directional transceiver array. Compared with the traditional TDMA protocol, the network throughput has increased by about 184%. In addition, due to the introduction of the beam angle adjustment mechanism based on Q learning, it can effectively adapt to topology changes. When the network topology structure of the AUV cluster formation changes, the network throughput remains relatively stable overall.

[0072] Figure 8 The figure is a comparison of the number of data packets transmitted in each time slot of the underwater unmanned cluster formation time division-space division multiple access protocol based on Q learning of the present invention and the comparison scheme, which more intuitively demonstrates the role of the beam angle adjustment mechanism based on Q learning. In one frame of SDMA, 4 slave nodes in each time slot can transmit 4 data packets through 4-way transceivers (the last time slot of each frame is the master node transmitting 1 data packet through the omnidirectional mode). When the topology changes, the number of transmitted data packets decreases, and the beam angle adjustment mechanism based on Q learning is started, and the number of transmitted data packets in the time slot can be quickly restored to 4. Experiments have shown that when the topology changes and causes the number of transmitted data packets to decrease, the mechanism provided by the present invention can quickly adjust the beam angle so that the number of data packets transmitted in the time slot can be quickly restored. This proves that the Q learning algorithm can autonomously learn and optimize the beam angle in a dynamic environment to maximize data transmission efficiency.

[0073] In summary, the scheme of the present invention gives full play to the advantages of directional transceiver array, and introduces Q learning technology to adapt to the dynamic underwater environment, which can achieve high network throughput and high robustness multiple access suitable for AUV cluster formation. The present invention has broad application prospects in the fields of underwater unmanned cluster formation, marine environment monitoring, underwater communication network, etc.

[0074] The above embodiments are only preferred embodiments of the present invention and cannot be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A Q-learning-based underwater unmanned swarm formation time-space division multiple access protocol, characterized by The following steps are involved: 1) AUV cluster formation scene setting: Consider an AUV cluster formation, which includes a master node and N slave nodes; the maximum transmission range of each node is equal, denoted as R; each slave node has an independent number n (n = 1, 2, ..., N); during the operation of the AUV cluster formation, the slave node is responsible for sensing the information in the ocean environment and transmitting it to the master node; the master node is located at the center of the formation and is responsible for sending formation control instructions and collecting information data sensed by the slave nodes; Each master node and slave node is equipped with a directional transceiver array and M (M≥N) transceivers. The directional transmission array contains K beams, K=β·M, that is, each transceiver corresponds to β beams. A certain transceiver is denoted as transceiver m (m=1,2,…,M), and its corresponding beams are denoted as beams (k-β+1), (k-β+2), (k-β+3),…, k, where k=β·m. When the transceiver is working, in order to suppress interference in other directions, one of the beams will be activated as needed. The directional transmission array switches between omnidirectional working mode and directional working mode as needed. For convenience, M=N, that is, the number of transceivers of the master node is equal to the number of slave nodes, so that the master node can simultaneously transmit and receive different data to N slave nodes at most. 2) MAC protocol in the initialization phase of AUV cluster formation: In the initialization phase of the AUV cluster formation, since the master node is completely blind to external factors such as the position, distance, and ocean channel environment of the slave nodes, the time division multiple access (TDMA) protocol is required to obtain more accurate information at the expense of time, so as to facilitate the completion of the network access task of the slave nodes: the master node directional transceiver array adopts an omnidirectional working mode, calculates and broadcasts TDMA scheduling time slot information to each slave node; assuming that the nodes have completed time synchronization, for slave node n, its scheduling time slot starts at T n : T n =n·T, (1) Among them, T n represents the start time of the scheduling time slot of slave node n, n represents the number of slave node n, and T represents the length of a single time slot. To ensure that each slave node can complete the task of accessing the network, the time slot length T is expressed by the following formula: T=R / v water +t packet , (2) Among them, v water is the speed of sound in water, t packet represents the time required to send a data packet; R represents the maximum transmission range of each node; when the slave node n receives the scheduled time slot data packet from the master node, T n , and at T n Feedback information is broadcasted at all times; after receiving feedback information from each slave node, the master node obtains information such as the distance, position, and ocean channel environment of the slave node, and completes the initialization of the AUV cluster formation; 3) MAC protocol for AUV cluster formation navigation phase: During the navigation phase, the task of the AUV cluster formation is to collect underwater information data. This phase mainly involves transmitting data from the nodes to the master node. The master node transmits data every T c Broadcast formation control information once in omnidirectional mode, including movement speed and direction, to ensure that the cluster formation can maintain formation; c represents the time interval of the master node's omnidirectional mode broadcast; since the positions of the master and slave nodes are relatively fixed at this stage, the transmission of the slave nodes adopts the space division multiple access SDMA protocol according to the relative positions of the nodes: the master node's directional transceiver array adopts a directional working mode, and according to the slave node position information obtained in step 2), adjusts each transceiver so that a beam corresponding to each transceiver points to a slave node, and activates the beam through the beam forming technology; since M=N, the slave node corresponding to transceiver m is slave node m, and after one of the beams (k-β+1), (k-β+2), (k-β+3), ..., k of transceiver m is aligned with slave node m, the master node completes information exchange with slave node m through the beam; since the master node has M transmitters in total, different data can be sent to M slave nodes in different directions at the same time without wasting the waiting time of the slave nodes; In this stage, the master node records the received signal-to-noise ratio (SNR) while receiving information sent by the slave node. DT is used to represent the signal detection threshold. It is considered that when SNR≥DT, the physical layer has 100% error-free decoding, that is, the receiving end receives 100% of the data packets sent by the sending end. Due to dynamic changes in the ocean environment such as water currents, the position of the slave node may drift, resulting in deviation from the original beam coverage. It is necessary to adjust the beam to realign the slave node. For transceiver m, when node m is within the beam coverage, SNR≥DT. When SNR<DT, it means that the slave node m deviates from the beam coverage, and the beam angle adjustment mechanism based on Q learning is started: 3.1) Q value table initialization: Generate a Q value table with A rows and B columns. The number of rows and columns of the Q value table satisfies the following formula: Among them, the A value represents the maximum number of iterations. If the topology structure changes drastically, a larger value is taken; the B value represents the number of beams corresponding to transceiver m; the row a (a=1,2,…,A) of the Q value table represents the number of time slots experienced after the node position leaves the beam coverage range, excluding the time slot occupied by the master node broadcasting the formation control information; the column b (b=1,2,…,B) of the Q value table represents the adjustment of the transceiver m activation beam (k-β+b); therefore, Q(a,b) represents the Q value corresponding to the action of adjusting the transceiver m activation beam (k-β+b) in the ath time slot; the larger the value of Q(a,b), the greater the priority of adjusting the transceiver m activation beam (k-β+b) in the ath time slot; that is, in the ath time slot, transceiver m will give priority to activating the beam with the largest Q value; if multiple largest Q values ​​appear in the ath row, transceiver m randomly selects one of the beams to activate; 3.2) Q value update mechanism: In the ath time slot, after the transceiver m activates the beam (k-β+b) according to the Q value at this time, it determines the value of the reward r according to the received signal-to-noise ratio SNR: Where r represents the reward value, SNR represents the received signal-to-noise ratio, and DT represents the signal detection threshold. If SNR ≥ DT, it means that the beam activated by transceiver m can normally receive information from slave node m, and a positive reward is given. Otherwise, it means that the beam range activated by transceiver m still cannot cover slave node m, and a negative reward is given. The Q value of the a+1th row is updated as follows: Q(a+1,b)←(1-γ)Q(a,b)+γ·r (5) Among them, γ is the learning rate, the value range is (0,1]; r represents the reward value; When initializing the Q value table, Q(1,b) is also initialized according to formulas (4) and (5) (Q(0,b)=0); when a=A, that is, when the Q value table is full, the beam corresponding to the maximum Q value is selected for activation, a round of update ends, the Q value table is cleared, and the beam angle adjustment mechanism based on Q learning ends; if SNR≥DT at this time, it means that the current beam range can cover the slave node m; if SNR<DT, the slave node m may leave the coverage range of all beams corresponding to the transceiver m, and the TDMA protocol in step 2) is used to re-initialize the AUV cluster formation; 3.3) Q value update of multiple transceivers: Each transceiver has an independent Q value table. If SNR < DT occurs in multiple transceivers, the beam angle adjustment based on Q learning of these transceivers will be performed simultaneously without interfering with each other. However, when a transceiver ends the beam angle adjustment mechanism based on Q learning, if the TDMA protocol in step 2) needs to be selected to re-initialize the AUV cluster formation, the beam angle adjustment of all other transceivers will also end and the Q value table will be cleared.

Citation Information

Cited By

  • Underwater sound opportunistic routing method based on directional transmission

    CN119997140A

  • An underwater acoustic opportunistic routing method based on directional transmission

    CN119997140B