Low-cost underwater acoustic communication medium access control method and device based on reinforcement learning

Abstract

The application discloses a low-cost underwater acoustic communication medium access control method and device based on reinforcement learning, and the method comprises the following steps: acquiring a time slot length, dividing a transmission time into fixed time slots according to the time slot length; transmitting data packets according to the fixed time slots to obtain a transmission result of the data packets; and updating a dynamic reward value of an upper bound confidence algorithm based on the upper bound confidence algorithm according to the transmission result. The application can improve the transmission rate of underwater communication data and can be widely applied to the technical field of underwater acoustic communication.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underwater acoustic communication, and in particular to a low-cost underwater acoustic communication medium access control method and device based on reinforcement learning. BACKGROUND

[0002] The medium access control protocol (Meaia Access Control, MAC) is an important research branch of underwater acoustic communication network, which mainly manages and coordinates the transmission of data to avoid conflicts. The MAC protocol can be divided into fixed allocation type, random competition type and hybrid MAC protocol, which contains classic protocols such as time division multiple access (TDMA), frequency division multiple access (FDMA), ALOHA protocol, carrier sense multiple access (CSMA) and the like.

[0003] The propagation speed of sound wave in water is about 1500m / s, which is 5 orders of magnitude slower than that of radio wave (3x10 8 m / s), so the MAC protocol of wireless communication cannot be directly used for underwater acoustic communication. The above-mentioned protocols have shortcomings, such as CSMA which consumes a lot of energy due to continuous channel listening, and the traditional ALOHA protocol which easily increases the collision rate due to the increase of traffic, resulting in reduced throughput. Due to the environment, the cost of equipment for underwater acoustic communication is increased, and complex equipment and protocols often increase a lot of expenditure. The current underwater acoustic MAC protocol is insufficient in dynamicity, energy efficiency and anti-collision ability, and it is difficult to meet the needs of complex underwater acoustic communication environment. SUMMARY

[0004] Therefore, the embodiments of the present application mainly aim to provide a low-cost underwater acoustic communication medium access control method and device based on reinforcement learning, so as to solve at least one of the problems in the prior art, and the present application can improve the data transmission rate.

[0005] To achieve the above-mentioned purpose, one aspect of the embodiments of the present application provides a low-cost underwater acoustic communication medium access control method based on reinforcement learning, comprising:

[0006] obtaining a time slot length, and dividing a transmission time into each fixed time slot according to the time slot length;

[0007] sending a data packet according to the fixed time slot to obtain a sending result of the data packet;

[0008] updating a dynamic reward value of an upper confidence bound algorithm based on the upper confidence bound algorithm according to the sending result.

[0009] In some embodiments, the step of obtaining the time slot length comprises the following steps:

[0010] acquiring a maximum data packet transmission time and a maximum transmission distance;

[0011] presetting a sound speed in water and a protection time;

[0012] obtaining the time slot length according to the maximum data packet transmission time, the maximum transmission distance, the sound speed in water and the protection time.

[0013] In some embodiments, the formula used for obtaining the time slot length comprises:

[0014]

[0015] wherein, T slot denotes the time slot length; T tx denotes the maximum data packet transmission time; D max denotes the maximum transmission distance; v sound denotes the sound speed in water; T guard denotes the protection time.

[0016] In some embodiments, the transmitting the data packet according to the fixed time slot to obtain a transmission result of the data packet comprises the following steps:

[0017] When the channel state is in an idle state, it is checked whether the data packet is empty. If the data packet is empty, the time slot length is obtained, and the transmission time is divided into fixed time slots according to the time slot length. If the data packet is not empty, the data packet is transmitted from the beginning of the fixed time slot to obtain the transmission result.

[0018] In some embodiments, the updating a dynamic reward value of the upper bound confidence algorithm based on the upper bound confidence algorithm according to the transmission result comprises the following steps:

[0019] When the transmission result is a transmission failure of the data packet, it is determined whether the transmission failure is caused by collision of the data packet.

[0020] If the transmission failure is caused by collision of the data packet, a first updating operation is performed on the dynamic reward value, and the data packet is subjected to a backoff process.

[0021] If the transmission failure is caused by a timeout of an acknowledgement, a second updating operation is performed on the dynamic reward value, and the data packet is subjected to a backoff process.

[0022] In some embodiments, the backoff process of the data packet comprises the following steps:

[0023] selecting a number of backoff time slots;

[0024] According to the number of backoff time slots, the data packet is retransmitted.

[0025] In some embodiments, the step of updating the dynamic reward value of the upper confidence bound algorithm based on the upper confidence bound algorithm according to the transmission result further comprises the steps of:

[0026] When the transmission result is that the data packet is transmitted successfully, performing a third updating operation on the dynamic reward value, and returning to the step of transmitting the data packet according to the fixed time slots to obtain the transmission result of the data packet.

[0027] To achieve the above object, another aspect of the embodiment of the present application provides a low-cost underwater acoustic communication medium access control device based on reinforcement learning, which comprises:

[0028] A first module is configured to obtain a time slot length, and divide a transmission time into fixed time slots according to the time slot length;

[0029] A second module is configured to transmit a data packet according to the fixed time slots to obtain a transmission result of the data packet;

[0030] A third module is configured to update a dynamic reward value of an upper confidence bound algorithm based on the upper confidence bound algorithm according to the transmission result.

[0031] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0032] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0033] To achieve the above object, another aspect of the embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the above method.

[0034] The embodiments of the present application at least have the following beneficial effects: the present application provides a low-cost underwater acoustic communication medium access control method and device based on reinforcement learning, the scheme obtains a time slot length, divides a transmission time into each fixed time slot according to the time slot length, transmits a data packet according to the fixed time slot, obtains a transmission result of the data packet, and updates a dynamic reward value of an upper bound confidence algorithm based on the upper bound confidence algorithm according to the transmission result, thereby improving the transmission rate of underwater communication data. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0036] Figure 1 is a flowchart of the low-cost underwater acoustic communication medium access control method based on reinforcement learning provided by the embodiments of the present application;

[0037] Figure 2 is a timing diagram of the UCB-sAloha protocol provided by the embodiments of the present application;

[0038] Figure 3 is a schematic diagram of the UCB-sAloha protocol framework provided by the embodiments of the present application;

[0039] Figure 4 is a schematic diagram of the reward mechanism key process provided by the embodiments of the present application;

[0040] Figure 5 is a comparison diagram of the data packet transmission rate provided by the embodiments of the present application;

[0041] Figure 6 is a comparison diagram of the total energy consumption per bit provided by the embodiments of the present application;

[0042] Figure 7 is a comparison diagram of the total energy consumption provided by the embodiments of the present application;

[0043] Figure 8 is a schematic diagram of the hardware structure of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0045] It should be noted that although the functional modules are divided in the system schematic diagram and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the system or the sequence in the flowchart. The terms "first / S100", "second / S200" in the specification and claims and the above drawings can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to a determination".

[0046] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0048] As Figure 1 shown, the embodiments of the present application provide a low-cost underwater acoustic communication medium access control method based on reinforcement learning, which can include but is not limited to steps S100 to S300:

[0049] Step S100, acquiring a time slot length, dividing a transmission time into each fixed time slot according to the time slot length;

[0050] Step S200, transmitting a data packet according to the fixed time slot to obtain a transmission result of the data packet;

[0051] Step S300, updating a dynamic reward value of the upper confidence bound algorithm based on the upper confidence bound algorithm according to the transmission result.

[0052] In the embodiments of the present application, a low-cost medium access control protocol based on reinforcement learning is proposed for underwater acoustic communication medium access control. The protocol is an enhanced time slot ALOHA protocol based on Upper confidence bound (UCB) (hereinafter referred to as UCB-sAloha protocol).

[0053] In some embodiments, step S100 can include but is not limited to steps S101-S103:

[0054] Step S101, obtaining the maximum data packet transmission time and the maximum transmission distance;

[0055] Step S102, pre-setting the underwater sound speed and the protection time;

[0056] Step S103, obtaining the time slot length according to the maximum data packet transmission time, the maximum transmission distance, the underwater sound speed and the protection time.

[0057] In steps S101-S103 of some embodiments, the time slot length is obtained by the maximum data packet transmission time, the maximum transmission distance, the underwater sound speed and the protection time. The formula used includes:

[0058]

[0059] In the formula, T slot represents the time slot length; T tx represents the maximum data packet transmission time; D max represents the maximum transmission distance; v sound represents the underwater sound speed; T guard represents the protection time.

[0060] Exemplarily, taking 2 sensor nodes and 1 center node as an example, the graphical process of UCB-sAloha protocol is shown in Figure 2 Node 0 is the center node, and nodes 1 and 2 are sensor nodes. The transmission time is divided into time slots. The time slot is a fixed time slot, but it will change according to the specific environment. The setting of the fixed time slot can be determined by formula (1). Optionally, the underwater sound speed is set to 1500 m / s, and the protection time is set to 0.01 s.

[0061] In step S200 of some embodiments, when the channel state is in an idle state, it is checked whether the data packet is empty. If the data packet is empty, the length of a time slot is obtained, and the transmission time is divided into fixed time slots according to the length of the time slot. If the data packet is not empty, the data packet is sent starting from the beginning of the fixed time slot, and the sending result is obtained.

[0062] Exemplarily, after sending the data packet, the receiving node determines whether to send an acknowledgement (ACK) according to the collision of the data packet. When the sending node does not receive the acknowledgement, it indicates that the data packet transmission fails this time. The transmission failure conditions include data packet collision and acknowledgement timeout. When the sending node receives the acknowledgement, it indicates that the data packet transmission succeeds this time.

[0063] Reference Figure 2 , the data packet can only be sent at the beginning of the time slot. When the data packets of node 1 and node 2 are sent at the same time, they will collide at node 0, and data packet collision occurs. At this time, the backoff mechanism takes effect, and the data packet selects the next time slot for retransmission. The backoff mechanism of the upper bound confidence algorithm is introduced in the embodiments of the present application, a dynamic reward mechanism is added, and the backoff time is determined by the upper bound confidence algorithm. The number of time slots for backoff is dynamically selected. The UCB-sAloha protocol architecture is shown in Figure 3 .

[0064] In some embodiments, step S300 can include but is not limited to steps S301 to S303:

[0065] Step S301, when the sending result is that the data packet transmission fails, the transmission failure condition is determined.

[0066] Step S302, if the transmission failure condition is that the data packet collides, a first updating operation is performed on the dynamic reward value, and the data packet is subjected to backoff processing.

[0067] Step S303, if the transmission failure condition is that the acknowledgement times out, a second updating operation is performed on the dynamic reward value, and the data packet is subjected to backoff processing.

[0068] In some embodiments, step S300 can further include but is not limited to step S311:

[0069] Step S311, when the sending result is that the data packet transmission succeeds, a third updating operation is performed on the dynamic reward value, and the step of sending the data packet according to the fixed time slot to obtain the sending result of the data packet is returned.

[0070] As Figure 4As shown, the start of the fixed time slot is taken as the starting point, at the start of the fixed time slot, if the channel state is idle and the queue is not empty, the data packet is sent, it is judged whether ACK is needed, if needed, the periodic time slot timer is started, the channel state is changed to waiting for the receiver's confirmation, if the receiver receives the data packet and the target is the node, the ACK is replied, and the data is uploaded to the upper layer; if the sender receives the ACK and the target is the node, the sending result is that the data packet transmission is successful, the sender cancels the periodic time slot timer, and the corresponding backoff arm is rewarded; if the sender does not receive the ACK, it means that the sending result is that the data packet transmission fails, or the data packet collision occurs, or the acknowledgement timeout. According to the sending result of the data packet, the reward mechanism of the UCB-sAloha protocol is executed. When the sending result of the data packet is transmission failure, it is judged whether the transmission failure condition is that the data packet collision occurs, if yes, the first updating operation is performed on the dynamic reward value, and the data packet is subjected to backoff processing, and after the backoff ends, the start of the fixed time slot is returned; if the transmission failure condition is that the acknowledgement timeout, the second updating operation is performed on the dynamic reward value, and the data packet is subjected to backoff processing, and after the backoff ends, the start of the fixed time slot is returned. When the sending result is that the data packet transmission is successful, the third updating operation is performed on the dynamic reward value, and the start of the fixed time slot is returned.

[0071] Optionally, the reward updating mechanism is shown in Table 1. When the data packet collision occurs, the dynamic reward is updated, the moderate punishment avoids repeated collision; when the data packet is sent staggered and the sensor node can receive the ACK sent by the center node, it means that the data transmission is successful, at this time, the dynamic reward is updated, and the current backoff strategy is strengthened; when the sensor node cannot receive the ACK sent by the center node, the waiting ACK timeout, the dynamic reward is updated, and the inefficient strategy is eliminated. After stabilization, the success rate of transmission will be greatly improved, and the data transmission rate can be significantly improved.

[0072]

[0073] In some embodiments, the step of backoff processing the data packet can include but is not limited to: selecting the number of backoff time slots; according to the number of backoff time slots, the data packet is retransmitted. The backoff mechanism of the upper bound confidence algorithm is introduced in the embodiment of the application, the backoff time is determined by the upper bound confidence algorithm, and the number of backoff time slots can be dynamically selected. For example Figure 3In the UCB-sAloha protocol architecture shown, the UCB-sAloha protocol of the embodiment of the application constructs a multi-armed bandit model for backoff time: repeatedly face the action of selecting one of K different options (actions, here the selection of backoff time slots), and after each selection, a scalar reward (here the data packet sending state and the corresponding dynamic reward mechanism) is obtained, which is sampled from an action-dependent probability distribution, and then after a certain number of selections, the maximum total reward is obtained. Among them, any action a in the K actions has a mean or expected value. At time t, the selected action is A t The corresponding reward is R t The value of any action a is denoted as q * (a). The following expression is obtained:

[0074]

[0075] In the formula, q * (a) is the expected reward of action a; (·) * Ground Truth represents the true value that is usually unknown and can only be approximated; E[·] is the expected function.

[0076] Through the action-value estimation method, the estimated value Q t (a) of action a at time t can be obtained, and the following expression is obtained:

[0077]

[0078] In the formula, R i represents the reward of the i-th selection; A i represents the action of the i-th selection; represents the indicator function, which takes the value 1 only when the i-th selection is action a, and 0 otherwise.

[0079] In some embodiments, for the multi-armed bandit model, a balance between exploration and utilization is designed, and the UCB decision value UCB i is determined by formula (4), which is also the updated value of the estimated value Q t (a). The estimated value Q t(a) will be remembered, then at any time t, at least one action has an estimated value that is the largest, the action is called the greedy action, when the greedy action is selected, it is called "exploiting" (Exploiting), because it takes advantage of the existing knowledge of the value of the action; otherwise, if the non-greedy action is selected, it is called "exploring" (Exploring), because it can better understand the value of the non-greedy action.

[0080]

[0081] In the formula, represents the exploitation term, which represents the average reward of the ith arm; represents the exploration term; n i represents the number of times the ith arm is selected; T represents the total number of selections; c represents the exploration coefficient, which is taken as c = 2.0 in the UCB-sAloha protocol.

[0082] In some embodiments, the main logic pseudocode in the low-cost underwater acoustic communication medium access control method based on reinforcement learning is also given:

[0083] (1) The pseudocode of the UCB algorithm is summarized as shown in Table 2:

[0084] Table 2

[0085]

[0086]

[0087] (2) The pseudocode of the workflow of the UCB-sAloha protocol is summarized as shown in Table 3:

[0088] Table 3

[0089]

[0090]

[0091] In some embodiments, the UCB-sAloha protocol is implemented by taking the simulation platform NS3 as an example, and the low-cost underwater acoustic communication medium access control process based on reinforcement learning includes the following steps:

[0092] Step 1: Construct a simulation environment, set the simulation parameters as shown in Table 4, and all numerical values are realized by setting the code of the running script.

[0093] Table 4

[0094] Attribute Value Topology Multi-hop underwater sensor network Number of nodes 128 Node distribution Densely deployed Propagation speed 1500 m / s Maximum transmission distance 3 km Channel rate 10 kbps Slot length Dynamic adjustment of T slot ]]> Application layer traffic rate 0.01-0.08 pkts / sec UCB exploration coefficient 2.0 UCB back-off slot selection range 1-10 Node reception consumption power 0.82W Simulation time 100s

[0095] Step 2: After UCB-sAloha protocol is started, it enters the initialization phase, and the protocol will obtain the time slot length according to formula (1), divide the simulation time into fixed time slots, start the time slot timer, check whether the data packet starts at the beginning of the time slot, check the state of the channel, and enter the next process according to different states, wherein the state is defined in the header file of the protocol, as follows:

[0096] PASSIVE: idle state, check the data queue;

[0097] SEND_DATA: send data packet;

[0098] WAIT_ACK: wait for the receiver to confirm;

[0099] BACKOFF: backoff waiting state.

[0100] Step 3: If the channel state is in the idle state, check whether the data packet is empty, if it is empty, return to step 2, if it is not empty, send the data packet.

[0101] After sending the data packet, the receiving node decides whether to send ACK according to the collision of the data packet, and if the sending node does not receive ACK, it means that this transmission fails, either the data packet collides or the ACK times out.

[0102] Initially, set i = 0 and n i = 0;

[0103] If the data packet collides, the UCB dynamic reward value is updated, and the UCB decision value is recorded At this time, the protocol will back off the data packet, and the number of backoff time slots is selected according to i = i + 1, and i ∈ {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};

[0104] Similarly, if the ACK times out, the UCB dynamic reward value is updated, and the UCB decision value is recorded At this time, the protocol will back off the data packet, and the number of backoff time slots is selected according to i = i + 1, and i ∈ {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};

[0105] When the sending node receives ACK, it means that the data transmission is successful, the UCB dynamic reward value is updated, and the UCB decision value is recorded Return to step 3;

[0106] With the increase of cumulative data, the protocol will dynamically adjust its strategy with the reward value feedback, which is also an advantage of UCB algorithm compared with other algorithms. In UCB-sAloha protocol, the selection of time slot is determined by the size of UCB i , and satisfies , so as to obtain the optimal number of backoff time slots, greatly increase the data packet transmission rate, and also reduce the energy consumption.

[0107] Step 4: After the execution of steps 1 to 3, simulation analysis is carried out in step 4, and the values of the related quantities can be obtained. The required indicators can be calculated by the following formula:

[0108]

[0109] Total energy consumption E Total =∑(transmission energy consumption P Tx ×data packet transmission time t+receiving power P Rx ×data packet transmission time t)

[0110]

[0111] Wherein, N TxPackets , N RxPackets , t, t sim , P Tx , P Rx can be given by simulation results.

[0112] Through simulation analysis, the index comparison diagram of UCB-sAloha protocol and Aloha protocol as shown in Figure 5 , Figure 6 and Figure 7 can be obtained. Figure 5 is the comparison of data packet transmission rate between UCB-sAloha protocol and Aloha protocol, Figure 6 is the comparison of total energy consumption per bit between UCB-sAloha protocol and Aloha protocol, Figure 7For the total energy consumption comparison between UCB-sAloha protocol and aloha protocol, the horizontal axis is the application layer traffic rate. It can be seen that the performance of UCB-sAloha protocol is much better than that of aloha protocol, the data packet transmission rate is not only higher than that of aloha protocol, and with the increase of data traffic, the collision of aloha protocol is severe, and the transmission rate is significantly reduced, but the UCB-sAloha protocol performs stably, even if the data traffic increases, the data packet transmission rate will not fluctuate too much, and is stable in an excellent performance interval. According to the energy consumption comparison, it can be known that with the increase of data traffic, the total energy consumption per bit of aloha protocol grows very fast, and has a faster and faster trend, but the total energy consumption per bit of UCB-sAloha protocol is very low, even if the data traffic increases, it still maintains at a low energy consumption level, and the total energy consumption of UCB-sAloha protocol is also much lower than that of aloha protocol.

[0113] The embodiment of the application further provides a low-cost underwater acoustic communication medium access control device based on reinforcement learning, which can realize the low-cost underwater acoustic communication medium access control method based on reinforcement learning.

[0114] The first module is configured to obtain a time slot length, and divide a transmission time into fixed time slots according to the time slot length.

[0115] The second module is configured to send data packets according to the fixed time slots, and obtain a sending result of the data packets.

[0116] The third module is configured to update a dynamic reward value of the upper confidence bound algorithm based on the upper confidence bound algorithm according to the sending result.

[0117] It can be understood that the contents in the method embodiments are applicable to the device embodiments, the device embodiments specifically realize the functions of the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0118] The embodiment of the application further provides an electronic device, which comprises a processor and a memory, the memory stores a computer program, and the processor realizes the method described above when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0119] It can be understood that the contents in the method embodiments are applicable to the device embodiments, the device embodiments specifically realize the functions of the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0120] Reference Figure 8 , Figure 8The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device comprises:

[0121] The processor 401 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0122] The memory 402 can be implemented by a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), and the like. The memory 402 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 402 and are called and executed by the processor 401 to implement the method of the embodiments of the present application.

[0123] The input / output interface 403 is configured to realize information input and output.

[0124] The communication interface 404 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0125] The bus 405 is configured to transmit information between the components (for example, the processor 401, the memory 402, the input / output interface 403, and the communication interface 404) of the device.

[0126] The processor 401, the memory 402, the input / output interface 403, and the communication interface 404 are connected to each other through the bus 405 to realize the communication connection between the components in the device.

[0127] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above method.

[0128] It can be understood that the above method embodiments are applicable to the storage medium embodiments, the storage medium embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0129] The embodiment of the present application also provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium.

[0130] To sum up, the low-cost underwater acoustic communication medium access control method and device based on reinforcement learning has the following advantages:

[0131] 1. The embodiment of the present application is based on the upper bound confidence algorithm enhanced time slot ALOHA protocol (UCB-sAloha protocol), which realizes intelligent selection and real-time update of backoff time slots through a multi-armed bandit model and lightweight reinforcement learning, and provides an innovative solution with high transmission rate, low energy consumption and low cost for underwater communication.

[0132] 2. The UCB-sAloha protocol of the low-cost underwater acoustic communication medium access control method based on reinforcement learning inherits the advantages of low cost, simple structure and high flexibility of the aloha protocol.

[0133] 3. The UCB-sAloha protocol of the low-cost underwater acoustic communication medium access control method based on reinforcement learning dynamically optimizes backoff time by using the UCB algorithm, intelligently balances exploration and utilization, and the data obtained through NS3 platform simulation verifies its efficacy, which not only improves the data transmission rate but also reduces the energy consumption, and effectively reduces the cost.

[0134] 4. The embodiment of the present application introduces a reinforcement learning method, which can adapt to more environments, and neither the setting of time slots nor the selection of backoff time is fixed, so that the UCB-sAloha protocol can adapt to different environments and improve the generality of the protocol.

[0135] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example, with the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0136] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the described functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary to an understanding of the application. Rather, the actual implementation of the modules, in combination with their attributes, functions, and internal relationships, are to be understood within the context of the devices disclosed herein. Thus, those skilled in the art with access to patents, scientific journals, and other public sources known by those skilled in the art will be able to practice the application as set forth in the claims without undue experimentation, using ordinary skill in the art along with the present disclosure. It is also to be understood that the specific concepts disclosed are merely illustrative and that the scope of the present application is to be determined by the entire scope of the claims, along with all equivalents of the claims and their equivalents.

[0137] If the functions are implemented in software, the functions can be stored in or implemented as one or more computer program products, which can be incorporated into a computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be for example but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.

[0138] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions executed in an order to achieve a logic function, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0139] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0140] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the embodiments described above, various steps or methods can be implemented, for example, by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following techniques can be used to implement the hardware used in the described embodiments: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0141] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In the specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific feature, structure, material or characteristic described can be combined in any appropriate manner in one or more embodiments or examples.

[0142] Although embodiments of the application have been shown and described, it would be recognized by those of ordinary skill in the art that various changes, modifications, substitutions, and alterations can be made hereto without departing from the spirit and scope of the application. The scope of the application is limited only by the claims and the equivalents thereof.

[0143] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A low-cost underwater acoustic communication medium access control method based on reinforcement learning, characterized in that, Includes the following steps: The time slot length is obtained, and the transmission time is divided into fixed time slots according to the time slot length. The process of obtaining the time slot length includes: obtaining the maximum data packet transmission time and the maximum transmission distance; presetting the underwater sound speed and the protection time; and obtaining the time slot length according to the maximum data packet transmission time, the maximum transmission distance, the underwater sound speed, and the protection time. Data packets are sent according to the fixed time slot to obtain the sending result of the data packets; Based on the transmission result, and using the upper bound confidence algorithm, the dynamic reward value of the upper bound confidence algorithm is updated, including: When the sending result indicates that the data packet transmission failed, the transmission failure situation is determined; If the transmission failure is due to a data packet collision, then a first update operation is performed on the dynamic reward value, and the data packet is backed off; the formula used for the first update operation on the dynamic reward value includes: In the formula, The representative of the use term indicates the first Average reward per arm; If the transmission failure is due to a confirmation response timeout, a second update operation is performed on the dynamic reward value, and the data packet is backed up; the formula used for the second update operation on the dynamic reward value includes: .

2. The method according to claim 1, characterized in that, The formula used to obtain the time slot length includes: ； In the formula, Indicates the time slot length; Indicates the maximum data packet transmission time; Indicates the maximum transmission distance; Indicates the speed of sound in water; Indicates the protection period.

3. The method according to claim 1, characterized in that, The step of sending data packets according to the fixed time slot and obtaining the sending result of the data packets includes the following steps: When the channel is in an idle state, check whether the data packet is not empty. If the data packet is empty, return to obtain the time slot length. Based on the time slot length, divide the transmission time into fixed time slots. If the data packet is not empty, send the data packet starting from the beginning of the fixed time slot to obtain the sending result.

4. The method according to claim 1, characterized in that, The backoff process for the data packet includes the following steps: Choose the number of backoff slots; The data packet is retransmitted according to the number of backoff slots.

5. The method according to claim 1, characterized in that, The step of updating the dynamic reward value of the upper bound confidence algorithm based on the sending result further includes the following steps: When the sending result indicates that the data packet was successfully transmitted, a third update operation is performed on the dynamic reward value, and the process of sending the data packet according to the fixed time slot to obtain the sending result of the data packet is returned. The formula used for the third update operation on the dynamic reward value includes: .

6. A low-cost underwater acoustic communication medium access control device based on reinforcement learning, characterized in that, include: The first module is used to obtain the time slot length and divide the transmission time into fixed time slots according to the time slot length. Obtaining the time slot length includes: obtaining the maximum data packet transmission time and the maximum transmission distance; presetting the underwater sound speed and the protection time; and obtaining the time slot length according to the maximum data packet transmission time, the maximum transmission distance, the underwater sound speed, and the protection time. The second module is used to send data packets according to the fixed time slot and obtain the sending result of the data packets; The third module is used to update the dynamic reward value of the upper bound confidence algorithm based on the sending result; the third module is specifically used for: When the sending result indicates that the data packet transmission failed, the transmission failure situation is determined; If the transmission failure is due to a data packet collision, then a first update operation is performed on the dynamic reward value, and the data packet is backed off; the formula used for the first update operation on the dynamic reward value includes: In the formula, The representative of the use term indicates the first Average reward per arm; If the transmission failure is due to a confirmation response timeout, a second update operation is performed on the dynamic reward value, and the data packet is backed up; the formula used for the second update operation on the dynamic reward value includes: .

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 5.

Images