Double-perception online fountain code transmission system and method based on deep reinforcement learning

Abstract

The invention relates to a double-perception online fountain code transmission system and method based on deep reinforcement learning. The system is applied to an underwater wireless sensor network, and comprises a network modeling module which is used for constructing an underwater acoustic sensor network model of a three-dimensional hierarchical clustering topology by adopting a fuzzy clustering algorithm according to spatial distribution and residual energy of nodes, and determining a transmission mechanism of half-duplex underwater acoustic communication; the reinforcement learning optimization module is used for modeling a decoding process of an online fountain code into a Markov decision process and training an agent in combination with a deep learning algorithm to generate an optimal coding degree sequence; and the coding transmission execution module is used for executing the optimal coding degree sequence to generate a coding data packet, performing multi-hop transmission, and performing coding degree switching based on limited feedback triggering until the aggregation node recovers all original data. Based on a half-duplex underwater acoustic communication mechanism, channel conflicts are avoided, the optimal coding degree sequence can be dynamically matched with a decoding state, and the data transmission reliability in an extreme underwater environment is improved.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to a dual-sensor online fountain coding transmission system and method based on deep reinforcement learning. Background Technology

[0002] With the rapid development of marine environmental research, natural resource exploration, and environmental monitoring, underwater acoustic sensor networks (UASNs), as a key infrastructure for long-distance underwater data transmission, are increasingly demonstrating their application value. UASNs connect various sensor nodes distributed underwater via acoustic links, enabling real-time acquisition and transmission of critical information such as marine environmental parameters and target monitoring data. This allows for real-time, distributed sensing and information aggregation across vast waters, possessing significant scientific and practical value. However, the extremely complex physical characteristics of the underwater environment pose severe challenges to the reliable transmission of data in UASNs. Specifically, underwater acoustic channels suffer from inherent defects such as extremely limited bandwidth, high propagation delay, large path loss, significant multipath effects, and strong time- and space-varying characteristics. These characteristics severely restrict the application effectiveness of traditional data transmission technologies in underwater environments.

[0003] On the one hand, most traditional data transmission technologies rely on retransmission requests, which suffer from significant drawbacks such as short network lifetime, unreliable data transmission, and high energy consumption. On the other hand, online fountain code (OFC) schemes used in other wireless network scenarios do not consider the unique physical factors of underwater environments, resulting in low channel utilization and network throughput in UASNs, as well as excessively high end-to-end latency. Therefore, ensuring reliable data transmission in UASNs has become a critical issue that urgently needs to be addressed in the field. Furthermore, existing online fountain code (OFC) schemes used in terrestrial wireless networks and other scenarios do not fully consider the unique physical constraints of underwater acoustic channels. These schemes employ fixed or single degree distribution strategies, which cannot adapt to the time-varying, limited bandwidth, and long propagation delay characteristics of underwater channels, leading to significant shortcomings in underwater acoustic sensor network applications.

[0004] Therefore, the traditional method of using underwater acoustic sensor networks for encoding and transmission often results in low data transmission reliability during the encoding and transmission process due to the short lifespan, high energy consumption, low channel utilization, and large transmission delay of underwater acoustic sensor networks. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, a dual-sensor online fountain encoding and transmission system and method based on deep reinforcement learning is provided, which can realize efficient and reliable data transmission in underwater acoustic sensor networks.

[0006] A dual-sensor online fountain encoding and transmission system based on deep reinforcement learning, applied to underwater wireless sensor networks, includes a network modeling module, a reinforcement learning optimization module, and an encoding and transmission execution module that are sequentially constructed and operate collaboratively.

[0007] The network modeling module is used to construct a three-dimensional hierarchical and clustered topology underwater acoustic sensor network model based on the spatial distribution and remaining energy of each sensor node in the underwater wireless sensor network, using a fuzzy clustering algorithm, and to establish a transmission mechanism based on half-duplex underwater acoustic communication.

[0008] The reinforcement learning optimization module is used to model the decoding process of the online fountain code as a Markov decision process and train the agent by combining the underwater acoustic sensor network model and transmission mechanism as environmental input, thereby generating the optimal coding degree sequence.

[0009] The encoding transmission execution module is used to generate encoded data packets by executing the optimal encoding degree sequence and perform multi-hop transmission, and to control the encoding degree switching based on limited feedback until the aggregation node recovers all the original data.

[0010] In one embodiment, the network modeling module is further configured to divide the sensor nodes into clusters according to their spatial distribution and remaining energy, and elect a cluster head node for each cluster; the cluster head node is used for intra-cluster coordination and inter-cluster multi-hop forwarding.

[0011] The cluster head node forwards the aggregated data to the aggregation node via a multi-hop path. The aggregation node is equipped with an underwater acoustic modem and a radio frequency modem. Each of the sensor nodes is equipped with a half-duplex underwater acoustic modem.

[0012] In the underwater acoustic sensor network model, at each hop of data transmission, the sender generates an encoded data packet from the local raw data using an XOR-based encoding method and transmits it to the next hop node; after successfully receiving and decoding, the intermediate node re-encodes the recovered data and forwards it to complete the data transmission.

[0013] In one embodiment, the reinforcement learning optimization module includes a Markov decision process modeling unit, a degree space pruning unit, and a deep network training unit, wherein:

[0014] The Markov decision process modeling unit is used to model the online fountain code decoding process as a Markov decision process by recursively relating the online fountain code recovery rate to the buffer occupancy rate.

[0015] The degree space pruning unit is used to calculate and determine the actual upper limit of the coding degree based on the preset recovery rate threshold and recovery gain increment. When the candidate coding degree is less than or equal to the actual upper limit, the candidate coding degree is selected as the coding degree; otherwise, the actual upper limit is selected as the coding degree.

[0016] The deep network training unit is used to train the agent using a two-stage training mechanism to generate the optimal coding degree sequence.

[0017] In one embodiment, the two-stage training mechanism includes an offline preparation stage and an online training stage, wherein:

[0018] During the offline preparation phase, a simulation decoding trajectory initialization strategy based on multiple recovery rate intervals is employed.

[0019] During the online training phase, a greedy strategy is adopted, in which the agent interacts iteratively with the simulation environment to generate transition tuples, which are stored in the experience replay buffer. The network is then updated through stochastic gradient descent until the network converges to the optimal encoding strategy.

[0020] In one embodiment, the reinforcement learning optimization module further includes an initialization unit for defining the state space, action space, and designing a reward function;

[0021] The state space includes the recovery rate interval, expected buffer occupancy, current coding degree, check node, and degree distribution of undecoded symbols; the action space is a set of coding degrees; the reward function balances transmission overhead and intermediate recovery rate through weight coefficients and imposes penalties on actions that do not increase coding degree.

[0022] In one embodiment, the encoding transmission execution module includes a dual-sensor online fountain encoder, which, when generating encoded packets, simultaneously senses network environment constraint information from the network modeling module and decoding state information from the decoding end, and dynamically adjusts the degree distribution to generate encoded symbols based on the optimal coding degree sequence output by the reinforcement learning optimization module.

[0023] In one embodiment, the encoding transmission execution module further includes a finite feedback iterative transmission controller, which controls the sender to generate and send encoded packets according to the optimal coding degree sequence, and after receiving feedback information triggered by the decoding end meeting preset conditions, switches to the next coding degree in the optimal coding degree sequence, iterating until the data is completely recovered.

[0024] A deep reinforcement learning-based dual-sensor online fountain coding transmission method is applied to a deep reinforcement learning-based dual-sensor online fountain coding transmission system. The method includes:

[0025] The spatial distribution and remaining energy of sensor nodes are obtained, a three-dimensional hierarchical and clustered topology underwater acoustic sensor network model is constructed using fuzzy clustering algorithm, and a multi-hop transmission mechanism based on half-duplex underwater acoustic communication is established.

[0026] The decoding process of online fountain codes is modeled as a Markov decision process, defining a state space, action space, and reward function; the action space is pruned and optimized, and trained using a deep learning algorithm to obtain the optimal coding degree sequence;

[0027] During network transmission, the encoder generates encoded data packets based on the optimal coding degree sequence and performs multi-hop transmission; the decoder sends limited feedback to trigger coding degree switching according to preset feedback conditions until all original data packets are successfully recovered at the aggregation node.

[0028] In one embodiment, the method further includes:

[0029] The sensor nodes are divided into clusters according to their spatial distribution and remaining energy, and a cluster head node is elected for each cluster; the cluster head node is used for intra-cluster coordination and inter-cluster multi-hop forwarding.

[0030] The cluster head node forwards the aggregated data to the aggregation node via a multi-hop path. The aggregation node is equipped with an underwater acoustic modem and a radio frequency modem. Each of the sensor nodes is equipped with a half-duplex underwater acoustic modem.

[0031] In the underwater acoustic sensor network model, at each hop of data transmission, the sender generates an encoded data packet from the local raw data using an XOR-based encoding method and transmits it to the next hop node; after successfully receiving and decoding, the intermediate node re-encodes the recovered data and forwards it to complete the data transmission.

[0032] The aforementioned dual-sensor online fountain coding transmission system and method based on deep reinforcement learning constructs a three-dimensional hierarchical and clustered topological underwater acoustic sensor network model. By using fuzzy clustering to cluster nodes according to spatial distribution and remaining energy, it avoids ordinary nodes directly participating in long-distance multi-hop transmission, significantly reducing the communication energy consumption of a single node. The half-duplex underwater acoustic communication-based transmission mechanism avoids channel conflicts and improves the utilization rate of limited underwater channel resources. The optimal coding degree sequence trained by reinforcement learning can dynamically match the decoding state, ensuring that the sinking node can ultimately fully recover all original data packets, greatly improving the reliability of data transmission in extreme underwater environments. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the application framework of a dual-sensor online fountain encoding and transmission system based on deep reinforcement learning in one embodiment;

[0034] Figure 2 This is a block diagram of a dual-sensor online fountain encoding transmission system based on deep reinforcement learning in one embodiment;

[0035] Figure 3 This is a schematic diagram of an underwater acoustic sensor network model in one embodiment;

[0036] Figure 4 This is a schematic diagram of a DQN framework for degree optimization in one embodiment;

[0037] Figure 5 This is a flowchart illustrating a dual-sensor online fountain encoding transmission method based on deep reinforcement learning in one embodiment. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] In one embodiment, such as Figure 1 As shown, an application framework for a dual-sensor online fountain encoding and transmission system based on deep reinforcement learning is presented, mainly including the reinforcement learning process, the practical application process, and the transmission process. Among them, as... Figure 1 As shown, the reinforcement learning process mainly includes problem modeling, initialization, deep Q-learning (DQN) training, and obtaining the optimal degree sequence. In practical applications, the obtained optimal degree sequence is mainly embedded into the decoder to guide encoding decisions. Data transmission between two nodes can be seen as iterative interaction between the decoder and encoder until all data reaches the sink node. The transmission process mainly involves the sender generating encoded data packets based on the current degree in the sequence, the decoder updating the decoding graph and monitoring the recovery rate. When the predefined feedback conditions are met, the decoder notifies the sender, prompting a transition to the next degree in the sequence. This iterative process continues until all original data packets are successfully recovered at the sink node.

[0040] In one embodiment, such as Figure 2 As shown, a dual-sensor online fountain encoding and transmission system based on deep reinforcement learning is provided, applied to underwater wireless sensor networks. It includes a network modeling module 210, a reinforcement learning optimization module 220, and an encoding and transmission execution module 230, which are sequentially constructed and operate collaboratively.

[0041] The network modeling module 210 is used to construct a three-dimensional hierarchical and clustered topology underwater acoustic sensor network model based on the spatial distribution and remaining energy of each sensor node in the underwater wireless sensor network, and to establish a transmission mechanism based on half-duplex underwater acoustic communication.

[0042] The reinforcement learning optimization module 220 is used to model the decoding process of the online fountain code as a Markov decision process and combine it with deep learning algorithms to train the agent and generate the optimal coding degree sequence, using the underwater acoustic sensor network model and transmission mechanism as environmental input.

[0043] The encoding transmission execution module 230 is used to generate encoded data packets by executing the optimal encoding degree sequence and perform multi-hop transmission, and to control the encoding degree switching based on limited feedback until the sink node recovers all the original data.

[0044] In one embodiment, the network modeling module 210 is further configured to divide the sensor nodes into clusters according to their spatial distribution and remaining energy, and elect a cluster head node for each cluster; the cluster head node is used for intra-cluster coordination and inter-cluster multi-hop forwarding; the cluster head node forwards the aggregated data to the sink node through a multi-hop path, and the sink node is equipped with an underwater acoustic modem and an RF modem; each sensor node is equipped with a half-duplex underwater acoustic modem; in the underwater acoustic sensor network model, for each hop of data transmission, the sender uses an XOR-based encoding method to generate an encoded data packet from the local raw data and transmits it to the next hop node; after successfully receiving and decoding, the intermediate node re-encodes the recovered data and forwards it to complete the data transmission.

[0045] In this embodiment, a three-dimensional hierarchical clustered topology underwater acoustic sensor network model is first established, such as... Figure 3 As shown, a hierarchical clustering structure is adopted, and the sensor nodes are divided into k clusters using fuzzy clustering. The sensor nodes are organized into clusters based on their spatial distribution and remaining energy. Each cluster elects a cluster head (CH) responsible for intra-cluster coordination and inter-cluster forwarding. Specifically, in this embodiment, the inter-node data transmission process may include: ordinary sensor nodes sending data packets to their corresponding CHs; the CHs forwarding the aggregated data to the sink node via multi-hop paths; each underwater node is equipped with a half-duplex underwater acoustic modem, while the sink node is further equipped with an underwater acoustic modem (for underwater communication) and a radio frequency modem (for communication with the ground base station). It is assumed that the sender and receiver share knowledge of the channel state.

[0046] At each hop, the sender uses XOR-based encoding to generate encoded data packets from the local raw data and transmits them to the next-hop node. After successful reception and decoding, intermediate nodes re-encode the recovered data and forward it; this process continues until the sink node receives and recovers all the original data packets.

[0047] In one embodiment, during the reinforcement learning phase, the decoding process of the online fountain code (OFC) is modeled as a Markov decision process (MDP), and the agent is trained using a deep Q-learning network (DQN) algorithm. The agent learns to select the encoding degree that maximizes recovery efficiency while minimizing redundancy, and after convergence, the training produces an optimal degree sequence.

[0048] Specifically, in one embodiment, the reinforcement learning optimization module includes a Markov decision process modeling unit, a degree space pruning unit, and a deep network training unit, wherein: the Markov decision process modeling unit is used to model the online fountain code decoding process as a Markov decision process by recursively relating the online fountain code recovery rate to the buffer occupancy rate; the degree space pruning unit is used to calculate and determine the actual upper limit of the coding degree based on a preset recovery rate threshold and recovery gain increment; when the candidate coding degree is less than or equal to the actual upper limit, the candidate coding degree is selected as the coding degree; otherwise, the actual upper limit is selected as the coding degree; the deep network training unit is used to train the agent using a two-stage training mechanism to generate the optimal coding degree sequence.

[0049] To make the degree selection of dual-sensing online fountain code (OFC) in underwater acoustic sensor networks (UASNs) intelligent and adaptive, in this embodiment, the encoding process is described as a reinforcement learning problem, allowing the encoder to dynamically adjust the degree distribution according to the decoding state and environmental constraints.

[0050] Specifically, the completion phase of Online Fountain Code (OFC) exhibits a recursive relationship between recovery rate and buffer occupancy rate. This relationship can be used to model the decoding process as a Markov Decision Process (MDP), allowing... This represents the degree of the i-th coded symbol. This represents the recovery rate after processing the i-th symbol. Buffer usage status and The expected values ​​are respectively used as and The recursive relation can be represented as: ; ; ; ;in, This represents the average degree of unrecovered packets, given the decoding state. Given a fixed coding degree m, there exists a unique z that satisfies This indicates that the target recovery rate has been achieved. The number of groups required is determined solely by the current state and the selected coding degree, thus confirming the memoryless nature of open network coding and decoding.

[0051] In one embodiment, the reinforcement learning optimization module further includes an initialization unit for defining the state space, action space, and designing a reward function; wherein the state space includes the recovery rate interval, expected buffer occupancy, current coding degree, check node, and degree distribution of undecoded symbols; the action space is a set of coding degrees; the reward function balances transmission overhead and intermediate recovery rate through weight coefficients and imposes penalties on actions that do not increase coding degree.

[0052] By modeling the decoding process as a Markov decision process, it is evident that online fountain codes possess Markov properties. The OFC completion process is modeled as a Markov decision process (MDP), where the encoder acts as a reinforcement learning (RL) agent, and the environment includes an underwater acoustic channel, a decoder, and a constrained feedback mechanism. At each decision step, the agent selects the degree of the next encoded data packet based on the current decoding state to optimize transmission efficiency.

[0053] The state space S is defined to simultaneously capture the decoder state and degree distribution features. Specifically, the state at step i is represented as: ;in This represents the current recovery rate interval between consecutive degrees. This is the expected buffer size. It is the current degree, and These represent the degree distribution of the check node and the undecoded symbol, respectively.

[0054] Action space is defined as: Each action corresponds to selecting a new degree for the next symbol to be encoded. .

[0055] To enable the agent to employ an efficient encoding strategy, a reward function is designed in this embodiment. This function considers two important objectives: minimizing the number of symbols (overhead) required to achieve the target recovery rate, and improving the recovery rate in intermediate stages. The total reward at time step i can be expressed as: ;in To balance the weighting of the two objectives, the reward associated with the cost is defined as follows: ;in, This indicates the estimated number of encoded packets required to reach the next target recovery rate based on the current decoding state and the selected degree. The reward for intermediate decoding improvements is defined as: This demonstrates the efficiency of improving the recovery rate with fewer coding symbols. To avoid inefficient or redundant operations, a penalty is applied if the selected degree is not increased. This reward mechanism incentivizes agents to select actions that facilitate rapid recovery while avoiding stagnation or unnecessary transmissions, thereby ensuring the efficiency and robustness of the encoding process.

[0056] In one embodiment, the degree-space pruning unit is primarily used for degree-space pruning and learning strategies. Although the action space... Flexible degree selection is allowed, but overexploration can lead to slow convergence and potential performance degradation. To improve efficiency, the degree space is dynamically pruned based on restitution information. The restitution threshold for the transition between degree m and degree m+1 can be expressed as: Then, the relationship between the recovery gain increments between adjacent degree transformations is analyzed, and the relationship can be expressed as: Let M be the upper limit of the acceptable value, and choose M that satisfies: The rules for selecting the degree are as follows: .

[0057] In this embodiment, to further control complexity, a maximum allowed length is imposed on the encoded sequence. These constraints narrowed the search space and improved the learning rate of the reinforcement learning (RL) agent. Training was conducted using deep Q-learning in a deterministic simulation environment.

[0058] In one embodiment, the deep network training unit is mainly used for deep Q-learning (DQN) training. Specifically, the two-stage training mechanism includes an offline preparation stage and an online training stage. In the offline preparation stage, a simulated decoding trajectory initialization strategy based on multiple recovery rate intervals is used. In the online training stage, a greedy strategy is adopted, and the agent interacts iteratively with the simulated environment to generate transition tuples that are stored in the experience replay buffer. The network is updated through stochastic gradient descent until the network converges to the optimal encoding strategy.

[0059] Specifically, the learning framework based on Deep Q-Learning (DQN) consists of two phases: an offline preparation phase to initialize the strategy and an online training phase for iterative optimization. Figure 4 This demonstrates the integration of DQN into Underwater Acoustic Sensor Networks (UASNs): In each iteration, encoder observations are used to construct a state vector; the optimal coding degree is selected using Deep Q-Learning (DQN); decoder feedback is received; and the policy is updated based on the received reward. (Transition tuple) Stored in an empirical buffer for batch training via stochastic gradient descent.

[0060] During the offline preparation phase, the agent is trained based on simulated decoding trajectories covering multiple recovery rate ranges. For each representative state, the agent determines the optimal degree by selecting the action with the highest predicted Q value. The purpose of the offline preparation phase is to provide coarse-grained initial settings for the policy, thereby shortening the convergence time in the subsequent training process.

[0061] During the online training phase, the agent iteratively interacts with the environment simulating the underwater decoding process. Each round corresponds to a complete encoding and decoding flow, and the environment is reset at the start of each round. In each decision step, the agent observes the current state, which reflects the decoding graph structure, buffer occupancy, and recovery rate; subsequently, the agent adopts... A greedy strategy is used to balance exploration and exploitation, choosing one action. This action represents the degree of the next encoded symbol; after executing this action, the decoder updates its internal graph and decoding state; the encoder obtains limited feedback based on the updated state and calculates the corresponding reward according to pre-set performance indicators. Each transition tuple All data is stored in an experience replay buffer; subsequently, small batches of data are periodically extracted from this buffer to update the Q-network. Online training proceeds through multiple rounds until the agent's policy reaches convergence, meaning the selected degree sequence stabilizes and no further performance improvement is observed. This iterative reinforcement learning process allows the encoder to progressively optimize its degree selection strategy, reducing transmission overhead while improving intermediate recovery performance.

[0062] In one embodiment, the encoding transmission execution module includes a dual-sensory online fountain encoder, which, when generating encoded packets, simultaneously senses network environment constraint information from the network modeling module and decoding state information from the decoding end, and dynamically adjusts the degree distribution to generate encoded symbols based on the optimal coding degree sequence output by the reinforcement learning optimization module.

[0063] In one embodiment, the encoding transmission execution module further includes a finite feedback iterative transmission controller, which controls the sender to generate and send encoded packets according to the optimal coding degree sequence, and after receiving feedback information triggered by the decoding end meeting preset conditions, switches to the next coding degree in the optimal coding degree sequence, iterating until the data is completely recovered.

[0064] After Markov modeling, the agent is trained using deep reinforcement learning techniques. Through multiple iterations of training until the encoding policy converges, an optimal degree sequence is generated. This optimal degree sequence is then applied in practical applications. In the practical application phase, the learned degree sequence is embedded into the decoder; the sender uses this sequence to guide encoding decisions without frequent interaction with the decoder. Data transmission between two nodes can be viewed as data transmission between the decoder and encoder, with iterative interaction between the decoder and encoder until all data reaches the convergence node.

[0065] In one embodiment, such as Figure 5As shown, a dual-sensor online fountain coding transmission method based on deep reinforcement learning is provided, which is applied to a dual-sensor online fountain coding transmission system based on deep reinforcement learning, and includes the following steps:

[0066] Step 502: Obtain the spatial distribution and remaining energy of the sensor nodes, construct a three-dimensional hierarchical clustered topology underwater acoustic sensor network model using fuzzy clustering algorithm, and establish a multi-hop transmission mechanism based on half-duplex underwater acoustic communication.

[0067] Step 504: Model the decoding process of the online fountain code as a Markov decision process, defining the state space, action space, and reward function; perform pruning optimization on the action space, and train it using a deep learning algorithm to obtain the optimal coding degree sequence;

[0068] Step 506: During network transmission, the encoder generates encoded data packets based on the optimal coding degree sequence and performs multi-hop transmission; the decoder sends limited feedback to trigger coding degree switching according to preset feedback conditions until all original data packets are successfully recovered at the aggregation node.

[0069] In one embodiment, a dual-sensor online fountain encoding and transmission method based on deep reinforcement learning may further include a data transmission process, specifically comprising: dividing sensor nodes into clusters according to spatial distribution and remaining energy, and electing a cluster head node for each cluster; the cluster head node is used for intra-cluster coordination and inter-cluster multi-hop forwarding; the cluster head node forwards the aggregated data to the sink node through a multi-hop path, the sink node being equipped with an underwater acoustic modem and an radio frequency modem; each sensor node is equipped with a half-duplex underwater acoustic modem; in the underwater acoustic sensor network model, for each hop of data transmission, the sender uses an XOR-based encoding method to generate an encoded data packet from the local raw data and transmits it to the next-hop node; after successfully receiving and decoding, the intermediate node re-encodes and forwards the recovered data to complete the data transmission.

[0070] In one embodiment, the provided dual-sensor online fountain code transmission method based on deep reinforcement learning may further include a reinforcement learning optimization process. Specifically, this process includes: a recursive relationship between the online fountain code recovery rate and the buffer occupancy rate, modeling the online fountain code decoding process as a Markov decision process; calculating and determining the actual upper limit of the coding degree based on a preset recovery rate threshold and recovery gain increment; selecting the candidate coding degree as the coding degree when it is less than or equal to the actual upper limit, otherwise selecting the actual upper limit; and training the agent using a two-stage training mechanism to generate the optimal coding degree sequence.

[0071] In one embodiment, the provided dual-sensor online fountain encoding transmission method based on deep reinforcement learning may further include a two-stage training process. The two-stage training mechanism includes an offline preparation stage and an online training stage. The specific process includes: in the offline preparation stage, a strategy is initialized based on a simulated decoding trajectory with multiple recovery rate intervals; in the online training stage, a greedy strategy is adopted, the agent interacts iteratively with the simulated environment, a transition tuple is generated and stored in the experience replay buffer, and the network is updated through stochastic gradient descent until the network converges to the optimal encoding strategy.

[0072] In one embodiment, the provided dual-sensor online fountain coding transmission method based on deep reinforcement learning may further include an initialization process, specifically including: defining a state space and an action space, and designing a reward function; wherein, the state space includes the recovery rate interval, the expected buffer occupancy, the current coding degree, the check node, and the degree distribution of undecoded symbols; the action space is a set of coding degrees; the reward function balances the transmission overhead and intermediate recovery rate through weight coefficients, and imposes a penalty on actions that do not increase the coding degree.

[0073] In one embodiment, a dual-sensor online fountain coding transmission method based on deep reinforcement learning may further include a process of generating coded symbols. The specific process includes: when generating coded packets, simultaneously sensing network environment constraint information from the network modeling module and decoding state information from the decoding end, and dynamically adjusting the degree distribution to generate coded symbols according to the optimal coding degree sequence output by the reinforcement learning optimization module.

[0074] In one embodiment, a dual-sensor online fountain encoding transmission method based on deep reinforcement learning may further include an iterative process for recovering the original data. The specific process includes: controlling the sender to generate and send encoded packets according to the optimal encoding degree sequence, and after receiving feedback information triggered by the decoding end meeting preset conditions, switching to the next encoding degree in the optimal encoding degree sequence, and iterating until the data is completely recovered.

[0075] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0076] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A dual-sensing online fountain coding transmission system based on deep reinforcement learning, applied to an underwater wireless sensor network, characterized in that, The system comprises a network modeling module, a reinforcement learning optimization module and an encoding transmission execution module which are sequentially constructed and cooperatively operated, wherein: The network modeling module is configured to construct a three-dimensional layered and clustered underwater acoustic sensor network model by using a fuzzy clustering algorithm according to the spatial distribution and residual energy of each sensor node in the underwater wireless sensor network, and establish a transmission mechanism based on half-duplex underwater acoustic communication. The reinforcement learning optimization module is configured to take the underwater acoustic sensor network model and the transmission mechanism as environmental inputs, model the decoding process of the online fountain code as a Markov decision process, train an agent by combining a deep learning algorithm, and generate an optimal encoding degree sequence. The encoding transmission execution module is configured to execute the optimal encoding degree sequence to generate an encoded data packet and perform multi-hop transmission, and control the encoding degree switching based on limited feedback triggering until the sink node restores all original data.

2. The dual-sensing online fountain code transmission system based on deep reinforcement learning according to claim 1, wherein, The network modeling module is further configured to divide the sensor nodes into clusters according to the spatial distribution and residual energy, and elect a cluster head node for each cluster; the cluster head node is configured to perform intra-cluster coordination and inter-cluster multi-hop forwarding. The cluster head node forwards the aggregated data to the sink node through a multi-hop path, and the sink node is equipped with an underwater acoustic modem and a radio frequency modem; each sensor node is equipped with a half-duplex underwater acoustic modem. In the underwater acoustic sensor network model, at each hop of data transmission, the sender generates an encoded data packet from the local original data by using an encoding method based on XOR, and transmits it to the next hop node. The intermediate node re-encodes and forwards the restored data after successful reception and decoding, completing the data transmission.

3. The dual-sensing online fountain code transmission system based on deep reinforcement learning according to claim 1, wherein, The reinforcement learning optimization module comprises a Markov decision process modeling unit, a degree space pruning unit and a deep network training unit, wherein: The Markov decision process modeling unit is configured to model the decoding process of the online fountain code as a Markov decision process based on the recursive relationship between the recovery rate and the buffer occupancy rate of the online fountain code. The degree space pruning unit is configured to calculate and determine the actual upper limit value of the encoding degree according to the preset recovery rate threshold and recovery gain increment, select the candidate encoding degree as the encoding degree number when the candidate encoding degree is less than or equal to the actual upper limit value, or select the actual upper limit value as the encoding degree number. The deep network training unit is configured to train the agent by using a two-stage training mechanism to generate an optimal encoding degree sequence.

4. The dual-sensing online fountain code transmission system based on deep reinforcement learning according to claim 3, wherein, The two-stage training mechanism comprises an offline preparation stage and an online training stage, wherein: In the offline preparation stage, the simulation decoding trajectory based on multiple recovery rate intervals is used to initialize the strategy. In the online training stage, a greedy strategy is used, the agent iteratively interacts with the simulation environment, the transition tuples are stored in the experience replay buffer, the network is updated by stochastic gradient descent, and the network converges to the optimal encoding strategy.

5. The dual-sensing online fountain code transmission system based on deep reinforcement learning according to claim 3, wherein, The reinforcement learning optimization module further comprises an initialization unit configured to define the state space, the action space and design the reward function. The state space comprises a recovery rate interval, an expected buffer occupancy, a current encoding degree, a check node, and a degree distribution of undecoded symbols; the action space is an encoding degree set; and the reward function balances transmission overhead and intermediate recovery rate by a weight coefficient and imposes a penalty on an action that does not increase the encoding degree.

6. The dual-sensing online fountain code transmission system based on deep reinforcement learning according to claim 1, wherein, The encoding transmission execution module comprises a dual-sensing online fountain encoder configured to simultaneously sense network environment constraint information from the network modeling module and decoding state information from a decoding end when generating an encoded packet, and dynamically adjust a degree distribution to generate encoded symbols according to the optimal encoding degree sequence output by the reinforcement learning optimization module.

7. The dual-sensing online fountain code transmission system based on deep reinforcement learning according to claim 1, wherein, The encoding transmission execution module further comprises a limited feedback iterative transmission controller configured to control a sender to generate and send encoded packets according to the optimal encoding degree sequence, and switch to a next encoding degree in the optimal encoding degree sequence after receiving feedback information triggered by the decoding end satisfying a preset condition, and iteratively perform until data is completely recovered. 8.A dual-sensing online fountain code transmission method based on deep reinforcement learning, applied to the dual-sensing online fountain code transmission system based on deep reinforcement learning in any of claims 1-7, characterized in that, The method comprises: obtaining spatial distribution and residual energy of sensor nodes, constructing a three-dimensional layered clustering topology of an underwater acoustic sensor network model by using a fuzzy clustering algorithm, and establishing a multi-hop transmission mechanism based on half-duplex underwater acoustic communication; modeling a decoding process of an online fountain code as a Markov decision process, defining a state space, an action space, and a reward function, pruning and optimizing the action space, and training the action space by using a deep learning algorithm to obtain an optimal encoding degree sequence; during network transmission, an encoder generates encoded data packets based on the optimal encoding degree sequence and performs multi-hop transmission; and a decoder sends limited feedback to trigger encoding degree switching according to a preset feedback condition until all original data packets are successfully recovered at a sink node.

9. The dual-sensing online fountain code transmission method based on deep reinforcement learning according to claim 8, characterized in that, The method further comprises: dividing the sensor nodes into clusters according to spatial distribution and residual energy, and electing a cluster head node for each cluster; the cluster head node is configured to perform intra-cluster coordination and inter-cluster multi-hop forwarding; the cluster head node forwards aggregated data to a sink node through a multi-hop path; the sink node is equipped with an underwater acoustic modem and a radio frequency modem; each sensor node is equipped with a half-duplex underwater acoustic modem; in the underwater acoustic sensor network model, a sender generates encoded data packets from local original data by using an encoding method based on exclusive OR at each hop of data transmission, and transmits the encoded data packets to a next hop node; an intermediate node re-encodes and forwards recovered data after successfully receiving and decoding the data, thereby completing data transmission.

Images