Intelligent compression method for underwater acoustic sensor network based on reinforcement learning

Abstract

The application discloses a kind of underwater acoustic sensor network intelligent compression methods based on reinforcement learning, comprising: initialization experience replay buffer D, the network parameter of reinforcement learning intelligent agent;Node carries out real-time perception and feature extraction of local transmission state, obtains the state vector of current time, inputs it into the intelligent agent trained, and the original data packet is compressed with the corresponding compression ratio after optimal compression action of current intelligent decision, and is sent to the next hop node through underwater acoustic channel;After completion, the node obtains environmental feedback information, and the state vector of next time is generated according to the calculation of instant reward value;The transition sample of this interaction is stored in D, and is randomly sampled from D, to minimize loss function as the goal training intelligent agent, until loss function converges.The application can dynamically select optimal compression ratio according to local state, improve data packet delivery rate, reduce transmission delay and energy consumption simultaneously.

Description

Technical Field

[0001] This invention relates to the field of underwater acoustic communication and data transmission technology, and in particular to an intelligent compression method for underwater acoustic sensor networks based on reinforcement learning, which is mainly used in underwater acoustic sensor network systems with limited bandwidth, unstable channel quality, and limited energy. Background Technology

[0002] With the rapid development of applications such as underwater observation, marine resource exploration, and environmental monitoring, the deployment scale of underwater acoustic sensor networks is constantly expanding, and the amount of data transmitted within these networks is continuously increasing. However, underwater acoustic channels have inherent characteristics such as extremely low bandwidth, long propagation delay, highly time-varying channel characteristics, and severely limited node energy. Data packet transmission length has become a key bottleneck affecting network transmission reliability, end-to-end latency, and energy consumption. Traditional underwater acoustic data transmission methods, such as direct transmission of raw data and static compression transmission with a fixed ratio, often suffer from low transmission efficiency, frequent collisions and retransmissions, and rapid depletion of node energy when facing dynamically changing channel conditions and traffic loads due to their inability to flexibly adjust data packet length.

[0003] Adaptive compression, a technique that dynamically adjusts compression intensity based on network conditions, has garnered significant attention in recent years. Existing heuristic adaptive compression methods adjust compression ratios based on preset thresholds for queue length or channel quality, thus improving transmission performance to some extent. However, these methods rely on manually designed empirical rules, making it difficult to generalize and achieve optimal decision-making in complex and dynamic underwater environments. This is especially true in multi-hop network topologies where the local states of each node are highly coupled, and a single threshold rule cannot fully capture the dynamic trade-off between transmission reliability and resource consumption. Therefore, there is an urgent need for an intelligent decision-based underwater acoustic sensor network data compression method that can adaptively select the optimal compression level based on the real-time local transmission status of nodes (including queue congestion levels, channel quality fluctuations, remaining energy levels, retransmission counts, packet types, and original packet lengths), thereby minimizing end-to-end latency and energy consumption while ensuring successful packet delivery. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes an intelligent compression method for underwater acoustic sensor networks based on reinforcement learning. By modeling compression decisions as an online sequential decision problem, each node can act as an intelligent decision-making agent, dynamically selecting the optimal compression ratio based on its real-time perceived local state. Compared to traditional compression methods, this invention offers significant advantages in handling dynamic network environments, effectively improving data packet delivery rates, reducing transmission latency and energy consumption, and has broad application prospects in underwater acoustic communication, marine monitoring, and underwater Internet of Things (IoT).

[0005] The specific technical solution is as follows:

[0006] A smart compression method based on underwater acoustic sensor networks using reinforcement learning includes the following steps:

[0007] S1: Initialize the experience replay buffer D and the network parameters of the reinforcement learning agent;

[0008] S2: When a node receives the original data packet to be transmitted, it performs real-time perception and feature extraction of the local transmission status to obtain the current state vector.

[0009] S3: Input the state vector into the trained reinforcement learning agent, and make intelligent decisions to obtain the current optimal compression action;

[0010] S4: Compress the original data packet according to the compression ratio corresponding to the current optimal compression action;

[0011] S5: Send the compressed data packet to the next-hop node via the underwater acoustic channel;

[0012] S6: After this transmission is completed, the node obtains environmental feedback information, calculates the instant reward value based on it, and generates the state vector for the next moment.

[0013] S7: Store the transition samples from this interaction into D, randomly sample from D, and repeat S2-S7 iteratively to train the reinforcement learning agent with the goal of minimizing the loss function until the loss function converges.

[0014] Furthermore, the local transmission status includes: queue occupancy status, channel quality, node remaining energy, retransmission count, data packet type, and original data packet length;

[0015] The feature extraction method is to normalize the local transmission status.

[0016] Furthermore, in the reinforcement learning agent, a set of candidate compression ratios is preset. , among which, element Let η be the compression ratio of the m-th candidate, used to represent the compression level, where m = 1, 2, 3, ..., M, and M is the total number of candidate compression ratios; if η m =1 indicates that the data packet is transmitted without compression, η m The smaller the value, the higher the compression level.

[0017] The discrete action space design of the intelligent decision-making is as follows: Each compression action Corresponding to a discrete compression ratio , t=1,2,3,…,T.

[0018] Furthermore, in S3, the reinforcement learning agent selects the action corresponding to the optimal compression level from the predefined discrete action space A based on the current state vector as the optimal compression action; during the first iteration, the optimal compression action is selected according to the ε-greedy strategy.

[0019] Furthermore, the reinforcement learning agent uses a deep Q-network and selects the optimal compression action according to an ε-greedy strategy. Specifically, it randomly explores the compression ratio in the discrete action space with a probability of ε and selects the compression action with the largest current Q value with a probability of 1-ε.

[0020] Furthermore, in step S4, the compression process employs any lossless or lossy compression algorithm. By adjusting the parameters of the compression algorithm, the length of the compressed data packet is made equal to the product of the length of the original data packet and the compression ratio corresponding to the selected optimal compression action, rounded up.

[0021] Furthermore, in S6, the environmental feedback information includes: whether the data packet was successfully delivered, end-to-end latency, energy consumption, and whether there was packet loss.

[0022] Furthermore, the instant reward value is a weighted sum of the successful delivery indicator, time cost, energy consumption, reward function, and packet loss indicator, with the weight coefficients of the successful delivery indicator and reward function being positive and the weight coefficients of the remaining items being negative.

[0023] If the data packet is successfully delivered, the indicator for the successful delivery indication is 1; otherwise, it is 0. The indicator for the time cost is normalized delay, which is the ratio of the observed transmission delay associated with the current data packet to the delay normalization constant. The indicator for the energy consumption is normalized energy consumption, which is the ratio of the observed transmission energy consumption associated with the current data packet to the energy consumption preset normalization constant. The indicator for the reward function is the data retention rate. If packet loss occurs, the indicator for the packet loss indication is 1; otherwise, it is 0.

[0024] Furthermore, in S7, the transferred sample includes four parameters: the current state vector, the current optimal compression action, the immediate reward value, and the next state vector; the loss function is the Bellman loss function.

[0025] Furthermore, the reinforcement learning agent is deployed independently of the underwater acoustic sensor network, and the network parameters of the reinforcement learning agent, which are updated offline, are periodically synchronized to the network parameters of the data compression module in the underwater acoustic sensor network to complete the online deployment.

[0026] The beneficial effects of this invention are:

[0027] This invention models data packet compression as an online sequential decision problem using a reinforcement learning-based intelligent compression framework. This allows each node in the underwater acoustic sensor network to dynamically select the optimal compression action based on real-time perceived local transmission status, effectively reducing end-to-end latency and energy consumption while ensuring data packet delivery rate. This method overcomes the limitations of traditional fixed compression methods, which cannot adapt to dynamic channel changes, and the poor generalization ability of heuristic methods that rely on manual thresholds. Furthermore, through continuous interactive learning via a deep Q-network and an experience replay mechanism, the compression strategy is continuously optimized, further enhancing the system's adaptability and long-term stability in complex underwater environments. Attached Figure Description

[0028] Figure 1 This is a flowchart of the intelligent compression method based on underwater acoustic sensor network using reinforcement learning in an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram of the feedback learning and decision-making process of the intelligent compression architecture in an embodiment of the present invention. Detailed Implementation

[0030] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The objectives and effects of the present invention will become clearer as a result. The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0031] like Figure 1 and Figure 2 As shown, an intelligent compression method based on underwater acoustic sensor networks using reinforcement learning includes the following steps:

[0032] S1: Network parameters θ of the experience replay buffer D, the reinforcement learning agent (hereinafter referred to as the agent), and the network parameters of the data compression module in the underwater acoustic sensor network (hereinafter referred to as the target network parameters). Perform initialization, and in the initial state .

[0033] S2: State Awareness. When a node receives the raw data packet to be transmitted, the raw data packet enters the queue buffer. The node performs real-time awareness and feature extraction of its local transmission state to obtain the network transmission state vector at the current moment. It is used to characterize network congestion, link reliability, and transmission difficulty.

[0034] The local transmission status includes: queue occupancy status, channel quality, node remaining energy, retransmission count, data packet type, and original data packet length. In this embodiment, the feature extraction method is to normalize the local transmission status.

[0035] Specifically, the queue occupancy status is the length of the node's current cache queue, represented by a normalized queue length. This metric reflects the current buffer backlog and local congestion level of the node.

[0036] Channel quality is estimated based on statistics of the most recent successful transmissions, channel error statistics, or link quality measurements, and normalized to the [0,1] interval.

[0037] The remaining energy of a node is the ratio of its current remaining power to its initial power.

[0038] The retransmission count is the number of times the current data packet has been attempted to be sent, normalized to the maximum retransmission limit, reflecting the difficulty of real-time transmission.

[0039] Packet types include regular packets or control packets, represented by a type index, used to distinguish heterogeneous traffic with different importance or service requirements.

[0040] The length of the original data packet is the number of bits before compression, which directly determines the baseline transmission overhead before compression and is normalized to the maximum packet length.

[0041] S3: Intelligent decision-making yields the optimal compression action. This is achieved through the following sub-steps:

[0042] (3.1) Preset a set of candidate compression ratios Where M is the total number of candidate compression ratios, and the elements (m=1,2,3,…,M) represents the compression ratio of the m-th candidate, used to indicate the compression level; if η m =1 indicates that the data packet is transmitted without compression. m The smaller the value, the higher the compression level.

[0043] The discrete action space design for intelligent decision-making is as follows: Each compression action (t=1,2,3,…,T) corresponds to a discrete compression ratio. .

[0044] (3.2) Input the current state vector into the trained reinforcement learning agent. In this embodiment, the agent selects a deep Q-network (DQN) architecture. Based on the current state vector, the agent selects the action corresponding to the optimal compression level from the predefined discrete action space A as the optimal compression action. In particular, during the first iteration, the optimal compression action is selected according to the ε-greedy strategy, that is, the compression ratio in the discrete action space is randomly explored with probability ε, and the compression action with the largest current Q value is selected with probability 1-ε.

[0045] S4: Data Compression. Based on the compression ratio corresponding to the selected optimal compression action, the original data packet is compressed to obtain the compressed data packet, and the actual transmission length is calculated.

[0046] Specifically, the compression process employs any lossless or lossy compression algorithm. By adjusting the parameters of the compression algorithm, the length of the compressed data packet is made equal to the floor value of the product of the length of the original data packet and the compression ratio corresponding to the selected optimal compression action, as shown in the following expression:

[0047]

[0048] In the formula, This is the length of the compressed data packet. The compression ratio corresponding to the selected optimal compression action. The length of the original data packet.

[0049] S5: Data transmission. The compressed data packet is sent to the next-hop node via the underwater acoustic channel.

[0050] During data transmission, the transmission time of the compressed data packet (i.e., the channel occupancy time) is the ratio of the compressed data packet length to the effective transmission bit rate of the underwater acoustic channel, expressed as follows:

[0051]

[0052] In the formula, Channel occupancy time, This represents the effective transmission bit rate of the underwater acoustic channel.

[0053] S6: Feedback Calculation. After this transmission is completed, the node obtains environmental feedback information, including: whether the data packet was successfully delivered, end-to-end latency, energy consumption, and whether there was packet loss. Using this environmental feedback information as indicators, the node calculates an immediate reward value to quantify the merits of the current compression strategy; simultaneously, it adjusts the current state vector based on the environmental feedback information to generate the state vector for the next time step.

[0054] Specifically, the expression (i.e., the reward function) for calculating the immediate reward value using environmental feedback information as an indicator is as follows:

[0055]

[0056] In the formula, r t For immediate reward values, α1, α2, α3, α4, and α5 are all non-negative weighting coefficients used to balance the relative importance of different items; This is a successful delivery indication. The value is 1 if the data packet is successfully delivered, and 0 otherwise. This item is used to prompt the agent to select the appropriate action. Normalized delay is used to penalize behaviors that result in excessively high time costs. , It is the observed transmission delay associated with the current data packet. The delay normalization constant is preset by the user; Normalized energy consumption is used to penalize behaviors that result in excessive energy expenditure. , It is the energy consumption of observed transmission related to the current data packet. Similarly, the energy consumption normalization constant is preset by humans; For data retention rate; This is a packet loss indicator; the value is 1 if packet loss occurs, and 0 otherwise.

[0057] S7: Experience learning and compression strategy update. This is implemented through the following sub-steps:

[0058] (7.1) Generate the transfer sample for this interaction. , where s t Let a be the current state vector. t For compressing actions at the current moment, r t For instant reward value, s t+1 Let be the state vector for the next moment; store the current transition sample into the experience replay buffer D.

[0059] (7.2) Randomly sample a small batch of samples from the experience replay buffer D to train the agent offline, and update the agent’s network parameters θ by minimizing the loss function.

[0060] In this embodiment, the Bellman loss function is selected, and its expression is as follows:

[0061]

[0062] In the formula, Here, θ is the discount factor, Q represents the depth of the Q-network, and θ represents the network parameters before the update. These are the updated network parameters.

[0063] (7.3) Determine whether the loss function has converged. If yes, output the compression strategy obtained by the agent using the network parameters at this time; if not, use the agent with the updated network parameters and return to S2 for iteration. During the convergence of the loss function, the compression strategy gradually converges to the optimal value. After training, the optimal compression strategy is output. This strategy can be used by nodes to autonomously decide the optimal compression strategy based on the real-time status in actual deployment, thereby improving transmission reliability and reducing latency and energy consumption.

[0064] In practical applications, due to limited node resources and the independent deployment of the agent from the underwater acoustic sensor network, it is impossible to apply the agent with updated network parameters to the compression strategy generation in real time. Therefore, it is necessary to perform periodic offline updates on the agent. That is, each iteration (i.e., S2-S7) is counted as one step, and the offline update cycle is C steps. Every C steps, the network parameters θ of the agent are synchronized to the target network. Complete online deployment.

[0065] The method of this invention forms a closed loop from state perception, intelligent decision-making, compressed transmission to feedback learning. Through continuous environmental interaction, new transfer samples are continuously added to the experience replay buffer, and the agent is fine-tuned and trained periodically. This enables the node to adaptively adjust the compression strategy in a dynamic underwater acoustic environment to adapt to the dynamic changes in the agent's environment, achieving continuous learning and online adaptive optimization, and effectively balancing transmission reliability, end-to-end latency and energy consumption.

[0066] The above-described intelligent compression method for underwater acoustic sensor networks using reinforcement learning is merely an example of this invention. This invention is applicable to any adaptive compression method utilizing state awareness and reinforcement learning feedback mechanisms, including but not limited to variations employing different reinforcement learning algorithms, different state space designs (i.e., the design for calculating state vectors in S2), different reward function configurations, or different compression execution methods. These variations do not constitute a limitation on this invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0067] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A smart compression method based on underwater acoustic sensor networks using reinforcement learning, characterized in that, Includes the following steps: S1: Initialize the experience replay buffer D and the network parameters of the reinforcement learning agent; S2: When a node receives the original data packet to be transmitted, it performs real-time perception and feature extraction of the local transmission status to obtain the current state vector. S3: Input the state vector into the trained reinforcement learning agent, and the agent makes an intelligent decision to obtain the current optimal compression action; the reinforcement learning agent has a preset set of candidate compression ratios. , among which, element Let η be the compression ratio of the m-th candidate, used to represent the compression level, where m = 1, 2, 3, ..., M, and M is the total number of candidate compression ratios; if η m =1 indicates that the data packet is transmitted without compression, η m The smaller the value, the higher the compression level. The discrete action space design for the intelligent decision-making is as follows: Each compression action Corresponding to a discrete compression ratio , t=1,2,3,…,T; In step S3, the reinforcement learning agent selects the action corresponding to the optimal compression level from the predefined discrete action space A based on the current state vector as the optimal compression action; in the first iteration, the optimal compression action is selected according to the ε-greedy policy. S4: Compress the original data packet according to the compression ratio corresponding to the current optimal compression action; S5: Send the compressed data packet to the next-hop node via the underwater acoustic channel; S6: After this transmission is completed, the node obtains environmental feedback information, calculates the instant reward value based on it, and generates the state vector for the next moment. S7: Store the transition samples from this interaction into D, randomly sample from D, and repeat S2-S7 iteratively to train the reinforcement learning agent with the goal of minimizing the loss function until the loss function converges.

2. The intelligent compression method based on reinforcement learning underwater acoustic sensor network according to claim 1, characterized in that, The local transmission status includes: queue occupancy status, channel quality, node remaining energy, retransmission count, data packet type, and original data packet length; The feature extraction method is to normalize the local transmission status.

3. The intelligent compression method based on underwater acoustic sensor network according to claim 1, characterized in that, The reinforcement learning agent uses a deep Q-network and selects the optimal compression action according to an ε-greedy strategy. Specifically, it randomly explores the compression ratio in the discrete action space with a probability of ε and selects the compression action with the largest current Q value with a probability of 1-ε.

4. The intelligent compression method based on reinforcement learning underwater acoustic sensor network according to claim 1, characterized in that, In step S4, the compression process employs any lossless or lossy compression algorithm. By adjusting the parameters of the compression algorithm, the length of the compressed data packet is made equal to the product of the length of the original data packet and the compression ratio corresponding to the selected optimal compression action, rounded up.

5. The intelligent compression method based on reinforcement learning underwater acoustic sensor network according to claim 1, characterized in that, In S6, the environmental feedback information includes: whether the data packet was successfully delivered, end-to-end latency, energy consumption, and whether there was packet loss.

6. The intelligent compression method based on underwater acoustic sensor network according to claim 1, characterized in that, The instant reward value is a weighted sum of the successful delivery indicator, time cost, energy consumption, reward function, and packet loss indicator. The weight coefficients of the successful delivery indicator and the reward function are positive, while the weight coefficients of the other items are negative. If the data packet is successfully delivered, the indicator for the successful delivery indication is 1; otherwise, it is 0. The indicator for the time cost is normalized latency, which is the ratio of the observed transmission latency associated with the current data packet to the latency normalization constant. The indicator for the energy consumption is normalized energy consumption, which is the ratio of the observed transmission energy consumption associated with the current data packet to the energy consumption preset normalization constant. The indicator for the reward function is the data retention rate. If packet loss occurs, the index of the packet loss indicator is set to 1; otherwise, it is set to 0.

7. The intelligent compression method based on underwater acoustic sensor network according to claim 1, characterized in that, In S7, the transferred sample includes four parameters: the current state vector, the current optimal compression action, the immediate reward value, and the next state vector; the loss function is the Bellman loss function.

8. The intelligent compression method based on underwater acoustic sensor network according to claim 1, characterized in that, The reinforcement learning agent is deployed independently of the underwater acoustic sensor network. The network parameters of the reinforcement learning agent, which are updated offline, are periodically synchronized to the network parameters of the data compression module in the underwater acoustic sensor network to complete the online deployment.

Images