Intelligent decision method for cross-sea-air transmission parameters based on reinforcement learning and related equipment

Abstract

The application discloses a cross-sea-air transmission parameter intelligent decision-making method based on reinforcement learning and related equipment, and relates to the technical field of cross-sea-air communication. In the application, a cross-sea-air channel model is constructed, and the cross-sea-air channel model is used as a dynamic environment of reinforcement learning. Reinforcement learning is used instead of expert rules to intelligently decide cross-sea-air transmission parameters according to environmental changes, thereby improving the communication performance of the system. The environmental reward, action mapping and cross-sea-air channel state are integrated into experience and stored in an experience replay pool. The experience in the experience replay pool is used to train an intelligent agent. The intelligent agent outputs an optimal cross-sea-air transmission parameter strategy after training. The modulation mode, coding mode and transmission sound source level are jointly decided, thereby improving the adaptive management and decision-making capability of the cross-sea-air communication system.

Description

Technical Field

[0001] This invention relates to the field of cross-sea and air communication technology, specifically to a method and related equipment for intelligent decision-making of cross-sea and air transmission parameters based on reinforcement learning. Background Technology

[0002] Cross-sea and air-based three-dimensional collaboration has become the mainstream form of marine Internet of Things and integrated air-sea networks. In this context, accurate decision-making on cross-sea and air communication parameters is the foundation for ensuring the execution of various tasks and is also a key technical aspect. However, cross-sea and air links face complex influences from the dynamic coupling of multiple factors such as weather, sea state, Doppler effect, and sudden interference. Traditional parameter configuration methods based on static tables or manual experience often lead to decreased spectrum utilization, increased energy consumption, or even link interruption due to matching lag when the environment changes drastically. There is an urgent need to introduce a new decision-making mechanism with online learning and autonomous optimization capabilities.

[0003] Intelligent decision-making, by mapping the environment, services, and equipment states to the optimal parameter combination in real time and utilizing intelligent algorithms to achieve adaptive decision-making, reduces reliance on human experience and prior knowledge bases, becoming an important research direction for improving the performance of cross-sea and air communication systems. Existing research has made initial progress in empowering communication parameter decision-making with reinforcement learning, such as using reinforcement learning and federated learning to collaboratively schedule multi-dimensional resources in wireless edge networks, or utilizing improved reinforcement learning algorithms to cope with external malicious interference and complex electromagnetic environments, thus improving the system's adaptability and anti-interference capabilities. However, these studies are mostly concentrated on terrestrial or specific wireless scenarios, and exploration specifically for intelligent decision-making of cross-sea and air transmission parameters remains insufficient.

[0004] Comprehensive analysis shows that existing reinforcement learning-based solutions mostly focus on dealing with single-dimensional or specific disturbances. In the highly dynamic and multi-variable coupled complex scenario of cross-sea and air communication, they struggle to achieve global joint optimization of transmission parameters and systematic intelligent decision-making. Most existing cross-sea and air information transmission systems still rely on fixed parameters or traditional decision-making algorithms, lacking a systematic intelligent decision-making framework capable of coping with the dynamic environment of integrated sea and air communication.

[0005] Therefore, in dynamic cross-sea and air scenarios, there is an urgent need for a novel decision-making framework that can integrate channel models online into a learning loop. In existing technologies, traditional rule-based or lookup table methods rely on offline strategies developed by experts, which cannot adapt to real-time environmental changes, leading to decision lag and decreased communication performance. Existing reinforcement learning methods are often limited to single-dimensional outputs, making it difficult to achieve multi-parameter collaborative optimization in complex cross-sea and air environments, and systematic research for this scenario is lacking. Therefore, this invention aims to propose an intelligent decision-making method for cross-sea and air transmission parameters based on reinforcement learning, to reduce reliance on human experience and improve the system's adaptability and robustness in complex sea and air environments. Summary of the Invention

[0006] Based on the problems raised in the background technology above, the purpose of this invention is to provide a method and related equipment for intelligent decision-making of cross-sea and air transmission parameters based on reinforcement learning. This solves the problems that traditional rule or lookup table methods rely on offline strategies formulated by experts, which cannot adapt to real-time changes in the environment, resulting in decision lag and decreased communication performance. Furthermore, existing reinforcement learning methods are often limited to a single-dimensional output, making it difficult to achieve multi-parameter collaborative optimization in complex cross-sea and air environments, and there is a lack of systematic research for this scenario.

[0007] This invention is achieved through the following technical solution:

[0008] The first aspect of this invention provides an intelligent decision-making method for cross-sea and air transport parameters based on reinforcement learning, comprising the following steps:

[0009] Construct a cross-sea air channel model and treat the cross-sea air channel model as a dynamic environment;

[0010] An intelligent agent is constructed, which obtains the current cross-sea-air environment and the transmission parameters for decision-making under the environment by observing the dynamic environment, and constructs the current cross-sea-air environment and the transmission parameters as the cross-sea-air channel state;

[0011] The intelligent agent performs action mapping of transmission parameters based on the state of the cross-sea and air channel.

[0012] The environmental reward is calculated based on the action mapping result. The environmental reward, action mapping and cross-sea-air channel status are integrated into an experience and stored in the experience playback pool.

[0013] The agent is trained using the experience in the experience replay pool, and the trained agent outputs the optimal transmission parameter strategy across sea and air.

[0014] In the above technical solution, firstly, a cross-sea-air channel model is constructed. This cross-sea-air channel model is used as a dynamic environment for reinforcement learning. Reinforcement learning is used to replace expert rules to intelligently decide cross-sea-air transmission parameters according to environmental changes, thereby improving the communication performance of the system.

[0015] Secondly, an intelligent agent is constructed. This agent perceives the communication environment of the cross-sea-air channel model, thereby obtaining the current cross-sea-air environment and the transmission parameters for decision-making under this environment. The current cross-sea-air environment and the transmission parameters for decision-making under this environment are mapped into the state space of the intelligent agent, generating the cross-sea-air channel state.

[0016] Then, the agent determines the cross-sea-air communication action based on the transmission parameters in the cross-sea-air channel state, so as to complete the action mapping of the transmission parameters. The action obtained through the action mapping is the cross-sea-air transmission parameter suitable for communication transmission in the cross-sea-air channel state.

[0017] Next, based on the environmental state after executing the cross-sea and air transmission parameters, the agent calculates the environmental reward in that environmental state, which is used to guide the agent's learning.

[0018] Finally, environmental rewards, action mapping, and cross-sea and air channel status are integrated into experience and stored in the experience replay pool. The experience in the experience replay pool is used to train the agent. The trained agent outputs the optimal cross-sea and air transmission parameter strategy, realizing joint decision-making on multiple parameters such as modulation mode, coding mode, and transmission source level, thereby improving the adaptive management and decision-making capabilities of the cross-sea and air communication system.

[0019] In one alternative embodiment, constructing a cross-sea-air channel model includes:

[0020] Calculate the empty segment loss and calculate the sea segment loss based on Urick;

[0021] The complex amplitude of the empty section is calculated based on the empty section loss, and the complex amplitude of the sea section is calculated based on the sea section loss;

[0022] The overall complex amplitude is calculated using the complex amplitude of the empty segment and the complex amplitude of the sea segment, and the total path loss is determined based on the overall complex amplitude;

[0023] Obtain the transmit power, transmit end gain, receive end gain, and noise power, and then convert the transmit power, transmit end gain, receive end gain, and noise power.

[0024] In one optional embodiment, the current cross-sea-air environment and the transmission parameters are used to construct a cross-sea-air channel state, including:

[0025] The transmitted sound source level is calculated using the sea segment model in the cross-sea air channel model and the received sound source level;

[0026] The current cross-sea-air environment, together with the transmitted sound source level, modulation method, and coding rate, are used to construct the cross-sea-air channel state.

[0027] In one optional embodiment, the agent performs action mapping of transmission parameters based on the cross-sea-air channel state, including:

[0028] The selection of the emission source level, the modulation method, and the coding rate is used as the selection action;

[0029] The action of selecting the emission source level, the action of selecting the modulation mode, and the action of selecting the coding rate are integrated into an action set;

[0030] Action mapping for transmitting parameters based on actions in the action set.

[0031] In an optional embodiment, the selection of the transmitted sound source level, the modulation scheme, and the coding rate is used as a selection action, including:

[0032] The selection of the emission source level is defined as the selection action of the emission source level, and the selection action of the emission source level comprises a total of [number] actions. kind;

[0033] Selecting the modulation order is used as the modulation mode selection action; the modulation mode selection action comprises a total of... kind;

[0034] The selection of the coding rate is used as the coding rate selection action, and the coding rate selection action comprises a total of kind.

[0035] In one alternative embodiment, calculating an environmental reward based on the result of the action mapping includes:

[0036] The bit error rate is normalized to obtain the normalized bit error rate.

[0037] The modulation order and coding rate are determined by motion mapping, and the successfully transmitted data is calculated using the modulation order and coding rate.

[0038] An environmental reward is obtained by calculating the normalized bit error rate and the successfully transmitted data using a reward function.

[0039] In one optional embodiment, the reward function includes:

[0040] Data transmission efficiency is determined by the number of successfully transmitted data.

[0041] Determine the bit error rate:

[0042] When the bit error rate is greater than the bit error rate threshold, a negative reward function is constructed using the bit error rate;

[0043] When the bit error rate is less than or equal to the bit error rate threshold, a positive reward function is constructed using the data transmission efficiency and the normalized bit error rate.

[0044] A second aspect of the present invention provides an intelligent decision-making system for cross-sea and air transport parameters based on reinforcement learning, comprising:

[0045] The environment module is used to construct a cross-sea air-channel model and to use the cross-sea air-channel model as a dynamic environment.

[0046] The state module is used to construct an intelligent agent. The intelligent agent obtains the current cross-sea-air environment and the transmission parameters for decision-making under the environment by observing the dynamic environment, and constructs the current cross-sea-air environment and the transmission parameters into a cross-sea-air channel state.

[0047] Action module, used by the agent to perform action mapping of transmission parameters according to the state of the cross-sea and air channel;

[0048] The experience module is used to calculate environmental rewards based on the results of action mapping, and integrate the environmental rewards, action mapping and cross-sea-air channel status into experience and store them in the experience playback pool.

[0049] The training module is used to train the agent using the experience in the experience replay pool, and outputs the optimal transmission parameter strategy across sea and air through the trained agent.

[0050] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a reinforcement learning-based intelligent decision-making method for cross-sea and air transport parameters.

[0051] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a reinforcement learning-based intelligent decision-making method for cross-sea and air transport parameters.

[0052] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0053] 1. Construct a cross-sea-air channel model and use this model as a dynamic environment for reinforcement learning. Reinforcement learning is used to replace expert rules to intelligently decide cross-sea-air transmission parameters according to environmental changes, thereby improving the communication performance of the system.

[0054] 2. Integrate environmental rewards, action mapping, and cross-sea and air channel status into experience and store it in the experience replay pool. Use the experience in the experience replay pool to train the agent. The trained agent outputs the optimal cross-sea and air transmission parameter strategy, realizes joint decision-making on multiple parameters such as modulation mode, coding mode, and transmission source level, and improves the adaptive management and decision-making capability of the cross-sea and air communication system. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0056] Figure 1 This is a schematic diagram of the intelligent decision-making method for cross-sea and air transmission parameters based on reinforcement learning provided in Embodiment 1 of the present invention;

[0057] Figure 2 This is a schematic diagram of the transmission parameter decision neural network provided in Embodiment 1 of the present invention;

[0058] Figure 3 This is the neural network training loss diagram provided in Embodiment 1 of the present invention;

[0059] Figure 4 This is a graph showing the change in neural network training rewards provided in Embodiment 1 of the present invention;

[0060] Figure 5 This is a graph showing the change in bit error rate of the system during the training process, provided in Embodiment 1 of the present invention.

[0061] Figure 6 This is the emission sound source level based on reinforcement learning decision-making provided in Embodiment 1 of the present invention;

[0062] Figure 7 This is the modulation order based on reinforcement learning decision-making provided in Embodiment 1 of the present invention;

[0063] Figure 8 This is the encoding rate based on reinforcement learning decision-making provided in Embodiment 1 of the present invention;

[0064] Figure 9 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0066] Embodiment 1 of this invention provides an intelligent decision-making method for cross-sea and air transport parameters based on reinforcement learning, such as... Figure 1 As shown, the intelligent decision-making method for cross-sea and air transport parameters based on reinforcement learning includes the following steps:

[0067] Construct a cross-sea air channel model and treat the cross-sea air channel model as a dynamic environment;

[0068] An intelligent agent is constructed, which obtains the current cross-sea-air environment and the transmission parameters for decision-making under the environment by observing the dynamic environment, and constructs the current cross-sea-air environment and the transmission parameters as the cross-sea-air channel state;

[0069] The intelligent agent performs action mapping of transmission parameters based on the state of the cross-sea and air channel.

[0070] The environmental reward is calculated based on the action mapping result. The environmental reward, action mapping and cross-sea-air channel status are integrated into an experience and stored in the experience playback pool.

[0071] The agent is trained using the experience in the experience replay pool, and the trained agent outputs the optimal transmission parameter strategy across sea and air.

[0072] It should be noted that traditional rule-based or table-lookup parameter configuration requires experts to formulate strategies offline, relying on human experience. In the face of dynamic cross-sea and air environments, it cannot be updated in real time, which can easily lead to decision lag and degraded communication performance. On the other hand, the output of reinforcement learning-based solutions is limited to a single dimension, making it difficult to obtain the global optimum. There is a lack of systematic research on joint optimization and intelligent decision-making of transmission parameters in complex cross-sea and air environments.

[0073] Therefore, in this embodiment, a method for intelligent decision-making of cross-sea and air transmission parameters based on reinforcement learning is provided. First, a cross-sea and air channel model is constructed, and this cross-sea and air channel model is used as a dynamic environment for reinforcement learning. Reinforcement learning is used to replace expert rules to intelligently decide cross-sea and air transmission parameters according to environmental changes, thereby improving the communication performance of the system.

[0074] Secondly, an intelligent agent is constructed. This agent perceives the communication environment of the cross-sea-air channel model, thereby obtaining the current cross-sea-air environment and the transmission parameters for decision-making under this environment. The current cross-sea-air environment and the transmission parameters for decision-making under this environment are mapped into the state space of the intelligent agent, generating the cross-sea-air channel state.

[0075] Then, the agent determines the cross-sea-air communication action based on the transmission parameters in the cross-sea-air channel state, so as to complete the action mapping of the transmission parameters. The action obtained through the action mapping is the cross-sea-air transmission parameter suitable for communication transmission in the cross-sea-air channel state.

[0076] Next, based on the environmental state after executing the cross-sea and air transmission parameters, the agent calculates the environmental reward in that environmental state, which is used to guide the agent's learning.

[0077] Finally, environmental rewards, action mapping, and cross-sea and air channel status are integrated into experience and stored in the experience replay pool. The experience in the experience replay pool is used to train the agent. The trained agent outputs the optimal cross-sea and air transmission parameter strategy, realizing joint decision-making on multiple parameters such as modulation mode, coding mode, and transmission source level, thereby improving the adaptive management and decision-making capabilities of the cross-sea and air communication system.

[0078] In one alternative embodiment, constructing a cross-sea-air channel model includes:

[0079] Calculate the empty segment loss and calculate the sea segment loss based on Urick;

[0080] The complex amplitude of the empty section is calculated based on the empty section loss, and the complex amplitude of the sea section is calculated based on the sea section loss;

[0081] The overall complex amplitude is calculated using the complex amplitude of the empty segment and the complex amplitude of the sea segment, and the total path loss is determined based on the overall complex amplitude;

[0082] Obtain the transmit power, transmit end gain, receive end gain, and noise power, and then convert the transmit power, transmit end gain, receive end gain, and noise power.

[0083] It should be noted that in a transoceanic system, the signal first propagates in the atmosphere (air segment) and then enters the seawater (sea segment) to complete acoustic transmission. Air segment propagation is affected by atmospheric refractive index, sea surface roughness, and the potential presence of negative refractive waveguides, while sea segment attenuation is mainly determined by geometric diffusion, seawater absorption, and bottom scattering. If traditional radio atmospheric models and Urick acoustic models are used for independent calculations, only the individual losses of the two segments can often be obtained, making it difficult to directly assess the overall link's delay, multipath propagation, and reliability.

[0084] Therefore, in this embodiment, by using a unified complex amplitude coupling method, the complex amplitude transfer function obtained in the air segment is multiplied by the linear amplitude of the sea segment to obtain a complete cross-sea air channel model.

[0085] Specifically, the first step is to determine whether the propagation is at line-of-sight or beyond:

[0086] ;

[0087] in, This is the limit of sight distance. , The height of the transmitting and receiving antennas relative to the sea level, measured in meters (m).

[0088] If the actual horizontal distance is less than or equal to the line-of-sight limit, a two-ray model (considering direct waves and ground-reflected waves) is used; otherwise, a three-ray model (adding diffraction or scattering) or ITU-R P.452 is used. ITU-R P.452 is suitable for beyond-line-of-sight, tropospheric propagation, considering complex factors such as atmospheric refraction, diffraction, scattering, and obstacles, and is suitable for frequencies of approximately 0.1–50 GHz.

[0089] Furthermore, when the actual horizontal distance is the line-of-sight distance, the gap loss is calculated as follows:

[0090] ;

[0091] ;

[0092] in, It is the loss of empty segments. It is the operating frequency. It is the sea surface reflectance, including the Fresnel reflectance and attenuation corrections for sea surface roughness. It's the wavelength. , It is the path difference between the direct wave and the mirror wave on the sea surface. This is the actual horizontal distance.

[0093] Calculate sea section loss:

[0094]

[0095] in, It is sea section loss, It is the expansion factor. It is the transmission distance, in units of , It is the absorption coefficient, with units of 1000 ppm. Frequency-dependent It is bottom scattering loss.

[0096] Calculate the complex amplitude of the empty section based on the empty section loss:

[0097] ;

[0098] in, It is the amplitude of the empty segment. It is a phase delay in the empty segment.

[0099] Calculate the complex amplitude of the sea section based on the sea section loss:

[0100] ;

[0101] in, It is the complex amplitude of the sea section.

[0102] The overall complex amplitude is obtained by multiplying the complex amplitude of the air segment and the complex amplitude of the sea segment. :

[0103] ;

[0104] Finally, the total path loss is calculated based on the overall complex amplitude:

[0105] ;

[0106] in, This is the total path loss.

[0107] Obtain the transmit power as The gain of the transmitter and receiver is and Noise power is The acceptable signal-to-noise ratio can be obtained. for:

[0108] ;

[0109] Cross-domain transmission systems can transmit stably under a certain signal-to-noise ratio. The optimal transmission performance of the system can be achieved by adjusting two transmission parameters: noise power and total path loss.

[0110] Therefore, in this embodiment, the agent adjusts two transmission parameters, noise power and total path loss, to achieve optimal transmission performance of the system.

[0111] In one optional embodiment, the current cross-sea-air environment and the transmission parameters are used to construct a cross-sea-air channel state, including:

[0112] The transmitted sound source level is calculated using the sea segment model in the cross-sea air channel model and the received sound source level;

[0113] The current cross-sea-air environment, together with the transmitted sound source level, modulation method, and coding rate, are used to construct the cross-sea-air channel state.

[0114] It should be noted that the reinforcement learning system consists of five parts: Agent, State, Reward, Action, and Environment. For cross-sea and air scenarios, this application treats the "cross-sea and air integrated channel" as a dynamic environment and quantifies it into measurable parameters such as noise, serving as the interface for reinforcement learning to interact with the real channel. The agent reads these parameters in each time slot, obtains environmental feedback, and continuously explores and adjusts transmission decisions such as the transmitted sound source level, modulation method, and coding rate accordingly.

[0115] In this embodiment, the emission sound source level The sound energy emitted by a sound source and propagating through a surface per unit time refers to the sound energy of the sound waves emitted by the source. The definition and specific calculation method are as follows:

[0116]

[0117] 170.8 is a constant determined by the reference sound power.

[0118] Received sound source level It is the sound pressure level received at a specific location. It describes the received intensity of the measuring instrument. The received sound source level is usually affected by factors such as distance and environment. The specific calculation method is as follows:

[0119]

[0120] in This is the propagation loss, calculated as follows:

[0121]

[0122] It is the expansion factor. It is the transmission distance, in units of , It is the absorption coefficient, with units of 1000 ppm. Frequency-dependent It is bottom scattering loss.

[0123] Specifically, the current signal-to-noise ratio is calculated based on the cross-sea-air channel model, forming environmental observations, which are then mapped to network inputs.

[0124] Based on environmental observations, a state mapping is performed. Since the system requires intelligent decision-making based on parameters including the transmitted sound source level, modulation scheme, and coding rate, making decisions for different environments, this embodiment constructs the cross-sea-air channel state by combining the current cross-sea-air environment with the transmitted sound source level, modulation scheme, and coding rate. To jointly characterize the real-time situation of intelligent agents in transoceanic and air channels, among which, It represents the current system environment, including channel quantization metrics such as noise intensity; Indicates the level of the emitted sound source; Indicates the modulation method. Indicates the encoding rate. It is the set of all states in a communication system.

[0125] In one optional embodiment, the agent performs action mapping of transmission parameters based on the cross-sea-air channel state, including:

[0126] The selection of the emission source level, the modulation method, and the coding rate is used as the selection action;

[0127] The action of selecting the emission source level, the action of selecting the modulation mode, and the action of selecting the coding rate are integrated into an action set;

[0128] Action mapping for transmitting parameters based on actions in the action set.

[0129] In an optional embodiment, the selection of the transmitted sound source level, the modulation scheme, and the coding rate is used as a selection action, including:

[0130] The selection of the emission source level is defined as the selection action of the emission source level, and the selection action of the emission source level comprises a total of [number] actions. kind;

[0131] Selecting the modulation order is used as the modulation mode selection action; the modulation mode selection action comprises a total of... kind;

[0132] The selection of the coding rate is used as the coding rate selection action, and the coding rate selection action comprises a total of kind.

[0133] It should be noted that, in order to ensure the signal-to-noise ratio at the receiving end, the transmitting sound source level is calculated using the sea segment model in the cross-sea-air channel model and the receiving sound source level. The selection action of the transmitting sound source level involves... Type; Modulation mode selection action Types, including but not limited to BPSK, QPSK, 8PSK, and 16PSK; a total of [number] coding rate selection actions. Types, including but not limited to 1 / 2 and 1 / 3.

[0134] Integrating the above selection actions, there are a total of Each action yields an action set. .

[0135] Taking action 9 in the action set as an example, the selection parameter index corresponding to action 9 is (2,1,0). The first index 2 indicates that the selected emission source level is the third level, the second index 1 indicates that the selected modulation order is 2, that is, QPSK modulation mode, and the third index 0 indicates that the selected coding rate is 1 / 2.

[0136] The agent maps transmission parameters, i.e., the strategy, based on the state of the cross-sea and air channel. In this embodiment, the strategy is adopted. The strategy selects the transmission parameters for this transmission under the cross-sea and air channel conditions. When updating the strategy in each time slot, the above three parameters, namely the transmitted sound source level, modulation order, and coding rate, are changed simultaneously, thereby outputting the Q value of discrete actions.

[0137] In one alternative embodiment, calculating an environmental reward based on the result of the action mapping includes:

[0138] The bit error rate is normalized to obtain the normalized bit error rate.

[0139] The modulation order and coding rate are determined by motion mapping, and the successfully transmitted data is calculated using the modulation order and coding rate.

[0140] An environmental reward is obtained by calculating the normalized bit error rate and the successfully transmitted data using a reward function.

[0141] It should be noted that the communication device completes a frame of data transmission according to the selected parameters, and the measured system bit error rate and successfully transmitted data are rewarded immediately.

[0142] Specifically, the bit error rate is normalized, the successfully transmitted data is calculated using the modulation order and coding rate, then weighting coefficients are selected to complete the design of the objective function. The objective function transforms the multi-objective problem into a single-objective problem, and the system parameters are configured to achieve optimal communication under the current environment.

[0143] The formula for normalizing the bit error rate is as follows:

[0144] ;

[0145] in, It is the normalized bit error rate. and These are the minimum and maximum bit error rates, respectively.

[0146] The calculation method for successful data transmission is as follows:

[0147] ;

[0148] in, This is the successful transmission rate. It is the bit error rate.

[0149] When the system bit error rate is greater than When the system bit error rate is less than or equal to 0, the number of successfully transmitted data is 0. When the modulation order and coding rate are used, the product of the modulation order and coding rate is used as the data for successful transmission. It is important to emphasize that the modulation order and coding rate here correspond to specific values. For example, the modulation order of QPSK is 2.

[0150] In one optional embodiment, the reward function includes:

[0151] Data transmission efficiency is determined by the number of successfully transmitted data.

[0152] Determine the bit error rate:

[0153] When the bit error rate is greater than the bit error rate threshold, a negative reward function is constructed using the bit error rate;

[0154] When the bit error rate is less than or equal to the bit error rate threshold, a positive reward function is constructed using the data transmission efficiency and the normalized bit error rate.

[0155] It should be noted that the reward function uses positive and negative rewards based on successfully transmitted data and the system error rate: positive rewards incentivize the agent to take better actions, while negative rewards penalize erroneous actions to prevent the agent from taking those actions. The specific reward function design is as follows:

[0156] ;

[0157] in, Indicates the reward value. Indicates data transmission efficiency. , It is a positive constant.

[0158] In this embodiment, the bit error rate threshold is When the bit error rate is greater than When the reward is negative, the system has a high error rate and poor communication performance. Therefore, the agent is penalized to avoid choosing this action next time. When the error rate is less than or equal to... When the system has a relatively low bit error rate and good communication performance, it is rewarded with data transmission efficiency and normalized bit error rate to encourage the agent to choose actions with high data transmission efficiency and low bit error rate.

[0159] The state, action, and reward are integrated and placed into the experience replay pool. After a certain amount of data is available in the experience replay pool, a small batch of data is sampled from the replay pool. The sampled data is used to train the estimation network, and the estimation Q network is updated through backpropagation of the gradient of the neural network. Every C steps, the parameters of the estimation network are copied to the target network, and the network is continuously trained according to the environment until the maximum number of iterations is reached. ε decays linearly from 0.99 to close to 0, and exploration gradually gives way to selecting actions with greater rewards.

[0160] Through the aforementioned closed loop of "environmental perception - state mapping - action output - reward calculation - network update", the reinforcement learning network DQN continuously updates the network parameters, ultimately obtaining an optimal parameter strategy that adapts to dynamic cross-sea and air channels.

[0161] The specific process is as follows:

[0162] Step 1: Sensing the transoceanic environment (i.e., Figure 1 The current perceived information in the data will be used to concatenate and map the cross-sea and air environment with transmission parameters into a state s (i.e., Figure 1 The current environment state in the context is used to map the communication parameters of the decision to action a. The values ​​Q corresponding to all states and actions are randomly initialized, and all parameters of the Q-network are estimated. Target Q network parameters Clear the set D of experience replays.

[0163] Step 2: for i from 1 to ( (where the maximum number of iterations is specified), and the iteration process is as follows:

[0164] a) Initialization Given the first state in the current state sequence, obtain its feature vector. ;

[0165] b) Using in Q networks As input, we obtain the Q-value outputs corresponding to all actions of the Q-network, and use them to... The greedy strategy selects the corresponding action from the estimated Q-value output. This refers to the currently selected transmission parameters.

[0166] c) In state Execute the current action , obtain a new state corresponding feature vector Rewards for both bit error rate and successfully transmitted data (Right now Figure 1 (Feedback rewards) and whether the status has been terminated. ;

[0167] d) will This quintuple is placed into the experience replay set D (where the first four parameters of the quintuple are...). Figure 1 Experience group 1);

[0168] e) Update the environment;

[0169] f) Sample m samples from the experience replay set D. Calculate the estimated Q value :

[0170] ;

[0171] in, This is the reward decay factor, typically between 0 and 1;

[0172] g) Using the mean squared error loss function The parameters of the estimated Q-network are updated through backpropagation of the gradients of the neural network. ;

[0173] h) If , It estimates the update interval between the target network and the target network, and updates the target Q-network parameters. ;

[0174] i) If the current iteration number i is equal to the maximum iteration number Once the current iteration is complete, the transmission parameter strategy in the dynamic environment is obtained; otherwise, proceed to step b) to continue learning through interaction with the environment.

[0175] The neural network structure in this embodiment is as follows: Figure 2As shown, the system consists of two fully connected neural networks and a ReLU activation function. The neural network contains two Q-networks: an estimation Q-network and a target Q-network. The two networks have identical structures, and the parameters of the estimation Q-network are periodically copied to the target Q-network to update its parameters. The network input is the current state, which mainly includes four features: environmental state, emitted sound source level, modulation scheme, and coding rate. These features are extracted through two fully connected layers and input to the input layer. The output layer outputs the predicted Q-values ​​corresponding to different actions. Then, based on the output of the neural network and... The method involves selecting actions.

[0176] Figure 3 and Figure 4 This refers to the training loss (number of training iterations versus loss) and reward changes (number of training iterations versus reward) of the communication intelligent decision-making neural network. During training, to ensure sufficient data in the experience replay region and avoid overfitting of network parameters to the initial data, training begins after 200 interactions with the environment. After 100 iterations, the model's loss remains relatively low, the model converges quickly, and loss fluctuations are relatively small, indicating good performance of the trained neural network model. As the number of training iterations increases, the reward function value gradually increases. After 400 iterations, the reward function maintains a high level, indicating that the model has gradually learned a better strategy during training, thus enabling it to obtain higher rewards.

[0177] Figure 5 The system bit error rate (BER) changes during the transmission parameter decision training process (number of training sessions and system BER). The overall system BER shows a decreasing trend. When the number of training sessions is small, the BER is high. As the number of training sessions increases, the BER gradually decreases. The system can maintain good communication performance and the communication is relatively reliable.

[0178] Figure 6 (Environmental noise level and emitted sound source level p) Figure 7 (Ambient noise level and modulation order) Figure 8 (Environmental noise level and coding rate) represent the decision-making results based on neural networks under different environments. When noise increases, the emission source level of the neural network decision increases accordingly to enhance the signal-to-noise ratio and thus ensure communication reliability. The modulation order of the neural network decision changes from 3 to 2, and the coding rate changes from 1 / 3 to 1 / 2. By reducing the modulation order, the system improves the overall transmission performance of the system by reducing the amount of information carried by each symbol.

[0179] Embodiment 2 of the present invention provides an intelligent decision-making system for cross-sea and air transport parameters based on reinforcement learning, comprising:

[0180] The environment module is used to construct a cross-sea air-channel model and to use the cross-sea air-channel model as a dynamic environment.

[0181] The state module is used to construct an intelligent agent. The intelligent agent obtains the current cross-sea-air environment and the transmission parameters for decision-making under the environment by observing the dynamic environment, and constructs the current cross-sea-air environment and the transmission parameters into a cross-sea-air channel state.

[0182] Action module, used by the agent to perform action mapping of transmission parameters according to the state of the cross-sea and air channel;

[0183] The experience module is used to calculate environmental rewards based on the results of action mapping, and integrate the environmental rewards, action mapping and cross-sea-air channel status into experience and store them in the experience playback pool.

[0184] The training module is used to train the agent using the experience in the experience replay pool, and outputs the optimal transmission parameter strategy across sea and air through the trained agent.

[0185] An electronic device provided in Embodiment 3 of the present invention, such as Figure 9 As shown, the electronic device includes a processor 21, a memory 22, an input device 23, and an output device 24; the number of processors 21 in the computer device can be one or more. Figure 9 Taking a processor 21 as an example; the processor 21, memory 22, input device 23, and output device 24 in an electronic device can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0186] The memory 22, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules. The processor 21 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 22, thereby realizing the reinforcement learning-based intelligent decision-making method for cross-sea and air transmission parameters in Embodiment 1.

[0187] The memory 22 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include memory remotely located relative to the processor 21, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0188] Input device 23 can be used to receive user input such as ID and password. Output device 24 is used to output the network configuration page.

[0189] Embodiment 4 of the present invention also provides a computer-readable storage medium, wherein the computer-executable instructions, when executed by a computer processor, are used to implement the intelligent decision-making method for cross-sea and air transmission parameters based on reinforcement learning as provided in Embodiment 1.

[0190] The storage medium containing computer-executable instructions provided in the embodiments of the present invention is not limited to the method operation provided in Embodiment 1, but can also execute related operations in the intelligent decision-making method for cross-sea and air transmission parameters based on reinforcement learning provided in any embodiment of the present invention.

[0191] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for intelligent decision-making of cross-sea and air transport parameters based on reinforcement learning, characterized in that, Includes the following steps: Constructing a cross-sea air-channel model, using the cross-sea air-channel model as a dynamic environment; wherein, constructing the cross-sea air-channel model includes: Calculate the empty segment loss and the sea segment loss based on Urick; the empty segment loss is calculated as follows: ； ； in, It is the loss of empty segments. It is the operating frequency. It is the sea surface reflectance, including the Fresnel reflectance and attenuation corrections for sea surface roughness. It's the wavelength. , It is the path difference between the direct wave and the mirror wave on the sea surface. This is the actual horizontal distance. , The height of the transmitting and receiving antennas relative to sea level; Calculate sea section loss: ； in, It is sea section loss, It is the expansion factor. It is the transmission distance, in units of , It is the absorption coefficient, with units of 1000 ppm. Frequency-dependent It is bottom scattering loss; The complex amplitude of the empty segment is calculated based on the empty segment loss, and the complex amplitude of the sea segment is calculated based on the sea segment loss; wherein, the complex amplitude of the empty segment is calculated based on the empty segment loss: ； in, It is the amplitude of the empty segment. It is a phase delay in the empty segment; Calculate the complex amplitude of the sea section based on the sea section loss: ； in, It is the complex amplitude of the sea section; The overall complex amplitude is calculated using the air segment complex amplitude and the sea segment complex amplitude, and the total path loss is determined based on the overall complex amplitude; wherein, the overall complex amplitude is obtained by multiplying the air segment complex amplitude and the sea segment complex amplitude. The total path loss is calculated based on the overall complex amplitude: ； in, This is the total path loss; The transmit power, transmit end gain, receive end gain, and noise power are obtained, and the transmit power, transmit end gain, receive end gain, and noise power are used as the transmission parameters of the cross-sea-air channel model; An intelligent agent is constructed, which acquires the current cross-sea-air environment and the transmission parameters for decision-making under this environment by observing the dynamic environment, and constructs the current cross-sea-air environment and the transmission parameters into a cross-sea-air channel state; wherein, constructing the current cross-sea-air environment and the transmission parameters into a cross-sea-air channel state includes: The transmitting sound source level is calculated using the sea segment model in the cross-sea air channel model and the received sound source level; wherein, the transmitting sound source level The calculation is as follows: , This represents the total power of the emitted sound source; The current cross-sea-air environment, together with the transmitted sound source level, modulation scheme, and coding rate, are used to construct the cross-sea-air channel state. ,in, It represents the current system environment, including channel quantization metrics such as noise intensity; Indicates the level of the emitted sound source; Indicates the modulation method. Indicates the encoding rate. It is the set of all states in a communication system; The agent performs action mapping of transmission parameters based on the cross-sea-air channel state; wherein, the agent performs action mapping of transmission parameters based on the cross-sea-air channel state, including: The selection of the emission source level, the modulation method, and the coding rate is used as the selection action; The action of selecting the emission source level, the action of selecting the modulation mode, and the action of selecting the coding rate are integrated into an action set; Action mapping for transmitting parameters based on actions in the action set; The environmental reward is calculated based on the action mapping result, and the environmental reward, action mapping, and cross-sea-air channel status are integrated into an experience and stored in the experience replay pool; wherein, the calculation of the environmental reward based on the action mapping result includes: The bit error rate is normalized to obtain the normalized bit error rate; where the bit error rate is the actual bit error rate measured by the communication device during data transmission; the formula for normalizing the bit error rate is as follows: ； in, It is the normalized bit error rate. and These are the minimum and maximum bit error rates, respectively. The modulation order and coding rate are determined through action mapping, and the successfully transmitted data is calculated using the modulation order and coding rate; wherein the calculation method for successfully transmitted data is as follows: ； in, This is the successful transmission rate. It is the bit error rate; An environmental reward is obtained by calculating the normalized bit error rate and the successfully transmitted data using a reward function; the reward function is as follows: ； in, Indicates the reward value. Indicates data transmission efficiency. , It is a positive constant; The agent is trained using the experience in the experience replay pool, and the trained agent outputs the optimal transmission parameter strategy across sea and air.

2. The intelligent decision-making method for cross-sea and air transport parameters based on reinforcement learning according to claim 1, characterized in that, The selection of the transmitted sound source level, the modulation scheme, and the coding rate is used as the selection action, including: The selection of the emission source level is defined as the selection action of the emission source level, and the selection action of the emission source level comprises a total of [number] actions. kind; Selecting the modulation order is used as the modulation mode selection action; the modulation mode selection action comprises a total of... kind; The selection of the coding rate is used as the coding rate selection action, and the coding rate selection action comprises a total of kind.

3. The intelligent decision-making method for cross-sea and air transport parameters based on reinforcement learning according to claim 1, characterized in that, The reward function includes: Determine data transmission efficiency by successfully transmitting data; Determine the bit error rate: When the bit error rate is greater than the bit error rate threshold, a negative reward function is constructed using the bit error rate; When the bit error rate is less than or equal to the bit error rate threshold, a positive reward function is constructed using the data transmission efficiency and the normalized bit error rate.

4. A reinforcement learning-based intelligent decision-making system for cross-sea and air transport parameters, used to implement the reinforcement learning-based intelligent decision-making method for cross-sea and air transport parameters as described in any one of claims 1 to 3, characterized in that, The intelligent decision-making system includes: The environment module is used to construct a cross-sea air-channel model and to use the cross-sea air-channel model as a dynamic environment. The state module is used to construct an intelligent agent. The intelligent agent obtains the current cross-sea-air environment and the transmission parameters for decision-making under the environment by observing the dynamic environment, and constructs the current cross-sea-air environment and the transmission parameters into a cross-sea-air channel state. Action module, used by the agent to perform action mapping of transmission parameters according to the state of the cross-sea and air channel; The experience module is used to calculate environmental rewards based on the results of action mapping, and integrate the environmental rewards, action mapping and cross-sea-air channel status into experience and store them in the experience playback pool. The training module is used to train the agent using the experience in the experience replay pool, and outputs the optimal transmission parameter strategy across sea and air through the trained agent.

5. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the intelligent decision-making method for cross-sea and air transport parameters based on reinforcement learning as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the intelligent decision-making method for cross-sea and air transport parameters based on reinforcement learning as described in any one of claims 1 to 3.

Images