Beam alignment method for water-to-air cross-medium laser communication based on deep reinforcement learning

Abstract

The present invention belongs to the technical field of optical wireless communication and deep reinforcement learning, and discloses a water-air cross-medium laser communication beam alignment method based on deep reinforcement learning, which includes the following steps: establishing a mathematical model of a water-air cross-medium laser channel by analyzing four parts, namely, emission, propagation loss, cross-interface and reception; establishing a deep reinforcement learning model based on the water-air cross-medium laser channel model of S1; training an intelligent agent based on the deep reinforcement learning model to achieve a beam alignment scheme with the strongest channel gain; and completing the test of the intelligent agent based on the beam alignment scheme with the strongest channel gain. The present invention adopts the above-mentioned water-air cross-medium laser communication beam alignment method based on deep reinforcement learning, predicts channel changes caused by factors such as waves through reinforcement learning, realizes beam alignment to maintain channel gain, and solves the stability problem of wireless optical channels in special environments of cross-medium communication.

Description

Technical Field

[0001] The present invention relates to the technical field of optical wireless communication and deep reinforcement learning, and in particular to a method for aligning light beams in water-to-air cross-medium laser communication based on deep reinforcement learning. Background Art

[0002] With the in-depth research on the new generation of 6G communication technology, in order to interconnect the communication networks in underwater and other environments, there is a need for cross-medium wireless communication between underwater and aerial devices. Traditional underwater acoustic communication has a low carrier frequency and small bandwidth, which makes it difficult to carry high communication rates. In addition, sound waves will be severely attenuated when crossing the interface between water and air, making it difficult to achieve cross-medium communication.

[0003] In a cross-domain communication environment, the direction of beam emission and reception is directly related to the strength of the received signal of optical wireless communication. One of the core requirements and difficulties of cross-medium optical communication is beam alignment. The receiving end of existing cross-medium laser communication solutions lacks effective measures to deal with the impact of waves. Some solutions do not have a mechanism to actively reduce the impact of waves. The link is only formed when the impact of waves is small, and the link is disconnected when the impact of waves is large, which has strong uncertainty. Some solutions use simple logic to control the direction of the light beam to cope with changes in waves, and the response range and effect are limited. There are also some methods that use active scanning to capture wave information and process it, but due to the limitations of sensor performance, it is often difficult to balance cost and accuracy.

[0004] To this end, the present invention proposes a water-to-air cross-medium laser communication beam alignment method based on deep reinforcement learning. Channel changes are predicted through reinforcement learning, and beam alignment is performed to maintain channel gain, thereby solving the stability problem of wireless optical channels in special environments of cross-medium communication. Summary of the invention

[0005] The purpose of the present invention is to provide a beam alignment method for water-to-air cross-medium laser communication based on deep reinforcement learning. By predicting channel changes caused by factors such as waves through reinforcement learning, beam alignment is achieved to maintain channel gain, thereby solving the problem of interference of water surface fluctuations on communication links in water-to-air cross-medium laser communication.

[0006] To achieve the above object, the present invention provides a water-air cross-medium laser communication beam alignment method based on deep reinforcement learning, comprising the following steps:

[0007] S1. According to the general structural characteristics of the communication system and the characteristics of the water-air cross-medium laser channel model, the mathematical model of the water-air cross-medium laser channel is established by analyzing the four parts of emission, propagation loss, cross-interface and reception;

[0008] S2. Based on the water-air cross-medium laser channel model in S1, a deep reinforcement learning model is established;

[0009] S3. Based on the deep reinforcement learning model, the intelligent agent is trained to achieve the beam alignment solution with the strongest channel gain;

[0010] S4. Complete the test of the intelligent agent based on the beam alignment scheme with the strongest channel gain.

[0011] Preferably, in S1, according to the requirement of beam energy concentration, a laser diode is used as the transmitter, and the angular spectrum of the emission gain G t (α t ), as follows:

[0012]

[0013] Among them, α t Represents the emission angle, that is, the angle between the emitted light and the main optical axis of the laser diode; ω t represents the maximum emission angle; λ represents the wavelength of the laser emitted by the laser diode.

[0014] Preferably, in S1, based on the absorption and scattering theory of light propagation underwater, the free space propagation theory and the distance attenuation theory, a propagation attenuation model of the optical signal in the underwater-air is established, and the attenuation equation of the light in the propagation process is as follows:

[0015]

[0016] Among them, G p (d water ,d air ) is the transmission path gain; d water is the underwater transmission path; d air is the transmission path in the air; α(λ) is the underwater attenuation coefficient, which is related to the light wavelength λ.

[0017] Preferably, in S1, the influence of sea waves on the water-air cross-medium interface in the actual environment is considered, and a water-air cross-medium propagation attenuation model is established according to the sea wave model. By simulating the dynamic change of sea waves over time, the change of the refraction angle and energy of the light beam is calculated in real time to obtain the cross-medium energy attenuation coefficient. The specific process is as follows:

[0018] S131. Based on the wave spectrum theory, the power distribution model of the wave in frequency is determined as follows:

[0019]

[0020] Among them, S(ω) is the power spectrum; G(ω,θ) is the directional spectrum, which describes the change of power with direction; Among them U 10 is the average wind speed at 10m above the sea surface, xf is the wind distance; g is the acceleration due to gravity; ω is the frequency; is the peak frequency; γ is the peak rise coefficient, which is 3.3; σ is the shape parameter, when ω≤ω p The value is 0.07, when ω>ω p The value is 0.09; is the harmonic component parameter; θ is the directional angle; the product of the power spectrum and the directional spectrum, the wave spectrum S(ω,θ)=S(ω)G(ω,θ), fully describes the power distribution of the waves in frequency;

[0021] S132. Based on the power distribution of the wave spectrum, the harmonic method is used to restore the wave morphology function T(x, y, t) at the water-air interface, as shown below:

[0022]

[0023] Where S(ω,θ) is the wave spectrum; i,j = 1, 2, ...; x, y are the horizontal and vertical coordinates respectively; ∈ is the random phase shift;

[0024] S133. Partially differentiate the wave morphology function to determine its normal vector, and solve the relationship between the incident angle and the refraction angle; based on the Fresnel equation, combined with the ratio of the total energy of the refracted beam to the total energy of the incident beam, the energy attenuation coefficient G across the medium is obtained. ref , as shown below:

[0025]

[0026] Wherein, n1 and n2 are the refractive indices of the incident water medium and the outgoing air medium, respectively, and θ1 and θ2 are the incident angle and the outgoing angle, respectively.

[0027] Preferably, in S1, a photodiode is used as a receiver, and its receiving angle spectrum is as follows:

[0028]

[0029] Where n2 represents the refractive index of the air medium at the receiving end; α r With ω r They are respectively the angle of light incident on the receiving end and the maximum viewing angle of the receiving end.

[0030] Preferably, the water-air cross-medium channel model is as follows:

[0031] G=G t (α t )·G p (d water ,d air )·G ref ·G r (αr );

[0032] Where G is the channel gain;

[0033] When the transmitting end power is I0, the receiving end power is I1, as shown below:

[0034] I1=G·I0.

[0035] Preferably, in S2, a deep reinforcement learning model is established, and the specific process is as follows:

[0036] S21. Calculate the received light intensity in the receiving channel model using a ray tracing method. The specific process is as follows:

[0037] First, perform interval sampling within the field of view of the receiving end; then, track the light intensity information of each light path in reverse; finally, sum the light intensities of all light paths to obtain the received light intensity;

[0038] S22. Set the channel as the environment of the deep reinforcement learning model. The specific settings are as follows:

[0039] (1) Initialize the environment of the deep reinforcement learning model: the position, orientation, water-air interface and other information of the transmitter and receiver are used as parameters of the environment, and the environmental parameters change over time;

[0040] (2) Initialize the agent: The transmitter / receiver is regarded as an agent, and the parameter affecting the agent is the orientation of the transmitter;

[0041] (3) Taking environmental parameters as observations and outputting orientation changes as responses;

[0042] (4) Using deep decision gradient as the decision rule of the intelligent agent;

[0043] S23, the deep decision gradient algorithm adopts the responder-evaluator mechanism. The responder evaluates the environment S according to the current state. t Respond as follows:

[0044] a t =π θ (S t )+χ;

[0045] Among them, a t is the response vector, including all response parameters; π θ is the prediction function, which predicts the response according to the state; χ is the exploration noise, which is a random quantity;

[0046] The evaluator evaluates the above response as follows:

[0047] Q ω (S t,a t );

[0048] Among them, Q ω It is the evaluation value, generally referred to as Q value;

[0049] S24, predict the next state of the environment and update the network, and the current network outputs the corresponding target value, as shown below:

[0050] y i =r i +γ'Q ω' (S i+1 ,π θ (S i+1 ));

[0051] Among them, y i is the target value; r i is the reward; γ' is the decision-making proportional coefficient; Q ω' is the Q value of the predicted state; S i+1 is the predicted next state;

[0052] The evaluator outputs the Q value through the state, evaluates the quality of the selected response under different environmental parameters, and updates the network by minimizing the loss function, as shown below:

[0053]

[0054] Among them, Loss is the loss function; N is the total number of evaluations; S i is a state parameter, affected by environmental factors; a i is the parameter for each evaluation;

[0055] The responder is used to provide the strategy training process for the next state, and the strategy gradient is obtained by combining the Q value given by the evaluator, as shown below:

[0056]

[0057] in, is the policy gradient; is the gradient of the evaluation vector; S i is the state parameter; is the network parameter gradient;

[0058] S25. Complete the update of network parameters w and θ, and adopt a partial update strategy to ensure the stability of learning.

[0059] Preferably, in S3, the agent is trained based on a deep reinforcement learning model, and the specific process is as follows:

[0060] S31. Establish a reset function Reset Function as shown below:

[0061] [NextState,Observation]=R(InitParams);

[0062] Among them, InitParams is the input initialization parameter; NextState is the next environment state; Observation is the current observation; the reset function is activated at the beginning of each training cycle and resets the environment state. The randomness of the environment is introduced during the reset process to ensure the portability of the training results;

[0063] S32. Create a step function as shown below:

[0064] [NextState,Observation,Isdone,Reward]=StepFcn(State,Action);

[0065] Among them, State is the current state; Action is the agent response; NextState is the next environment state; Observation is the current observation; Isdone is the termination training prompt, which is used to determine whether the training of this round has reached the termination standard and end this round of training in advance; Reward is the reward function value;

[0066] The reward value obtained by the agent's response is as follows:

[0067] Reward(I)=G(I)·(ln(aI+1)+lg(bI)+exp(cI))+B;

[0068] Where I is the received light intensity; G(I) is the channel gain; a, b, and c are adjustable parameters that determine the mapping form of the reward function; and B is the offset.

[0069] Therefore, the present invention adopts the above-mentioned water-to-air cross-medium laser communication beam alignment method based on deep reinforcement learning. It predicts the channel changes caused by factors such as waves through reinforcement learning, has superior ability in resisting horizontal deviation, accurately extracts the dynamic change information of waves, and realizes beam alignment to maintain channel gain, thus solving the stability problem of wireless optical channels in special environments of cross-medium communication.

[0070] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is the JONSWAP wave spectrum function of the present invention; wherein, (a) is the JONSWA P energy spectrum curve; (b) is the three-dimensional energy-direction spectrum simulation diagram;

[0072] Figure 2 It is a structural block diagram of the depth decision gradient system of the present invention;

[0073] Figure 3 is a schematic diagram of a water-air cross-medium laser communication system according to an embodiment of the present invention;

[0074] Figure 4 is a flow chart of training with an agent in an embodiment of the present invention;

[0075] Figure 5 is a channel gain-time curve in an embodiment of the present invention;

[0076] Figure 6 is a curve showing a change in the average channel gain versus the horizontal offset in an embodiment of the present invention;

[0077] Figure 7 is a curve showing a change in mean square error with horizontal offset in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0079] Considering that the optical path is affected by medium absorption attenuation and interface refraction in the water-air cross-domain laser communication environment, the present invention proposes a water-air cross-medium laser communication beam alignment method based on deep reinforcement learning to obtain the maximum received signal strength. The specific process is as follows:

[0080] S1. Establish a mathematical model for water-to-air cross-medium laser communication.

[0081] According to the general structural characteristics of the communication system and the characteristics of the water-air cross-medium laser channel model, the mathematical model of the water-air cross-medium laser channel is established by analyzing the four parts of emission, propagation loss, cross-interface and reception, as shown below:

[0082] S11, emission part: according to the need of beam energy concentration, a laser diode (LD) is used as the emitter, and the angular spectrum of the emission gain G t (α t ), as follows:

[0083]

[0084] Among them, α t Represents the emission angle, that is, the angle between the emitted light and the main optical axis of the LD; ω t represents the maximum emission angle; λ represents the wavelength of the laser emitted by LD.

[0085] S12, propagation path loss part: Based on the absorption and scattering theory of light propagation underwater, free space propagation theory and distance attenuation theory, a propagation attenuation model of optical signals in underwater-air is established. The attenuation equation of light in the propagation process is as follows:

[0086]

[0087] Among them, G p (d water ,d air ) is the transmission path gain; d water is the underwater transmission path; d air is the transmission path in the air; α(λ) is the underwater attenuation coefficient, which is related to the light wavelength λ.

[0088] S13, water-air interface cross-medium part: Considering the influence of waves on the water-air cross-medium interface in the actual environment, a water-air cross-medium propagation attenuation model is established based on the wave model. By simulating the dynamic changes of waves over time, the refraction angle and energy changes of the light beam are calculated in real time to obtain the cross-medium energy attenuation coefficient. The specific process is as follows:

[0089] S131. Based on the wave spectrum theory, the power distribution model of the wave in frequency is determined as follows:

[0090]

[0091] Among them, S(ω) is the power spectrum; G(ω,θ) is the directional spectrum, which describes the change of power with direction; Among them U 10 is the average wind speed at 10m above the sea surface, x f is the wind fetch; g is the acceleration due to gravity; ω is the frequency; is the peak frequency; γ is the peak rise coefficient, which is 3.3; σ is the shape parameter, when ω≤ω p The value is 0.07, when ω>ω p The value is 0.09;

[0092] is the harmonic component parameter; θ is the direction angle; the product of the power spectrum and the directional spectrum, the wave spectrum S(ω,θ)=S(ω)G(ω,θ), fully describes the power distribution of the waves in frequency.

[0093] S132, such as Figure 1 As shown, based on the power distribution of the wave spectrum, the harmonic method is used to restore the wave morphology function T(x,y,t) at the water-air interface, as shown below:

[0094]

[0095] Among them, S(ω,θ) is the wave spectrum; i,j=1,2,…; x, y are the horizontal and vertical coordinates respectively; ∈ is the random phase shift.

[0096] S133. Partially differentiate the wave morphology function to determine its normal vector, and solve the relationship between the incident angle and the refraction angle; based on the Fresnel equation, combined with the ratio of the total energy of the refracted beam to the total energy of the incident beam, the energy attenuation coefficient G across the medium is obtained. ref , as shown below:

[0097]

[0098] Where n1 and n2 are the refractive indices of the incident medium (water) and the exit medium (air), respectively, and θ1 and θ2 are the incident angle and the exit angle, respectively.

[0099] S14, receiving end: using photodiode (APD) as the receiver, its receiving angle spectrum is as follows:

[0100]

[0101] Where n2 represents the refractive index of the medium (air) where the receiving end is located; α r With ω r They are respectively the angle of light incident on the receiving end and the maximum viewing angle of the receiving end.

[0102] S15. Multiply the above four gains to obtain a water-air cross-medium channel model, as shown below:

[0103] G=G t (α t )·G p (d water ,d air )·G ref ·G r (α r );

[0104] Where G is the channel gain;

[0105] When the transmitting end power is I0, the receiving end power is I1, as shown below:

[0106] I1=G·I0.

[0107] S2. Based on the water-air cross-medium laser channel model of S1, a deep reinforcement learning model is established.

[0108] S21. Using the ray tracing method, the received light intensity in the receiving channel model is calculated. The specific process is as follows:

[0109] First, interval sampling is performed within the field of view (FOV) of the receiving end; then, the light intensity information of each light path is traced in reverse; finally, the light intensities of all light paths are summed to obtain the received light intensity.

[0110] S22. Set the channel as the environment of the deep reinforcement learning model. The specific settings are as follows:

[0111] (1) The position, orientation, water-air interface and other information of the transmitter and receiver are used as environmental parameters, and the environmental parameters change over time;

[0112] (2) The transmitter / receiver is regarded as an agent, and the parameter that affects the agent is the orientation of the transmitter;

[0113] (3) Taking environmental parameters as observation values ​​and outputting orientation changes as responses (Action);

[0114] (4) Figure 3 As shown in the figure, Deep Deterministic Policy Gradient (DDPG) is used as the decision rule of the agent.

[0115] S23, DDPG algorithm adopts the responder (Actor)-evaluator (Critic) mechanism. The responder evaluates the environment S according to the current state. t Respond as follows:

[0116] a t =π θ (S t )+χ;

[0117] Among them, a t is the response vector, including all response parameters; π θ is the prediction function, which predicts the response according to the state; χ is the exploration noise, which is a random quantity.

[0118] The evaluator evaluates the above response as follows:

[0119] Q ω (S t ,a t );

[0120] Among them, Q ω It is an evaluation value, generally referred to as Q value.

[0121] S24, predict the next state of the environment and update the network, and the current network outputs the corresponding target value, as shown below:

[0122] yi =r i +γ'Q ω' (S i+1 ,π θ (S i+1 ));

[0123] Among them, y i is the target value; r i is the reward; γ' is the decision-making proportional coefficient; Q ω' is the Q value of the predicted state; S i+1 is the predicted next state.

[0124] The evaluator outputs the Q value through the state, evaluates the quality of the selected response under different environmental parameters, and updates the network by minimizing the loss function, as shown below:

[0125]

[0126] Among them, Loss is the loss function; N is the total number of evaluations; S i is a state parameter, affected by environmental factors; a i is the parameter for each evaluation.

[0127] The responder is used to provide the strategy training process for the next state, and the strategy gradient is obtained by combining the Q value given by the evaluator, as shown below:

[0128]

[0129] in, is the policy gradient; is the gradient of the evaluation vector; S i is the state parameter; is the network parameter ladder.

[0130] S25. Complete the update of network parameters w and θ, and adopt a partial update strategy to ensure the stability of learning.

[0131] S3. Based on the deep reinforcement learning model, the intelligent agent is trained to achieve the beam alignment solution with the strongest channel gain.

[0132] S31. Create a reset function (Reset Function), as shown below:

[0133] [NextState,Observation]=R(InitParams);

[0134] Among them, the input and output parameters are shown in Table 1:

[0135] Table 1 Input and output parameters of the reset function

[0136] Input Parameters describe Output Parameters describe InitParams Initialization parameters NextState Next environment state Observation Current Observation

[0137] The reset function is activated at the beginning of each training cycle and resets the environment state. The randomness of the environment is introduced during the reset process to ensure the portability of the training results.

[0138] S32. Create a step function as shown below:

[0139] [NextState,Observation,Isdone,Reward]=

[0140] StepFcn(State,Action);

[0141] Among them, the input and output parameters are shown in Table 2:

[0142] Table 2 Input and output parameters of step functions

[0143] Input Parameters describe Output Parameters describe State Current Status NextState Next environment state Action Agent Response Observation Current Observation Isdone Termination of training prompt Reward Reward function value

[0144] This function is activated at each step of the training. After running, it will automatically iterate to the next step until a training round is completed. The termination signal (Isdone) is used to determine whether the training of this round has reached the termination standard and end this round of training in advance. In addition, the reward value (Reward) obtained by the agent's response is as follows:

[0145] Reward(I)=G(I)·(ln(aI+1)+lg(bI)+exp(cI))+B;

[0146] Where I is the received light intensity; G(I) is the channel gain; a, b, and c are adjustable parameters that determine the mapping form of the reward function; B is the offset; exponential and logarithmic terms are used to increase the discrimination of the reward function values ​​corresponding to different received light intensities, and to ensure that the light intensities of extremely large or small orders of magnitude will not overflow or disappear, so that the model can converge to the optimal situation with a greater probability during training.

[0147] S33. Establish a deep reinforcement learning environment and set the initialization hyperparameters as shown in Table 3:

[0148] Table 3 Initialization hyperparameters of deep reinforcement learning environment

[0149] Hyperparameters value noise Gaussian dynamic noise Attenuation coefficient 0.2 Sampling time 0.05s Buffer length 1e6 BatchSize 64 Reactor learning rate 1e-3 Evaluator learning rate 1e-4 Maximum number of training rounds 500 Maximum number of steps per training round 500

[0150] After completing the above settings, you can perform deep reinforcement learning training to achieve beam alignment with the strongest channel gain.

[0151] S4. Complete the test of the intelligent agent based on the beam alignment scheme with the strongest channel gain.

[0152] Example

[0153] like Figure 3 As shown in the figure, the water-air cross-domain laser wireless communication system of this embodiment uses the LD of the unmanned underwater vehicle (UUV) as the transmitter to transmit the optical signal across the water-air interface to the APD receiver on the unmanned vehicle in the air. In order to maintain the high gain of the channel, a beam alignment method based on deep reinforcement learning is adopted to predict the beam direction according to the feedback of the received signal strength. A mathematical model is established to make the transmitter direction vector e t =(x t ,y t ,z t ), the receiving end direction vector is e r =(x r ,y r ,z r ), adjust the direction of the devices at both ends by adjusting the values ​​of the transmit and receive vectors. The transmit and receive directions are controlled by a deep learning-based beam steering algorithm and change with the channel state to obtain the maximum channel gain. Among them I r ,I t is the intensity of light received and emitted. The specific process is as follows:

[0154] Step 1: Use an iterative ray tracing algorithm to solve the optical path of the light beam from the transmitter to the receiver. Divide the receiver into m×m vector groups (x i ,y j ,z c ), where the angular spacing between adjacent vectors is The ray tracing algorithm is used to trace back along the direction indicated by each vector, and detect whether the traced optical path intersects with the water surface and the light source; if it intersects, the cross-medium refraction information and the light source radiation energy information are recorded, and finally the sum is processed. Among them, the refraction point of the optical path in each direction is calculated based on the simulated reconstructed water-air cross-medium interface function T(x,y,t), and the receiver position point and the vector (x i ,y j ,z c The intersection of the straight line and the T(x,y,t) function is the refraction point; the refraction angle is obtained based on the gradient of the incident direction vector and the interface function, and then the refraction direction is obtained; finally, if the distance between the straight line represented by the refraction point and the direction vector and the center of the light source is less than a certain value ∈, it is determined that it intersects with the light source, and then the angle gain is calculated based on the angle between the light and the light source direction.

[0155] Step 2: Establish a deep reinforcement learning system. It is divided into environment and agent. First, initialize the environment and establish the environment coordinate system. Set the sea level as the zero value of the z-axis. Set the position of the transmitter and receiver p t (x0,y0,z0),p r (x1, y1, z1 and direction vector e t =(x t ,y t ,z t ), e r =(x r ,y r ,z r ) is set as the environmental parameter; at the same time, the wave surface function T(x, y, t) is embedded in the environment, and the parameters affecting the waves are set as environmental parameters at the same time. The received light intensity I can be calculated from the environmental parameters r And the channel gain G.

[0156] Secondly, initialize the agent. The agent is a DDPG network with an Actor-Cr itic structure composed of a deep neural network. It is included in the entire environment, observes the transmitter and receiver position and direction parameters and the received light intensity parameters in the environment, and gives feedback to adjust the joint feedback of the transmitter and receiver. t and p r With the direction e t and e r And the receiving end light intensity I r to obtain environmental information as input. Then, the agent can adjust the transmitter direction vector e t and the receiver direction vector e r , in order to simplify the structure of the neural network and reduce the computational overhead. Considering the physical limitation of the rotation angle of the receiving end, the agent only adjusts e t and e r The two dimensions in x t ,y t ,x r ,y r variables, thereby controlling the response to the environment.

[0157] Step 3: According to the environment established in the previous step, train the agent and establish a reset function (ResetFunction), as shown below:

[0158] [NextState,Observation]=R(InitParams);

[0159] This function needs to input initialization information, and its output is the next environment state (NextState) and the observation value (Observation) of the current environment state. The reset function is activated at the beginning of each training cycle and resets the environment state. The randomness of the environment is introduced during the reset process to ensure the portability of the training results. In addition, according to the structure of the environment and the agent, a step function is established as follows:

[0160] [NextState,Observation,Isdone,Reward]=StepFcn(State,Action)

[0161] This function takes the current environment state (State) and agent response (Action) as input, and updates the new state (NextState) and observation (Observation). In the step function, a termination signal (Isdone) is set to determine whether the training of this round has reached the termination standard and end this round of training in advance. In addition, the reward (Reward) value obtained by the agent's response is also calculated in this function, as shown below:

[0162] Reward(I)=G·(ln(aI+1)+lg(bI)+exp(cI))+B;

[0163] Exponential and logarithmic terms are used to increase the discrimination of the reward function values ​​corresponding to different received light intensities, and to ensure that the light intensities of extremely large or extremely small magnitudes will not overflow or disappear, so that the model can converge to the optimal situation with a greater probability during training. After determining the two equations, Figure 4 The training process is performed according to the process shown in the figure. During the training process, the hyperparameter settings are shown in Table 4:

[0164] Table 4 Initialization hyperparameters of the training process

[0165] Hyperparameters value noise Gaussian dynamic noise Attenuation coefficient 0.2 Sampling time 0.05s Buffer length 1e6 BatchSize 64 Reactor learning rate 1e-3 Evaluator learning rate 1e-4 Maximum number of training rounds 500 Maximum number of steps per training round 500

[0166] Step 4: Test the agent after completing the training in the above steps. Run the agent in a randomly generated wave environment for testing and record the received light intensity data. Figure 5 As shown in FIG. 1 , the channel gain-time curve shows that the method proposed in the present invention is far superior to the method without alignment in terms of channel gain performance, and is closer to the theoretical upper limit than other alignment methods. Figure 6 As shown in Figure 2, the change of the average channel gain with the horizontal offset shows that the method proposed in the present invention has superior ability in resisting horizontal offset. In addition, the stability of the alignment algorithm is also a consideration. For different algorithms, a mean square error of gain and theoretical upper limit is proposed. The stability factor is calculated, and the lower the value, the higher the stability. Figure 7 As shown, the curve of the parameter versus horizontal offset shows that the stability of the method proposed in the present invention is higher than that of other methods, indicating the accuracy of the method in extracting the dynamic change information of ocean waves.

[0167] Therefore, the present invention adopts the above-mentioned water-to-air cross-medium laser communication beam alignment method based on deep reinforcement learning. It predicts the channel changes caused by factors such as waves through reinforcement learning, has superior ability in resisting horizontal deviation, accurately extracts the dynamic change information of waves, and realizes beam alignment to maintain channel gain, thus solving the stability problem of wireless optical channels in special environments of cross-medium communication.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A beam alignment method for water-to-air cross-medium laser communication based on deep reinforcement learning, characterized in that: The following steps are involved: S1. According to the general structural characteristics of the communication system and the characteristics of the water-air cross-medium laser channel model, the mathematical model of the water-air cross-medium laser channel is established by analyzing the four parts of emission, propagation loss, cross-interface and reception; According to the requirement of beam energy concentration, a laser diode is used as the transmitter, and the angular spectrum of the emission gain G t (α t ), as follows: Among them, α t Represents the emission angle, that is, the angle between the emitted light and the main optical axis of the laser diode; ω t represents the maximum emission angle; λ represents the wavelength of the laser emitted by the laser diode; Based on the absorption and scattering theory of light propagation underwater, the free space propagation theory and the distance attenuation theory, a propagation attenuation model of optical signals in underwater-air is established. The attenuation equation of light in the propagation process is as follows: Among them, G p (d water ,d air ) is the transmission path gain; d water is the underwater transmission path; d air is the transmission path in the air; α(λ) is the underwater attenuation coefficient, which is related to the light wavelength λ; Considering the influence of waves on the water-air cross-medium interface in actual environment, a water-air cross-medium propagation attenuation model is established based on the wave model. By simulating the dynamic changes of waves over time, the refraction angle and energy changes of the light beam are calculated in real time to obtain the cross-medium energy attenuation coefficient. The specific process is as follows: S131. Based on the wave spectrum theory, the power distribution model of the wave in frequency is determined as follows: Among them, S(ω) is the power spectrum; G(ω,θ) is the directional spectrum, which describes the change of power with direction; Among them U 10 is the average wind speed at 10m above the sea surface, x f is the wind distance; g is the acceleration due to gravity; ω is the frequency; is the peak frequency; γ is the peak rise coefficient, which is 3.3; σ is the shape parameter, when ω≤ω p The value is 0.07, when ω>ω p The value is 0.09; is the harmonic component parameter; θ is the directional angle; the product of the power spectrum and the directional spectrum, the wave spectrum S(ω,θ)=S(ω)G(ω,θ), fully describes the power distribution of the waves in frequency; S132. Based on the power distribution of the wave spectrum, the harmonic method is used to restore the wave morphology function T(x, y, t) at the water-air interface, as shown below: Among them, S(ω,θ0 is the wave spectrum; i,j=1,2,…; x, y are the horizontal and vertical coordinates respectively; ∈ is the random phase shift; S133. Partially differentiate the wave morphology function to determine its normal vector, and solve the relationship between the incident angle and the refraction angle; based on the Fresnel equation, combined with the ratio of the total energy of the refracted beam to the total energy of the incident beam, the energy attenuation coefficient G across the medium is obtained. ref , as shown below: Among them, n1 and n2 are the refractive indices of the incident water medium and the outgoing air medium, respectively, and θ1 and θ2 are the incident angle and the outgoing angle, respectively; Using a photodiode as a receiver, its receiving angle spectrum is as follows: Where n2 represents the refractive index of the air medium at the receiving end; α r With ω r are the angle of light incident on the receiving end and the maximum viewing angle of the receiving end respectively; The water-air cross-medium channel model is as follows: G=G t (a t )·G p (d water ,d air )·G ref ·G r (a r ); Where G is the channel gain; When the transmitting end power is I0, the receiving end power is I1, as shown below: I1=G·I0; S2. Based on the water-air cross-medium laser channel model in S1, a deep reinforcement learning model is established. The specific process is as follows: S21. Calculate the received light intensity in the receiving channel model using a ray tracing method. The specific process is as follows: First, perform interval sampling within the field of view of the receiving end; then, track the light intensity information of each light path in reverse; finally, sum the light intensities of all light paths to obtain the received light intensity; S22. Set the channel as the environment of the deep reinforcement learning model. The specific settings are as follows: (1) Initialize the environment of the deep reinforcement learning model: the position, orientation, water-air interface and other information of the transmitter and receiver are used as parameters of the environment, and the environmental parameters change over time; (2) Initialize the agent: The transmitter / receiver is regarded as an agent, and the parameter affecting the agent is the orientation of the transmitter; (3) Taking environmental parameters as observations and outputting orientation changes as responses; (4) Using deep decision gradient as the decision rule of the intelligent agent; S23, the deep decision gradient algorithm adopts the responder-evaluator mechanism. The responder evaluates the environment S according to the current state. t Respond as follows: a t =π θ (S t )+x; Among them, a t is the response vector, including all response parameters; π θ is the prediction function, which predicts the response according to the state; χ is the exploration noise, which is a random quantity; The evaluator evaluates the above response as follows: Q ω (S t ,a t ); Among them, Q ω It is the evaluation value, generally referred to as Q value; S24, predict the next state of the environment and update the network, and the current network outputs the corresponding target value, as shown below: y i =r i +γ'Q ω' (S i+1 ,p θ (S i+1 )); Among them, y i is the target value; r i is the reward; γ' is the decision-making proportional coefficient; Q ω' is the Q value of the predicted state; S i+1 is the predicted next state; The evaluator outputs the Q value through the state, evaluates the quality of the selected response under different environmental parameters, and updates the network by minimizing the loss function, as shown below: Among them, Loss is the loss function; N is the total number of evaluations; S i is a state parameter, affected by environmental factors; a i is the parameter for each evaluation; The responder is used to provide the strategy training process for the next state, and the strategy gradient is obtained by combining the Q value given by the evaluator, as shown below: in, is the policy gradient; is the gradient of the evaluation vector; S i is the state parameter; is the network parameter gradient; S25, complete the update of network parameters w and θ, and adopt the partial update strategy to ensure the stability of learning; S3. Based on the deep reinforcement learning model, the intelligent agent is trained to achieve the beam alignment solution with the strongest channel gain; S4. Complete the test of the intelligent agent based on the beam alignment scheme with the strongest channel gain.

2. According to the method for water-air cross-medium laser communication beam alignment based on deep reinforcement learning in claim 1, it is characterized in that: In S3, the agent is trained based on the deep reinforcement learning model. The specific process is as follows: S31. Establish a reset function Reset Function as shown below: [NextState,Observation]=R(InitParams); Among them, InitParams is the input initialization parameter; NextState is the next environment state; Observation is the current observation; the reset function is activated at the beginning of each training cycle and resets the environment state. The randomness of the environment is introduced during the reset process to ensure the portability of the training results; S32. Create a step function as shown below: [NextState,Observation,Isdone,Reward]= StepFcn(State,Action); Among them, State is the current state; Action is the agent response; NextState is the next environment state; Observation is the current observation; Isdone is the termination training prompt, which is used to determine whether the training of this round has reached the termination standard and end this round of training in advance; Reward is the reward function value; The reward value obtained by the agent's response is as follows: Reward(I)=G(I)·(ln(aI+1)+lg(bI)+exp(cI))+B; Where I is the received light intensity; G(I) is the channel gain; a, b, and c are adjustable parameters that determine the mapping form of the reward function; and B is the offset.

Images