OBA optimization method and system based on reinforcement learning and storage medium

Abstract

The invention discloses an OBA optimization method and system based on reinforcement learning and a storage medium, and belongs to the technical field of communication. The method comprises the following steps: an OBA table construction stage; a reinforcement learning reward mechanism setting stage: setting a reinforcement learning reward mechanism based on the monotonicity and directivity of the OBA table; in the initialization stage, the network weight, the experience playback buffer area and the initial state of the reinforcement learning algorithm are initialized; in the reinforcement learning network parameter updating stage, a reinforcement learning algorithm is operated according to the OBA table, batch data are randomly taken from the experience playback buffer area, the parameters of the reinforcement learning network are updated by using a gradient descent method, and the monotonicity and directivity of the OBA table are improved; and an intelligent agent and beam forming weight output stage: when the monotonicity and directivity of the OBA table reach preset index requirements, the intelligent agent and beam forming weight is output. Through an actual external field test, the quantitative proportion of the OBA table meeting the direction finding requirement can be improved by more than 15%, and the direction finding precision of certain communication equipment can be greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to an OBA optimization method, system and storage medium based on reinforcement learning. Background Art

[0002] In a certain communication process, a single pulse direction finding method is used to accurately find the direction of the incoming wave of the communication signal. In the single pulse direction finding method, the OBA table lookup method is used to obtain the measurement angle, which requires the OBA to have good monotonicity and low gain loss. In the beamforming process, the phase shift code is calculated according to the incoming wave direction, frequency, and array element spacing, and Taylor weighting is used to suppress the side lobes. However, after the communication antenna is installed, the antenna radiation and the received signal are affected by the reflection and scattering of the fuselage. As a result, after the existing method performs beamforming, different equipment platforms will produce abnormal directional pattern shapes in specific frequency bands and specific angles, resulting in abnormal OBA shapes and reduced direction finding accuracy. There are some technical methods for OBA correction in the design, but the actual application effect is not ideal.

[0003] In a certain communication process, in order to improve the direction finding accuracy, the OBA table needs to have good monotonicity and directivity. The existing methods include methods such as monotonicity and directivity correction of the generated OBA table. The main disadvantages are: it can reduce the confusion caused by the non-monotonicity of the OBA table, but it cannot fundamentally reduce the direction finding error of the OBA, or the OBA optimization effect is not obvious, and the improvement of the direction finding accuracy in actual use is very small. The single pulse direction finding method is used to find the direction of the incoming wave of the communication signal. The single pulse direction finding method uses the OBA table lookup method to obtain the measurement angle, but the abnormal shape of the OBA due to the reflection and scattering of the fuselage will reduce the direction finding accuracy. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and provide an OBA optimization method, system and storage medium based on reinforcement learning.

[0005] The objective of the present invention is achieved through the following technical solutions: The first aspect of the present invention provides: an OBA optimization method based on reinforcement learning, comprising the following steps: In the OBA table construction phase, the OBA table is constructed by performing synthetic calculations on the multi-element directional patterns; In the reinforcement learning reward mechanism setting stage, the reinforcement learning reward mechanism is set based on the monotonicity and directionality of the OBA table; Initialization phase: Initialize the network weights, experience replay buffer, and initial state of the reinforcement learning algorithm; In the reinforcement learning network parameter update phase, the reinforcement learning algorithm is run according to the OBA table, and the single training records of reinforcement learning are accumulated in the experience replay buffer. Then batches of data are randomly taken from the experience replay buffer, and the parameters of the reinforcement learning network are updated using the gradient descent method to improve the monotonicity and directionality of the OBA table. In the stage of outputting the agent and beamforming weights, when the monotonicity and directivity of the OBA table meet the preset index requirements, the agent and beamforming weights are output.

[0006] Preferably, the reinforcement learning algorithm includes multiple neural networks and is provided with an experience replay mechanism.

[0007] Preferably, the synthetic calculation of the multi-element directional patterns to construct the OBA table includes the following steps: assigning amplitude and phase compensation values ​​to each element so that the phases of the signals received by each element are consistent, and then synthesizing the sum beam and the difference beam, calculating the difference between the sum beam and the difference beam, and calculating the optimal amplitude and optimal phase compensation difference through a reinforcement learning algorithm, so that the OBA curve reaches a preset shape, thereby optimizing the direction finding capability of the OBA table.

[0008] Preferably, the monotonicity and directionality-based reinforcement learning reward mechanism of the OBA table includes the following steps: calculating the slope of the OBA, the expected value of the slope is positive on the left and negative on the right, and the middle is the OBA maximum value; calculating the monotonicity reward value according to the slope and expected value of each point; calculating the directivity reward value by the difference between the angle at which the OBA maximum value is located and the wave position pointing angle.

[0009] The second aspect of the present invention provides: an OBA optimization system based on reinforcement learning, used to implement any of the above-mentioned OBA optimization methods based on reinforcement learning, comprising: An OBA table construction module is used to perform synthetic calculations on multi-element directional patterns to construct an OBA table; Reinforcement learning reward mechanism setting module, used to set the reinforcement learning reward mechanism based on the monotonicity and directionality of the OBA table; Initialization module, used to initialize the network weights, experience replay buffer and initial state of the reinforcement learning algorithm; The reinforcement learning network parameter update module is used to run the reinforcement learning algorithm according to the OBA table, accumulate the single training records of reinforcement learning into the experience replay buffer, and randomly take batch data from the experience replay buffer, and use the gradient descent method to update the parameters of the reinforcement learning network to improve the monotonicity and directionality of the OBA table; The intelligent agent and beamforming weight output module is used to output the intelligent agent and beamforming weights when the monotonicity and directivity of the OBA table meet the preset index requirements.

[0010] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned OBA optimization methods based on reinforcement learning is implemented.

[0011] The beneficial effects of the present invention are: 1) Through actual field tests, the present invention can increase the number of OBA tables that meet the direction finding requirements by more than 15%. Therefore, the OBA optimization method has a high use value and can greatly improve the direction finding accuracy of a certain communication equipment, thereby improving the actual use efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Flowchart of the OBA optimization method based on reinforcement learning. DETAILED DESCRIPTION

[0013] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0014] See also Figure 1 The first aspect of the present invention provides: an OBA optimization method based on reinforcement learning, comprising the following steps: In the OBA table construction phase, the OBA table is constructed by performing synthetic calculations on the multi-element directional patterns; In the reinforcement learning reward mechanism setting stage, the reinforcement learning reward mechanism is set based on the monotonicity and directionality of the OBA table; Initialization phase: Initialize the network weights, experience replay buffer, and initial state of the reinforcement learning algorithm; In the reinforcement learning network parameter update phase, the reinforcement learning algorithm is run according to the OBA table, and the single training records of reinforcement learning are accumulated in the experience replay buffer. Then batches of data are randomly taken from the experience replay buffer, and the parameters of the reinforcement learning network are updated using the gradient descent method to improve the monotonicity and directionality of the OBA table. In the stage of outputting the agent and beamforming weights, when the monotonicity and directivity of the OBA table meet the preset index requirements, the agent and beamforming weights are output.

[0015] In this embodiment, the calculation process of constructing the OBA table mainly includes the synthetic calculation of the multi-element directional diagram, which is used to support the operation of the reinforcement learning algorithm. The reward calculation method of reinforcement learning is set based on the monotonicity, directivity and other indicators of the OBA table. The setting of the reward value will affect the training direction of the reinforcement learning algorithm. The designed reinforcement learning algorithm contains multiple neural networks and an experience replay mechanism. In this type of algorithm, it is necessary to initialize the network weights, experience replay buffer and initial state in the reinforcement learning algorithm. According to the construction of the OBA table, the reinforcement learning algorithm is run. The algorithm will accumulate the experience replay buffer according to the architecture of the reinforcement learning algorithm, and randomly take batches of data from the experience replay buffer, and use the gradient descent method to update the parameters of the reinforcement learning network. With the continuous optimization of the reinforcement learning network, the monotonicity, directivity and other indicators of the OBA table are also improved. When the pre-set indicator requirements are met, the reinforcement learning training is stopped, and the intelligent agent and beamforming weights are output. A single training record of reinforcement learning includes the environment state, action, reward score, etc. of a single trial. These values ​​will be used to update the intelligent agent. The batch data of each run refers to a certain amount of data taken from the experience area for updating the network.

[0016] In some embodiments, the reinforcement learning algorithm includes multiple neural networks and is provided with an experience replay mechanism.

[0017] In some embodiments, the synthetic calculation of the multi-element directional patterns to construct the OBA table includes the following steps: assigning amplitude and phase compensation values ​​to each element so that the phases of the signals received by each element are consistent, and then synthesizing the sum beam and the difference beam, calculating the difference between the sum beam and the difference beam, and calculating the optimal amplitude and optimal phase compensation difference through a reinforcement learning algorithm, so that the OBA curve reaches a preset shape, thereby optimizing the direction finding capability of the OBA table.

[0018] In some embodiments, the monotonicity and directionality-based reinforcement learning reward mechanism of the OBA table includes the following steps: calculating the slope of the OBA, the expected value of the slope is positive on the left and negative on the right, and the middle is the OBA maximum value; calculating the monotonicity reward value according to the slope and expected value of each point; calculating the directionality reward value by the difference between the angle at which the OBA maximum value is located and the wave position pointing angle.

[0019] The second aspect of the present invention provides: an OBA optimization system based on reinforcement learning, used to implement any of the above-mentioned OBA optimization methods based on reinforcement learning, comprising: An OBA table construction module is used to perform synthetic calculations on multi-element directional patterns to construct an OBA table; Reinforcement learning reward mechanism setting module, used to set the reinforcement learning reward mechanism based on the monotonicity and directionality of the OBA table; Initialization module, used to initialize the network weights, experience replay buffer and initial state of the reinforcement learning algorithm; The reinforcement learning network parameter update module is used to run the reinforcement learning algorithm according to the OBA table, accumulate the single training records of reinforcement learning into the experience replay buffer, and randomly take batch data from the experience replay buffer, and use the gradient descent method to update the parameters of the reinforcement learning network to improve the monotonicity and directionality of the OBA table; The intelligent agent and beamforming weight output module is used to output the intelligent agent and beamforming weights when the monotonicity and directivity of the OBA table meet the preset index requirements.

[0020] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned OBA optimization methods based on reinforcement learning is implemented.

[0021] In the present invention, an OBA optimization algorithm is constructed based on a reinforcement learning algorithm to optimize the monotonicity, directivity and other indicators of OBA. The reinforcement learning algorithm can be modified to form different OBA optimization algorithms. The reward of reinforcement learning in the present invention is calculated based on the monotonicity and directivity of OBA, and can also be calculated based on other indicators that can improve the direction finding ability of OBA.

[0022] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. An OBA optimization method based on reinforcement learning, characterized in that: The following steps are involved: In the OBA table construction phase, the OBA table is constructed by performing synthetic calculations on the multi-element directional patterns; In the reinforcement learning reward mechanism setting stage, the reinforcement learning reward mechanism is set based on the monotonicity and directionality of the OBA table; Initialization phase: Initialize the network weights, experience replay buffer, and initial state of the reinforcement learning algorithm; In the reinforcement learning network parameter update phase, the reinforcement learning algorithm is run according to the OBA table, and the single training records of reinforcement learning are accumulated in the experience replay buffer. Then batches of data are randomly taken from the experience replay buffer, and the parameters of the reinforcement learning network are updated using the gradient descent method to improve the monotonicity and directionality of the OBA table. In the stage of outputting the agent and beamforming weights, when the monotonicity and directivity of the OBA table meet the preset index requirements, the agent and beamforming weights are output.

2. The OBA optimization method based on reinforcement learning according to claim 1, characterized in that: The reinforcement learning algorithm includes multiple neural networks and is provided with an experience replay mechanism.

3. The OBA optimization method based on reinforcement learning according to claim 1, characterized in that: The method of synthesizing and calculating the multi-element directional patterns to construct an OBA table includes the following steps: assigning amplitude and phase compensation values ​​to each element so that the phases of the signals received by each element are consistent, and then synthesizing the sum beam and the difference beam, calculating the difference between the sum beam and the difference beam, and calculating the optimal amplitude and optimal phase compensation difference through a reinforcement learning algorithm, so that the OBA curve reaches a preset shape, thereby optimizing the direction finding capability of the OBA table.

4. The OBA optimization method based on reinforcement learning according to claim 1, characterized in that: The monotonicity and directivity-based reinforcement learning reward mechanism of the OBA table includes the following steps: calculating the slope of the OBA, wherein the expected value of the slope is positive on the left and negative on the right, and the middle is the OBA maximum value; calculating the monotonicity reward value according to the slope and the expected value of each point; and calculating the directivity reward value by the difference between the angle at which the OBA maximum value is located and the wave position directivity angle.

5. An OBA optimization system based on reinforcement learning, characterized in that: The method for implementing the OBA optimization method based on reinforcement learning as claimed in any one of claims 1 to 4 comprises: An OBA table construction module is used to perform synthetic calculations on multi-element directional patterns to construct an OBA table; Reinforcement learning reward mechanism setting module, used to set the reinforcement learning reward mechanism based on the monotonicity and directionality of the OBA table; Initialization module, used to initialize the network weights, experience replay buffer and initial state of the reinforcement learning algorithm; The reinforcement learning network parameter update module is used to run the reinforcement learning algorithm according to the OBA table, accumulate the single training records of reinforcement learning into the experience replay buffer, and randomly take batch data from the experience replay buffer, and use the gradient descent method to update the parameters of the reinforcement learning network to improve the monotonicity and directionality of the OBA table; The intelligent agent and beamforming weight output module is used to output the intelligent agent and beamforming weights when the monotonicity and directivity of the OBA table meet the preset index requirements.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the OBA optimization method based on reinforcement learning as described in any one of claims 1 to 4 is implemented.

Images