Intelligent design method of low-pass filter based on reinforcement learning

Abstract

The invention discloses an intelligent design method for a low-pass filter based on reinforcement learning. The method comprises the following steps: step 1, obtaining structural parameters of the low-pass filter for matrix representation; 2, constructing a reinforcement learning environment based on a deep reinforcement learning framework, and taking a result of the step 1 as state input; 3, designing an intelligent agent, and enabling the intelligent agent to interact with the environment established in the step 2 based on a deep reinforcement learning framework; step 4, according to an interaction result of the step 3, converting a performance index of the low-pass filter into corresponding reward functions # imgabs0 # and # imgabs1 #, and taking a difference value # imgabs2 # between a reward corresponding to a current state and a reward corresponding to a previous state as a final reward to guide an optimization process; and step 5, after the model reaches a predetermined training target, determining final parameters of the low-pass filter according to the structure parameters obtained by final adjustment. Through the systematic steps, intelligent optimization of the low-pass filter design can be achieved, and the design efficiency and the filter performance are remarkably improved.

Description

Technical Field

[0001] The invention belongs to the field of electronic information, and relates to a filter design method, and in particular to an intelligent design method for a low-pass filter based on reinforcement learning. Background Art

[0002] With the continuous advancement of the global scientific and technological revolution and industrial transformation, communication spectrum resources are becoming increasingly scarce and communication protocols are becoming more complex. As key components for screening frequencies and suppressing out-of-band interference and noise, RF / microwave filters occupy an irreplaceable position in wireless communication systems and have long dominated the largest share of the RF front-end market. The huge market demand has not only promoted the rapid development of microwave filter technology, but also put forward higher requirements on the performance of the filter. The design method of the filter directly determines its performance. Therefore, how to improve the hardware performance of the filter through innovative software design methods has become a key and hot issue in the current research field.

[0003] Traditional filter design methods mainly rely on manual adjustment and optimization of parameters, but with the continuous increase in device complexity, this method has become inefficient and difficult. There is a highly complex nonlinear relationship between the structural parameters and performance of the filter, which makes parameter tuning more challenging. Traditional methods face the problems of long time, low efficiency and difficulty in optimization. The introduction of reinforcement learning (RL) provides a new idea for solving these problems. Its advantage is that it can realize the automation of parameter tuning. Through autonomous exploration of parameter space and trial and error learning, it can quickly find the optimal or suboptimal filter structure parameters, significantly reducing the time of manual intervention. At the same time, the reinforcement learning algorithm has efficient iteration capabilities and can quickly evaluate the performance of parameter combinations through a large number of simulations and experiments, thereby accelerating the optimization process and improving design efficiency. The powerful learning ability of reinforcement learning can capture the complex relationship between filter structure parameters and performance, providing support for more efficient and accurate design. In summary, the application of reinforcement learning in filter design not only effectively solves the drawbacks of traditional methods, but also significantly improves design efficiency, injecting new vitality into the development of filter technology. Summary of the invention

[0004] The purpose of this application is to provide an intelligent design method for low-pass filters based on reinforcement learning, aiming to solve the problems of long filter design time, low efficiency and difficulty in parameter adjustment and optimization existing in related technologies.

[0005] The technical solution is as follows:

[0006] The present invention proposes a low-pass filter intelligent design method based on reinforcement learning, which specifically includes the following steps:

[0007] Step 1: Matrixization of low-pass filter structural parameters: The low-pass filter structural parameters need to be converted into a matrix form suitable for calculation and analysis.

[0008] Step 2, build a reinforcement learning environment: build a reinforcement learning environment so that the agent can interact with it and define the state space and action space. The state space is the matrixed low-pass filter state, and the action space can be the adjustment of various design parameters.

[0009] Step 3: Design the interaction between the agent and the environment: determine how the agent makes decisions and learns in the environment.

[0010] Step 4, conversion of performance indicators to reward functions: Convert performance indicators into rewards. For example, you can set a positive reward when the filter performance meets the standard. The design of the reward function should encourage the agent to take actions that can improve performance.

[0011] Step 5, determine the final parameters: Through the strategy obtained through training, evaluate whether the parameters selected by the agent meet the design requirements.

[0012] The specific process of step 1 is:

[0013] First, the low-pass filter contains multiple adjustable structural parameters. These structural parameters have a significant impact on the performance of the filter and need to be carefully selected. These structural parameters are matrixed for system analysis and adjustment in subsequent steps.

[0014] The specific process of step 2 is:

[0015] Through the deep reinforcement learning framework, a special reinforcement learning environment is constructed, and the matrixed structural parameters in step 1 are used as the initial state input of the environment. The core of the environment includes three key modules: environment modeling, simulation interface, and reward function design. In the environment modeling, the state space is constructed based on the structural parameter matrix of the low-pass filter, and the action space is defined as the possible adjustment operations of these structural parameters. The simulation interface is automatically called through the API of the electromagnetic simulation software, thus ensuring efficient docking and seamless integration with the reinforcement learning framework.

[0016] The specific process of step 3 is:

[0017] An intelligent agent is designed based on the deep reinforcement learning framework. Its architecture includes key modules such as action space, experience pool, neural network, evaluation network, target network and greedy strategy. The action space dynamically optimizes the network architecture by adjusting structural parameters (such as network depth, convolution kernel size, etc.); the experience pool adopts a priority experience playback mechanism to store and preferentially extract important samples for training to improve learning efficiency and stability; the neural network uses a convolutional neural network (CNN) to extract features from the input state and output the results of each action. value, providing a basis for strategy optimization; the evaluation network is responsible for selecting a strategy based on the current state and updating The target network calculates the target The value is used to reduce the estimation bias and ensure the stability of the learning process; the greedy strategy balances exploration and utilization through the ϵ-greedy method and gradually optimizes the strategy performance. The sample sampling mechanism of the experience pool is shown in formula (1), which further improves the efficiency of data utilization.

[0018]

[0019] In the formula, is the current state, is the currently selected action, is the current reward. is the next state;

[0020] The convolutional neural network is shown in formula (2):

[0021]

[0022] In the formula, Indicates in status Next Action What you get value, Indicates the input status Row, No. The eigenvalues ​​at the columns, is a convolution kernel, represents the bias term;

[0023] The value is calculated as shown in formula (3):

[0024]

[0025] In the formula, Indicates in status Take action The optimal action value function is is an operator that selects the largest value from a series of values. represents a policy, which represents the probability distribution of choosing an action in a given state, is a constant between 0 and 1, called the discount factor;

[0026] The network update is shown in formula (4):

[0027]

[0028] Gradient descent is shown in formula (5):

[0029]

[0030] The specific process of step 4 is:

[0031] The performance of the low-pass filter is mainly determined by factors such as cutoff frequency, transition band width and passband characteristics. The reward function design is divided into two parts: (Rewards related to performance within the passband and rewards related to performance within the transition band). Rewards related to performance within the passband; The value is compared with -3dB. When the value is greater than -3dB, a higher reward is given; when the frequency point within the passband When the value is less than -3dB, a very low reward is given. Reward function The calculation formula is as follows:

[0032]

[0033] In the above formula, represents the weight coefficient, , is a constant and , is the reward calculated for the current state, is within the low-pass filter passband Parameter point value.

[0034] To calculate the reward of the transition zone, the area under the curve is calculated using the trapezoidal rule. The result of the integral is approximated by dividing the area under the curve into multiple trapezoids and summing them. The calculation formula is as follows:

[0035]

[0036] In the above formula, and The sum of the absolute values ​​of the coordinates. Used to calculate the average height of the trapezoid; the sum of the absolute values ​​of the coordinates of two adjacent points The coordinate difference represents the length of the base of the trapezoid.

[0037]

[0038]

[0039] Two parts of the reward and , the difference between the current state and the previous state is the final reward R.

[0040] The specific process of step 5 is:

[0041] When the model reaches the predetermined training target, it indicates that the agent has successfully found the optimized parameter combination. According to the final adjusted state of the agent, the final parameters of the low-pass filter are determined, thus completing the entire design process.

[0042] Through this series of carefully designed steps, the reinforcement learning framework of the present invention can intelligently complete the design task of the low-pass filter and significantly improve the design efficiency and filter performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 :Schematic diagram of a low-pass filter structure

[0044] Figure 2 : Schematic diagram of the process of the present invention

[0045] Figure 3 : Reinforcement learning DQN algorithm architecture diagram

[0046] Figure 4 : Schematic diagram of reward function design

[0047] Figure 5 : Reward curve during training

[0048] Figure 6 : The optimal strategy Result Plot DETAILED DESCRIPTION

[0049] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0050] The embodiments disclosed in the present invention are applicable to any type of low-pass filter. For ease of explanation, the LC filter is used as an example in the embodiments of the present invention. Figure 1 shown.

[0051] Figure 2 A flow chart of the intelligent design method of low-pass filter based on reinforcement learning is presented.

[0052] Step 1: The low-pass filter contains multiple adjustable structural parameters. These structural parameters have a significant impact on the performance of the filter and need to be carefully selected. These structural parameters are matrixed for system analysis and adjustment in subsequent steps.

[0053] The following table shows the initial structural parameters of the low-pass filter (unit / mm)

[0054] W1 L1 W2 L2 W3 L3 W4 L4 W5 L5 W6 L6 W7 L7 3.0 3.0 8.0 5.2 2.3 5.4 15.0 4.8 1.0 5.5 12.8 5.2 1.0 5.4

[0055] Step 2, see Figure 3 As shown in the figure, a special reinforcement learning environment is constructed through the deep reinforcement learning framework, and the matrixed structural parameters in step 1 are used as the initial state input of the environment. The core of the environment includes three key modules: environment modeling, simulation interface, and reward function design. In environmental modeling, the state space is constructed based on the structural parameter matrix of the low-pass filter, and the action space is defined as the possible adjustment operations of these structural parameters. The simulation interface realizes automatic calling through the API of the electromagnetic simulation software, thereby ensuring efficient docking and seamless integration with the reinforcement learning framework. Two initial structural parameters W3 and W5 are selected. Its action space update operation can be described as: Assume ,in , are the coordinates of the agent in space (or the values ​​of state variables), then depending on the action, the update rule can be expressed as:

[0056]

[0057] x and y represent The first and second elements in . According to the value of the action, we Perform corresponding addition and subtraction operations to represent the movement of the agent in space (or change of state).

[0058] Step 3, design an intelligent agent based on the deep reinforcement learning framework, whose structure consists of the following key modules: action space, experience pool, neural network, evaluation network, target network and greedy strategy.

[0059] The performance of the low-pass filter is mainly determined by factors such as cutoff frequency, transition band width and passband characteristics. The reward function design is divided into two parts: (Rewards related to performance within the passband and rewards related to performance within the transition band). The reward is related to the performance within the passband; the frequency points within the passband The value is compared with -3dB. When the value is greater than -3dB, a higher reward is given; when the frequency point within the passband When the value is less than -3dB, a very low reward is given. Reward function The calculation formula is as follows:

[0060]

[0061] In the above formula, represents the weight coefficient, , is a constant and , is the reward calculated for the current state, is within the low-pass filter passband Parameter point value.

[0062] To calculate the reward of the transition zone, the area under the curve is calculated using the trapezoidal rule. The result of the integral is approximated by dividing the area under the curve into multiple trapezoids and summing them. The calculation formula is as follows:

[0063]

[0064] In the above formula, and The sum of the absolute values ​​of the coordinates. Used to calculate the average height of the trapezoid; the sum of the absolute values ​​of the coordinates of two adjacent points The coordinate difference represents the length of the base of the trapezoid.

[0065]

[0066]

[0067] Two parts of the reward and , the difference between the current state and the previous state is the final reward R.

[0068] The low-pass filter optimization task indicators are formulated as shown in the following table:

[0069] Task Cut-off frequency (GHz) 1 3.4 2 2.0

[0070] It should be noted that different types of low-pass filters have different passbands. The frequency response curve is required to be as flat as possible; and within the transition band, The frequency response curve needs to drop quickly to form a narrower transition band to achieve better filtering performance.

[0071] Step 5: According to the above different task indicators, the structural parameters of the filter are intelligently optimized and designed. The results are shown in the following table:

[0072] result W3(mm) W5(mm) Task-1 2.1 3.2 Task-2 2.5 2.6

[0073] It should be noted that see Figure 5 As shown in the figure, two different tasks (Task-1, Task-2) correspond to the design of two different low-pass filters. The performance after a series of training rounds (Epochs). The change of the reward value of each task with the training round is visualized through a line chart.

[0074] Initial stage: The reward values ​​of the two curves are low and fluctuate significantly. This shows that the model has not yet learned an effective strategy in the early stage of training. Mid-term stage: The reward value rises rapidly and the model performance improves significantly. Late stage: The reward tends to be stable, and the red and blue curves hover around the reward of 450, indicating that the model has basically converged and learned a better strategy.

[0075] The structural parameters calculated in step 5 are obtained by electromagnetic simulation tools. The curves show that both designed low-pass filters meet the performance indicators required for the design. In Task-1, the agent first learns to adjust the parameters so that the frequency curve gradually shifts to the right until the design of the Task-1 low-pass filter is completed. In Task-2, the agent first shifts the frequency curve to the right through parameter adjustment, and then adjusts the parameters to achieve a left shift until the design of the Task-2 low-pass filter is completed.

[0076] It should be noted that the above specific implementation is only exemplary. Under the above guidance of the present invention, those skilled in the art can make various improvements and modifications based on the above embodiments, and these improvements or modifications are within the protection scope of the present invention.

[0077] Those skilled in the art should understand that the above detailed description is only for the purpose of explaining the present invention, and is not intended to limit the present invention. The protection scope of the present invention is defined by the claims and their equivalents.

Claims

1. A low-pass filter intelligent design method based on reinforcement learning, characterized in that: The specific steps include: Step 1, matrixing the structural parameters of the low-pass filter: The structural parameters of the low-pass filter need to be converted into a matrix form suitable for calculation and analysis; Step 2, establish a reinforcement learning environment: build a reinforcement learning environment so that the agent can interact with it and define the state space and action space; the state space is the matrixed low-pass filter state, and the action space is the adjustment of various design parameters; Step 3: Design the interaction between the agent and the environment: formulate how the agent makes decisions and learns in the environment; Step 4: Conversion of performance indicators to reward functions: convert performance indicators into rewards; for example, you can set a positive reward when the filter performance meets the standard; The design of the reward function should encourage the agent to take actions that improve performance; Step 5, determine the final parameters: Through the strategy obtained through training, evaluate whether the parameters selected by the agent meet the design requirements.

2. The low-pass filter intelligent design method based on reinforcement learning according to claim 1, characterized in that: The specific process of step 1 is as follows: First, the low-pass filter contains multiple adjustable structural parameters; these structural parameters have a significant impact on the performance of the filter and need to be carefully selected; these structural parameters are then matrixed to facilitate system analysis and adjustment in subsequent steps.

3. The intelligent design method of a low-pass filter based on reinforcement learning according to claim 1, characterized in that: The specific process of step 2 is as follows: A special reinforcement learning environment is constructed through the deep reinforcement learning framework, and the matrixed structural parameters in step 1 are used as the initial state input of the environment; the core of the environment includes three key modules: environment modeling, simulation interface, and reward function design; in environment modeling, the state space is constructed based on the structural parameter matrix of the low-pass filter, and the action space is defined as the possible adjustment operations on these structural parameters; the simulation interface realizes automatic calling through the API of the electromagnetic simulation software, thereby ensuring efficient docking and seamless integration with the reinforcement learning framework.

4. The low-pass filter intelligent design method based on reinforcement learning according to claim 1, characterized in that: The specific process of step 3 is as follows: An intelligent agent is designed based on the deep reinforcement learning framework, whose architecture includes key modules such as action space, experience pool, neural network, evaluation network, target network and greedy strategy; The action space dynamically optimizes the network architecture by adjusting structural parameters (such as network depth, convolution kernel size, etc.); the experience pool adopts a priority experience playback mechanism to store and prioritize important samples for training, thereby improving learning efficiency and stability; the neural network uses a convolutional neural network (CNN) to extract features from the input state and output the results of each action. value, providing a basis for strategy optimization; the evaluation network is responsible for selecting a strategy based on the current state and updating The target network calculates the target The value is used to reduce the estimation bias and ensure the stability of the learning process. The greedy strategy balances exploration and utilization through the ϵ-greedy method and gradually optimizes the strategy performance. The sample sampling mechanism of the experience pool is shown in formula (1), which further improves the efficiency of data utilization. ； In the formula, is the current state, is the currently selected action, is the current reward. is the next state; The convolutional neural network is shown in formula (2): ； In the formula, Indicates in status Next Action What you get value, Indicates the input status Row, No. The eigenvalues ​​at the columns, is a convolution kernel, represents the bias term; The value is calculated as shown in formula (3): ； In the formula, Indicates in status Take action The optimal action value function is is an operator that selects the largest value from a series of values. represents a policy, which represents the probability distribution of choosing an action in a given state, is a constant between 0 and 1, called the discount factor; The network update is shown in formula (4): ； The gradient descent is shown in formula (5): 。 5. The low-pass filter intelligent design method based on reinforcement learning according to claim 1 is characterized in that: The specific process of step 4 is as follows: The performance of the low-pass filter is mainly determined by factors such as cutoff frequency, transition band width and passband characteristics; the reward function design is divided into two parts: (a reward related to performance within the passband and a reward related to performance within the transition band); Rewards related to performance within the passband; The frequency points in the passband The value is compared with -3dB. When the value is greater than -3dB, a higher reward is given; When the frequency point in the passband When the value is less than -3dB, a penalty is given; reward function The calculation formula is as follows: ； In the above formula, represents the weight coefficient, , is a constant and , is the reward calculated for the current state, is within the low-pass filter passband Parameter point value; For calculating the bonus in the transition zone, the area under the curve is calculated using the trapezoidal rule. The result of the integral is approximated by dividing the area under the curve into multiple trapezoids and summing them. The calculation formula is as follows: ； In the above formula, and The sum of the absolute values ​​of the coordinates; used to calculate the average height of the trapezoid; the sum of the absolute values ​​of the coordinates of two adjacent points The coordinate difference represents the length of the base of the trapezoid; ； ； Two parts of the reward and , the difference between the current state and the previous state is the final reward R.

6. The low-pass filter intelligent design method based on reinforcement learning according to claim 1 is characterized in that: The specific process of step 5 is as follows: When the predetermined training goal is reached, it indicates that the intelligent agent has successfully found the optimized parameter combination; according to the final adjusted state of the intelligent agent, the final parameters of the low-pass filter are determined, thus completing the entire design process.

Images