Electromagnetic most sensitive waveform test method based on reinforcement learning

Abstract

The invention discloses an electromagnetic most sensitive waveform test method based on reinforcement learning. The method comprises the following steps: S1, establishing a reinforcement learning architecture of an electromagnetic sensitivity test, and constructing a state space and an action space; s2, establishing a reinforcement learning model based on the double-delay depth deterministic strategy gradient network; s3, establishing a sensitivity threshold automatic closed-loop test environment; s4, constructing an electromagnetic sensitive reward function, and designing a basic reward function and a shaping reward function; and S5, starting electromagnetic most sensitive waveform test and reinforcement learning training to obtain the most sensitive waveform. According to the method, the most sensitive waveform which enables the sensitivity threshold of the tested object to be the lowest can be quickly searched in the delimited modulation waveform set.

Description

Technical Field

[0001] The present invention relates to electromagnetic sensitivity testing, and in particular to a method for testing electromagnetic most sensitive waveforms based on reinforcement learning. Background Art

[0002] Electromagnetic susceptibility (EMS) refers to the property of a device, component, or system that may degrade its operating performance due to electromagnetic interference. It has become a key factor affecting the reliability and integrity of electronic equipment. Electromagnetic susceptibility can be measured by conducting EMS testing on electronic equipment. This test applies an interference signal of a certain power to the device under test (EUT). When the interference power exceeds a certain threshold, the device's operating performance degrades. This threshold is defined as the sensitivity threshold, which reflects the EUT's electromagnetic sensitivity. As the integration of devices becomes increasingly higher and the electromagnetic environment in which they are located becomes increasingly complex, revealing the subtle differences in the device's electromagnetic sensitivity to different interference signals is crucial to ensuring the safe and stable operation of the equipment. Therefore, it is very meaningful to find a signal waveform that can stimulate the EUT's electromagnetic susceptibility at a lower power.

[0003] Existing EMS testing standards use fixed waveforms for frequency sweep testing to determine sensitivity thresholds at different frequencies. For example, GJB-151B recommends a pulse amplitude modulation (PM) signal with a duty cycle of 50, a pulse repetition frequency of 1kHz, and a modulation depth of 100%. The International Electrotechnical Commission standard IEC 61000-4-3 recommends a sinusoidal amplitude modulation (AM) waveform with a modulation frequency of 1kHz and a modulation depth of 80%. These standards recommend waveforms based on potential interference in the real world. The waveform recommended by GJB-151B resembles interference caused by switching power supplies, while the waveform recommended by IEC 61000-4-3 could be interference caused by AM broadcast signals. Furthermore, interference generated by frequency modulation (FM) broadcasts or FM continuous wave radars can potentially trigger electromagnetic susceptibility. While the fixed waveforms recommended by these standards are easy to implement and generalize, they may not fully capture the specific electromagnetic susceptibility characteristics of certain EUTs.

[0004] Several studies, both domestically and internationally, have used complex test waveforms to test specific EUTs. For example, signal waveforms such as single-carrier frequency-division multiple access (SCFDMA), quadrature phase-shift keying (QPSK), and filtered noise have been used in radiated susceptibility testing of avionics systems. Furthermore, researchers have studied the electromagnetic susceptibility of CAN networks to electrical fast transient (EFT) pulse interference, the electromagnetic susceptibility of personal computer systems to double-exponential electromagnetic pulse interference, and the electromagnetic susceptibility of temperature sensors to basic emission element waveform interference. These studies consider various complex waveforms that may occur in real-world environments, but they may not represent the waveforms to which the EUT is most sensitive.

[0005] Some research has utilized optimization algorithms to determine the parameters of the most sensitive waveforms. For example, Bayesian optimization can be used for adaptive sensitivity testing of drone data links. It searches for combinations of interference waveform parameters that can cause sensitivity in complex electromagnetic environments and predicts the sensitivity threshold. When the operating principle of the EUT is known, traditional optimization methods can determine the most sensitive waveform that stimulates electromagnetic sensitivity in the EUT. However, for EUTs whose operating principle is unknown, scanning parameters or optimizing across a vast waveform parameter space can incur prohibitive computational overhead.

[0006] Reinforcement learning's optimization capabilities in high-dimensional spaces and adaptability to complex environments make it well-suited for waveform parameter search in EMS testing. Reinforcement learning can determine optimal strategies in complex, high-dimensional spaces, demonstrating strong learning and optimization capabilities even within vast parameter spaces. Recent research has demonstrated the potential for reinforcement learning in the field of electromagnetic compatibility, with applications such as optimizing PCB ground via layout and pin assignments in chip system-in-packages.

[0007] In summary, the existing standard test methods cannot obtain the most sensitive waveform, and the existing optimization methods expose some limitations when searching for the most sensitive waveform. Therefore, it is urgent to develop a new intelligent test method to solve these problems. Summary of the Invention

[0008] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide an electromagnetic most sensitive waveform testing method based on reinforcement learning, which can quickly search for the most sensitive waveform that makes the sensitivity threshold of the test object lowest in the defined modulation waveform set.

[0009] The object of the present invention is achieved through the following technical solution: a method for testing the most sensitive electromagnetic waveform based on reinforcement learning, comprising the following steps:

[0010] S1. Establish a reinforcement learning architecture for electromagnetic susceptibility testing and construct state space and action space;

[0011] The step S1 comprises:

[0012] The RF signal source in EMS testing is used as an agent in reinforcement learning. Its goal is to learn how to take actions at each step to maximize the cumulative reward and minimize the sensitivity threshold of the test object stimulated by the test waveform.

[0013] The test system in EMS testing is used as the environment in reinforcement learning to receive the signal waveform output of the RF signal source and return the sensitive state of the test product;

[0014] The signal source waveform parameters and the sensitive state of the test object in the EMS test are used as the states in the reinforcement learning. A total of 8 waveform parameters and 2 sensitive state parameters together constitute the state space in the reinforcement learning. Each parameter in the state space is a state, totaling 10 states, and the value of each state is in the range of [-1, 1].

[0015] The changes in waveform parameters in the EMS test are used as actions in reinforcement learning, with the aim of changing the current state of the agent in order to obtain greater rewards. There are a total of 8 actions, corresponding to changes in 8 waveform parameters, namely, changes in carrier frequency, modulation type parameters, modulation waveform parameters, modulation frequency, modulation depth, frequency offset, pulse duty cycle, and pulse period. The values ​​of these 8 actions are all in the range of [-1,1], constituting the action space in reinforcement learning.

[0016] The eight waveform parameters include:

[0017] Carrier frequency: The symbol is s(1), which represents the normalized carrier frequency value, and the value range is [-1,1]. Let the physical range of the actual carrier frequency be [f min ,f max ], according to s(1 and physical range [f min ,f max ], the real carrier frequency is: The unit is MHz;

[0018] Modulation type: symbol is s(2), value range is [-1, 1], and the value range is divided into four intervals: [-1, -0.5), [-0.5, 0), [0, 0.5), [0.5, 1]; each interval corresponds to a real modulation type, and the real modulation types include no modulation, amplitude modulation, frequency modulation, and pulse modulation;

[0019] Modulation waveform: The symbol is s(3), the value range is [-1, 1], and the value range is divided into five intervals: [-1, -0.6), [-0.6, -0.2), [-0.2, 0.2), [0.2, 0.6), [-0.6, 1); each interval corresponds to a real waveform, including sine, triangle, square wave, positive slope, and negative slope;

[0020] Modulation frequency: The symbol is s(4), which represents the normalized modulation frequency value, and its value range is [-1,1]. Assume that the physical range of the actual modulation frequency is According to s(4) and physical range The actual modulation frequency is: The unit is MHz;

[0021] Modulation depth: The symbol is s(5), which represents the normalized modulation depth value, and the value range is [-1,1]. Let the physical range of the actual modulation depth be [a min ,a max ], according to s(5) and the physical range [a min ,a max ], the actual modulation depth is: The unit is %;

[0022] Frequency offset: The symbol is s(6), which represents the normalized frequency offset value, and its value range is [-1,1]. Assume that the physical range of the actual frequency offset is According to s(6) and physical range The actual frequency offset is: The unit is MHz;

[0023] Pulse duty cycle: The symbol is s(7), which represents the normalized pulse duty cycle value, and the value range is [-1,1]. Let the physical range of the actual pulse duty cycle be [k min ,k max ], according to s(7) and the physical range [k min ,k max ], the actual pulse duty cycle is: The unit is %;

[0024] Pulse period: The symbol is s(8), which represents the normalized pulse period value, and the value range is [-1,1]. Let the physical range of the real pulse period be [T min ,T ma , according to s(8) and the physical range [T min ,T ma , the actual pulse period is: The unit is ms;

[0025] The two sensitive status parameters include:

[0026] Sensitivity, symbolized by s(9), indicates whether the test object is sensitive under the current test waveform, and its value range is [-1, 1]. The value range is divided into two intervals: [-1, 0) and [0, 1], which respectively indicate that the test object is sensitive under the current test waveform and the test object is insensitive under the current test waveform.

[0027] The sensitivity threshold, symbolized by s(10), represents the normalized sensitivity threshold value, and its value range is [-1,1]. Let the physical range of the real sensitivity threshold be [P min ,P max ], according to s(10) and physical range [P min ,P max ], the true sensitivity threshold is: The unit is dBm.

[0028] S2. Establish a reinforcement learning model based on a double-delayed deep deterministic policy gradient network;

[0029] The dual-delayed deep deterministic policy gradient network, i.e., the TD3 network, in step S2 includes an online learning network and a target network, each of which contains an actor network and a dual-Q network, wherein the dual-Q network contains two critic networks;

[0030] Both the actor network and the critic network are fully connected neural networks with two hidden layers, each containing N hidden neurons; the input dimension of the actor network is 10 for inputting state and the output dimension is 8 for outputting action, while the input dimension of the critic network is 18 for inputting state + action and the output dimension is 1 for outputting Q value; the activation function of the actor network is the hyperbolic tangent function, while the activation function of the critic network is the RELU function.

[0031] S3. Build an automated closed-loop sensitivity threshold test environment;

[0032] The step S3 comprises:

[0033] The output signal of the RF signal source is amplified by a power amplifier. The amplified signal is then connected to a directional coupler. The through-port of the directional coupler is connected to the DUT, injecting the signal power directly into the DUT. The coupled end of the directional coupler is connected to a signal receiver through an attenuator to measure the actual injected power. The power amplifier is powered by a DC power supply. The DUT is fixed on a test bench to improve the repeatability of the experiment.

[0034] The core of the automatic closed-loop test is a control computer, which obtains the sensitivity status of the test object from the microcontroller and monitors the power at the current test frequency through the signal receiver. Based on the information obtained, it makes intelligent decisions and controls the RF signal source to change the output signal waveform. The sensitivity stimulated by the new waveform is fed back to the control computer in real time to support the next intelligent decision, thus forming a closed loop to search for the most sensitive waveform that minimizes the sensitivity threshold of the test object.

[0035] S4. Construct electromagnetic sensitive reward function, design basic reward function and shaping reward function;

[0036] The step S4 comprises:

[0037] The reward function is used to guide the agent to achieve the goal. If the subject shows electromagnetic sensitivity, the agent will receive a reward. Considering that most waveforms do not stimulate the electromagnetic sensitivity of the subject, which will lead to the problem of sparse rewards, a reward shaping method is adopted: the total reward is divided into a basic reward ri b and shaping reward r i s To achieve;

[0038] Basic reward function r i b Will make the agent close to s in the current state min Get a larger reward when s min Indicates the most sensitive waveform state so far;

[0039] The reward mainly considers the carrier frequency s of the most sensitive waveform min (1), the basic reward function of step i is defined as:

[0040]

[0041] in * represents the non-normalized value, is the sensitivity threshold of the most sensitive waveform so far;

[0042] The state of the most sensitive waveform is unknown at the initial stage, so the following shaping reward function is designed to guide the agent to find a lower sensitivity threshold:

[0043]

[0044] The square term helps the agent search for actions that reduce the maximum sensitivity threshold in one step, thereby accelerating the search for the most sensitive waveform. When the sensitivity threshold of the i-th step is the lowest so far, the agent will be given a huge reward r i s =r i s +100;

[0045] In addition, the following integer reward function is used to accelerate convergence:

[0046] r i s =r i s +1ifs i+1 (9)==1

[0047] r i s =r i s -1ifs i+1 (9)==-1

[0048] r i s =r i s +5ifsi+1 (9)==1ands i (9)==-1

[0049] r i s =r i s -10ifs i+1 (9)==-1ands i (9)==1

[0050] Finally, the total reward is expressed as:

[0051] r i =r i b +μr i s

[0052] Where μ is the reward shaping attenuation factor, which starts from 1 and decays linearly to 0 according to the total step size.

[0053] S5. Start the electromagnetic most sensitive waveform test and reinforcement learning training to obtain the most sensitive waveform.

[0054] The step S5 comprises:

[0055] S501. Start the electromagnetic most sensitive waveform test and synchronously train the constructed reinforcement learning model. The training is divided into N ep Each game has N s Step 1: The environment needs to be reset at the beginning of each round: the output of the RF signal source is turned off, the state is reset to a random number uniformly distributed between -1 and 1, and the end of the round flag B is set. done Reset to 0; N warm The step is used as a warm-up phase, during which the action is a random number uniformly distributed between -1 and 1, rather than being inferred by the actor network in the TD3 network; in the non-warm-up phase, based on the current state s i Use the actor network in the online network to infer the action; add a mean of 0 and a variance of E to the action n Gaussian noise is used as exploration noise; the current waveform parameter state is based on the obtained action a i To update:

[0056] s i+1 (1:8)=s i (1:8)+βa i

[0057] Where β represents the maximum step size of the action, and the updated waveform state s i+1 (1:8) needs to be clipped to the range of [-1,1];

[0058] S502. By controlling the RF signal source, the output carrier power is P max The waveform parameter is the state s i+1 (1:8) represents the physical real value. After the signal output stays for 2 seconds, observe the sensitivity of the EUT. If the EUT is not sensitive, set s i+1 (9) = -1 and s i+1 (10) = 1;

[0059] Otherwise, let s i+1 (9)=1, and in [P min ,P max ] range; each change of carrier power should stay for 2 seconds; use binary search method to change carrier power n times to find the sensitivity threshold with an accuracy of

[0060] The sensitivity threshold is changed from [P min ,P max ] is normalized to [-1,1] and recorded in s i+1 (10) in;

[0061] S503. Calculate reward r according to step S4 i , if the sensitivity threshold is the lowest so far, the current state should be recorded in s min middle;

[0062] If this round is over, let B done =1, otherwise let B done =0;

[0063] S504. i 、a i 、r i 、s i+1 and B done Record it in the experience playback buffer and enter the next cycle until each round is completed and the most sensitive waveform state s is obtained min ;

[0064] During the test, every l rdelay Step 1 rγ Decay learning rate l r , every N En Step γ En Attenuated exploration noise variance E n , in T start After the step, every T train The TD3 network is trained once. The training method includes:

[0065] Get a batch of N from the experience replay buffer batch Samplesi 、a i 、r i 、s i+1 , will s i+1 Send it to the actor network of the target network to get a i+1 , will s i+1 and a i+1 The critic network fed into the target network gets Q * 1 and Q * 2. Take Q * 1 and Q * The minimum value of 2 is taken as Q, and the target Q value Q is calculated according to the Bellman formula t =r i +(~B done )γQ, where ~B done Indicates B done Negated, γ represents the discount factor;

[0066] Will s i and a i Send it to the double Q network of the online network to get Q1 and Q2, and calculate MSE Loss1=Q t -Q1,MSELoss2=Q t -Q2, total loss loss = MSE Loss1 + MSE Loss2, using loss as the loss function, and updating the dual-Q network parameters in the online network through the back propagation algorithm;

[0067] The parameters of the actor network in the online network are updated every T during training. delay Step update once, the update process is as follows: i Send it to the actor network of the online network and get Will s i and The double Q network in the online network is fed into the network, and only the value of the first critic network is obtained. Its negative value is used as the loss function, and the parameters of the actor network in the online network are updated through the back-propagation algorithm.

[0068] The weight parameters of the target network are updated every T during training through a soft update process. delay These parameters are merged with the parameters of the online network in the ratio (1-τ):τ).

[0069] The beneficial effects of this invention are: compared to standard electromagnetic susceptibility testing methods and optimized testing methods based on genetic algorithms, this invention can search for the most sensitive waveform within a defined set of modulation signals that minimizes the sensitivity threshold of the test object. This invention also incorporates concepts related to reinforcement learning into EMS testing, constructing a reinforcement learning framework based on a dual-delay deep deterministic policy gradient network and designing a reward function tailored to electromagnetic susceptibility testing. This invention can more quickly determine the most sensitive waveform, while also achieving a lower sensitivity threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of the method of the present invention;

[0071] Figure 2 Schematic diagram of the TD3 network framework for electromagnetic susceptibility testing;

[0072] Figure 3 This is the layout diagram of the sensitivity threshold automated closed-loop test environment;

[0073] Figure 4 This is a schematic diagram of the electromagnetic susceptibility threshold test results of the standard test method;

[0074] Figure 5 Schematic diagram of the optimization process for the minimum electromagnetic sensitivity threshold of the test method using GA optimization;

[0075] Figure 6 Schematic diagram of the process of optimizing the minimum electromagnetic sensitivity threshold of the test method of the present invention. DETAILED DESCRIPTION

[0076] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0077] This paper introduces the concepts of reinforcement learning into EMS testing and proposes a reinforcement learning-based electromagnetic most sensitive waveform testing method, called EMS-RL. This method searches for the most sensitive waveform that minimizes the sensitivity threshold of the test object within a defined set of modulation waveforms. Compared to current standard testing methods and genetic algorithm-based most sensitive waveform search methods, this method determines the most sensitive waveform more quickly and has a lower threshold for stimulating electromagnetic sensitivity in the test object. Specifically:

[0078] like Figure 1 As shown, a method for testing the most sensitive electromagnetic waveform based on reinforcement learning includes the following steps:

[0079] S1. Establish a reinforcement learning architecture for electromagnetic susceptibility testing and construct state space and action space;

[0080] The RF signal source in EMS testing is used as an intelligent agent in reinforcement learning. Its goal is to learn how to take actions at each step to maximize the cumulative reward, thereby minimizing the sensitivity threshold of the test object stimulated by the test waveform.

[0081] The test system in EMS testing is used as the environment in reinforcement learning to receive the signal waveform output of the RF signal source and return the sensitive state of the test product.

[0082] The signal source waveform parameters and the sensitive state of the test object in the EMS test are used as the states in the reinforcement learning. A total of 8 waveform parameters and 2 sensitive state parameters constitute the state space in the reinforcement learning. All state parameters are normalized to the range of [-1, 1]. Their physical meanings and physical ranges in the specific embodiment are shown in Table 1.

[0083] Table 1 State space parameter table of reinforcement learning in the present invention

[0084]

[0085] The waveform parameter changes during the EMS test are used as actions in reinforcement learning, aiming to alter the agent's current state in the hope of obtaining a greater reward. There are eight actions, corresponding to changes in the first eight states: carrier frequency, modulation type parameters, modulation waveform parameters, modulation frequency, modulation depth, frequency offset, pulse duty cycle, and pulse period. These eight actions constitute the action space used in reinforcement learning, and all action parameters are normalized to the range [-1, 1].

[0086] S2. Establish a reinforcement learning model based on a double-delayed deep deterministic policy gradient network;

[0087] A reinforcement learning model is established based on the double-delayed deep deterministic policy gradient network (TD3), such as Figure 2 As shown, the TD3 network mainly includes the following key networks:

[0088] Online learning network: It actively learns and updates parameters from online interactions with the environment, which directly affects the next decision and learning progress.

[0089] Target Network: It is used by both the actor network and the critic network to provide stable targets during the temporal difference (TD) update process, which can reduce the risk of overfitting during learning and improve convergence stability.

[0090] Double Q Network: It uses two independent critic networks to independently estimate the Q value (the potential reward after taking an action in a state) and uses the smaller one to more accurately predict the Q value.

[0091] Actor Network: Based on the current state, it decides the next action and directly optimizes the policy to obtain higher rewards.

[0092] Critic Network: It evaluates the potential reward of taking an action in a given state and guides the agent to take actions with greater rewards by accurately estimating Q-values.

[0093] Both the actor network and the critic network are fully connected neural networks with two hidden layers, each containing N hidden The actor network has an input dimension of 10 (state) and an output dimension of 8 (action), while the critic network has an input dimension of 18 (state + action) and an output dimension of 1 (Q value). The activation function of the actor network is the hyperbolic tangent function, while the activation function of the critic network is the RELU function.

[0094] S3. Build an automated closed-loop sensitivity threshold test environment;

[0095] Compared with the traditional electromagnetic susceptibility test method, the method proposed in the present invention needs to be run in a sensitivity threshold automatic closed-loop test environment.

[0096] The test layout can be based on direct power injection method or high current injection method. The following takes the direct power injection method as an example. Figure 3 As shown in the figure, the output signal of the RF signal source is amplified by a power amplifier. The amplified signal is then fed into a directional coupler. The through-port of the directional coupler is connected to the DUT, injecting the signal power directly into the DUT. The coupled port of the directional coupler is connected to a signal receiver via an attenuator to measure the actual injected power. A DC power supply is used to power the power amplifier. A test bench is used to secure the DUT to improve experimental repeatability.

[0097] The core of the automated closed-loop test is a control computer. It obtains the sensitivity status of the test object from a microcontroller and monitors the power at the current test frequency through a signal receiver. Based on this information, it uses the method proposed in this invention to make intelligent decisions, controlling the RF signal source to change the output signal waveform. The sensitivity stimulated by the new waveform is fed back to the control computer in real time to support further intelligent decision-making, thus forming a closed loop to search for the most sensitive waveform that minimizes the test object's sensitivity threshold.

[0098] S4. Construct electromagnetic sensitive reward function, design basic reward function and shaping reward function;

[0099] The reward function is used to guide the agent to achieve the goal. If the subject shows electromagnetic sensitivity, the agent will receive a reward. However, most waveforms do not stimulate the electromagnetic sensitivity of the subject, which will lead to the problem of sparse rewards. Therefore, a reward shaping method is needed. This can be achieved by dividing the total reward into a basic reward r i b and shaping reward r i s Reward shaping can alleviate the sparse reward problem and accelerate convergence. The design ideas of the basic reward function and the shaped reward function are as follows.

[0100] In this invention, the purpose of the proposed method is to determine the most sensitive waveform. Therefore, the basic reward function r i b Will make the agent close to s in the current state min Get a larger reward when s min Indicates the most sensitive waveform state so far. The reward mainly considers the carrier frequency s of the most sensitive waveform min (1). The basic reward function for step i is defined as:

[0101]

[0102] in * represents the non-normalized value, This is the sensitivity threshold of the most sensitive waveform so far.

[0103] However, the agent does not know the state of the most sensitive waveform at the beginning, so the following shaping reward function is designed to guide the agent to find a lower sensitivity threshold.

[0104]

[0105] The square term helps the agent search for actions that reduce the maximum sensitivity threshold in one step, thereby accelerating the search for the most sensitive waveform. When the sensitivity threshold of the i-th step is the lowest so far, the agent will be given a huge reward r i s =r i s +100. In addition, there are some other integer reward functions to accelerate convergence:

[0106] r i s =r i s +1ifs i+1 (9)==1,

[0107] r i s=r i s -1ifs i+1 (9)==-1,

[0108] r i s =r i s +5ifs i+1 (9)==1ands i (9)==-1,

[0109] r i s =r i s -10ifs i+1 (9)==-1ands i (9)==1.

[0110] Finally, the total reward is expressed as:

[0111] r i =r i b +μr i s ,

[0112] Where μ is the reward shaping decay factor. μ starts at 1 and decays linearly to 0 based on the total step size. The decay of reward shaping helps the agent find ways to lower the sensitivity threshold in the early stages and avoid relying on reward shaping in the later stages.

[0113] S5. Start the electromagnetic most sensitive waveform test and reinforcement learning training to obtain the most sensitive waveform.

[0114] Table 2 Parameters of the test method based on reinforcement learning proposed in this invention

[0115]

[0116]

[0117] Start the electromagnetic most sensitive waveform test and train the reinforcement learning model constructed simultaneously. The relevant parameters are shown in Table 2. The training is divided into N ep Each game has N s The environment needs to be reset at the beginning of each round: the output of the RF signal source is turned off, the state is reset to a random number uniformly distributed between -1 and 1, and the end of the round flag B is set. done is reset to 0. N warm The step is used as a warm-up phase, during which actions are randomly generated with a uniform distribution instead of being inferred by the actor network in the TD3 network. In the non-warm-up phase, based on the current state s iUse the actor network in the online network to infer the action. Add a mean of 0 and a variance of E to the action. n The Gaussian noise is used as the exploration noise. The current waveform parameter state is based on the obtained action a i To update:

[0118] s i+1 (1:8)=s i (1:8)+βa i

[0119] Where β represents the maximum step size of the action. The updated waveform state s i+1 (1:8) needs to be clipped to the range of [-1,1].

[0120] By controlling the RF signal source, the output carrier power is P max The waveform parameter is the state s i+1 (1:8) corresponds to the true value. After the signal output stays for 2 seconds, observe the sensitivity of the EUT. If the EUT is not sensitive, set s i+1 (9) = -1 and s i+1 (10) = 1. Otherwise, let s i+1 (9)=1, and in [P min ,P max ] perform binary search within the range of P min =-50dBm and P max = -10dBm. Each change in carrier power should stay for 2 seconds. Use binary search method to change the carrier power. n times to find the sensitivity threshold, the accuracy of which can reach Normalize the sensitivity threshold from [-50, -10] to [-1, 1] and record it in s i+1 (10). Then, the reward r is calculated according to the detailed method of step S4. i If the sensitivity threshold is the lowest so far, the current state should be recorded in s min If this round is over, let B done =1, otherwise let B done = 0. Finally, s i 、a i 、r i 、s i+1 and B done Record it in the experience playback buffer. Finally, enter the next loop until each round is finished. During the test, every l rdelay Step 1 rγ Decay learning rate l r , every N En Step γ En Attenuated exploration noise variance En , in T start After the step, every T train The TD3 network is trained once. The specific training methods include:

[0121] Get a batch of N from the experience replay buffer batch Samples i 、a i 、r i 、s i+1 , will s i+1 Send it to the actor network of the target network to get a i+1 , will s i+1 and a i+1 The critic network fed into the target network gets Q * 1 and Q * 2. Take Q * 1 and Q * The minimum value of 2 is taken as Q, and the target Q value Q is calculated according to the Bellman formula t =r i +(~B done )γQ, where ~B done Indicates B done Negated, γ represents the discount factor;

[0122] Will s i and a i Send it to the double Q network of the online network to get Q1 and Q2, and calculate MSE Loss1=Q t -Q1,MSELoss2=Q t -Q2, total loss loss = MSE Loss1 + MSE Loss2, using loss as the loss function, and updating the dual-Q network parameters in the online network through the back propagation algorithm;

[0123] The parameters of the actor network in the online network are updated every T during training. delay Step update once, the update process is as follows: i Send it to the actor network of the online network and get Will s i and The double Q network in the online network is fed into the network, and only the value of the first critic network is obtained. Its negative value is used as the loss function, and the parameters of the actor network in the online network are updated through the back-propagation algorithm.

[0124] The weight parameters of the target network are updated every T during training through a soft update process. delay These parameters are merged with the parameters of the online network in the ratio (1-τ):τ).

[0125] The technical effects of the present invention are described in detail below in conjunction with actual measurement experiments.

[0126] 1. Test object:

[0127] Using a direct power injection layout, the ADS1110 analog-to-digital converter (A / D) and the OPA2192 operational amplifier (OPA) were selected as test devices for testing. The test chips were mounted on a custom PCB and secured to a test bench. Within the PCB, a test interference signal was injected into the chip's VCC pin via a 6.8nF isolation capacitor.

[0128] 2. Obtaining the test product status:

[0129] The AD chip is used in the voltage acquisition circuit. The microcontroller reads the AD chip control register data via the I2C bus and forwards it to the control computer. The OPA chip is used in a second-order Butterworth low-pass filter circuit with a gain of 1. The microcontroller reads the output voltage data and forwards it to the control computer.

[0130] 3. Sensitive criteria:

[0131] For AD chips, if the read data is inconsistent with the written data, the test product is considered to be sensitive. For OPA chips, if the output DC voltage differs by 0.1V from the output when not disturbed, the test product is considered to be sensitive.

[0132] 4. Test result analysis

[0133] The test methods using the continuous wave waveform (CW), the GJB 151B CS114 standard test waveform (PM-S) and the IEC 61000-4-3 standard test waveform (AM-S) were selected as comparative test methods, and the genetic algorithm (GA) was selected as the comparative optimization method of the reinforcement learning method proposed in the present invention.

[0134] In the experiment using the standard test method, the carrier frequency of the test interference signal was swept from 20MHz to 400MHz in 1MHz increments. The measured sensitivity thresholds of the AD chip and OPA chip changed with frequency as shown in the following table: Figure 4 (a) and Figure 4 (b) shown.

[0135] In the test method using genetic algorithm optimization, the first 8 states (waveform parameters) are used as optimization variables, and the sensitivity threshold is used as the optimization target value. The population size is 50, the maximum number of iterations is 20, the mutation probability is 0.1, and the selection operator is a tournament operator with a scale of 30. The optimization process of the minimum electromagnetic sensitivity threshold of the AD chip and the OPA chip is as follows: Figure 5 As shown in (a) and (b).

[0136] The test method of the present invention is used, and the specific parameters are shown in Table 2. The optimization process of the minimum electromagnetic sensitivity threshold of the AD chip and the OPA chip is as follows: Figure 6 As shown in (a) and (b).

[0137] Test results show that the minimum sensitivity thresholds for the standard test waveforms are -30dBm and -24dBm, respectively. Using a genetic algorithm to optimize the search for the most sensitive waveform, the resulting minimum sensitivity thresholds are -35.20dBm and -26.37dBm, respectively. Using the proposed method to find the most sensitive waveform, the resulting minimum sensitivity thresholds are -35.20dBm and -28.71dBm, respectively.

[0138] Test results using the method of the present invention show that for AD chips, the most sensitive waveform is a square wave amplitude modulation signal with a carrier frequency of 20MHz, a modulation frequency of 34kHz, and a modulation depth of 100%. For OPA chips, the most sensitive waveform is a sinusoidal frequency modulation signal with a carrier frequency of 82.49MHz, a modulation frequency of 19kHz, and a modulation frequency deviation of 175kHz. These most sensitive waveforms are 5.20dB and 4.71dB lower than the minimum sensitivity thresholds of standard test waveforms, respectively. The result for the OPA chip is 2.34dB lower than that for the GA.

[0139] In terms of test efficiency, the present invention found the most sensitive waveform at the 133rd and 206th steps, respectively, while the GA method found the most sensitive waveform after the 12th and 20th iterations. Considering that there are 20 populations in one iteration, the number of test steps is 240 and 1000. The number of steps required for the test is much higher than that of the present invention, which proves that the test method of the present invention is more efficient.

[0140] The foregoing description is a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Instead, the present invention can be used in other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A method for testing the most sensitive electromagnetic waveform based on reinforcement learning, characterized in that: The following steps are involved: S1. Establish a reinforcement learning framework for electromagnetic susceptibility testing and construct state space and action space; S2. Establish a reinforcement learning model based on a double-delayed deep deterministic policy gradient network; S3. Build a sensitivity threshold automated closed-loop test environment; S4. Construct electromagnetic sensitive reward function, design basic reward function and shaping reward function; S5. Start the electromagnetic most sensitive waveform test and reinforcement learning training to obtain the most sensitive waveform.

2. The electromagnetic most sensitive waveform testing method based on reinforcement learning according to claim 1, characterized in that: The step S1 comprises: The RF signal source in the EMS test is used as an intelligent agent in reinforcement learning. Its goal is to learn how to take actions at each step to maximize the cumulative reward and minimize the sensitivity threshold of the test object stimulated by the test waveform. The test system in the EMS test is used as the environment in reinforcement learning to receive the signal waveform output of the RF signal source and return the sensitive state of the test product; The signal source waveform parameters and the sensitive state of the test object in the EMS test are used as the states in the reinforcement learning. A total of 8 waveform parameters and 2 sensitive state parameters together constitute the state space in the reinforcement learning. Each parameter in the state space is a state, a total of 10 states, and the value of each state is in the range of [-1,1]; The changes in waveform parameters in the EMS test are used as actions in reinforcement learning, with the aim of changing the current state of the agent in order to obtain greater rewards. There are a total of 8 actions, corresponding to changes in 8 waveform parameters, namely, changes in carrier frequency, modulation type parameters, modulation waveform parameters, modulation frequency, modulation depth, frequency offset, pulse duty cycle, and pulse period. The values ​​of these 8 actions are all in the range of [-1,1], constituting the action space in reinforcement learning.

3. The electromagnetic most sensitive waveform testing method based on reinforcement learning according to claim 2 is characterized in that: The eight waveform parameters include: Carrier frequency: The symbol is s(1), which indicates the normalized carrier frequency value, and its value range is [-1,1]. Assume that the physical range of the actual carrier frequency is [f min ,f max ], according to s(1 and physical range [f min ,f max ], the actual carrier frequency is: The unit is MHz; Modulation type: the symbol is s(2), the value range is [-1,1], and the value range is divided into four intervals: [-1, -0.5), [-0.5, 0), [0, 0.5), [0.5, 1]; each interval corresponds to a real modulation type, and the real modulation types include no modulation, amplitude modulation, frequency modulation, and pulse modulation; Modulation waveform: the symbol is s(3), the value range is [-1,1], and the value range is divided into five intervals: [-1, -0.6), [-0.6, -0.2), [-0.2, 0.2), [0.2, 0.6), [-0.6, 1); each interval corresponds to a real waveform, and the waveforms include sine, triangle, square wave, positive slope, and negative slope; Modulation frequency: The symbol is s(4), which represents the normalized modulation frequency value, and its value range is [-1,1]. Assume that the physical range of the actual modulation frequency is According to s(4) and physical range The actual modulation frequency is: The unit is MHz; Modulation depth: The symbol is s(5), which indicates the normalized modulation depth value, and its value range is [-1,1]. Assume that the physical range of the actual modulation depth is [a min ,a max ], according to s(5) and the physical range [a min ,a max ], the actual modulation depth is: The unit is %; Frequency offset: The symbol is s(6), which indicates the normalized frequency offset value, and its value range is [-1,1]. Assume that the physical range of the actual frequency offset is According to s(6) and physical range The actual frequency offset is: The unit is MHz; Pulse duty cycle: The symbol is s(7), which represents the normalized pulse duty cycle value, and its value range is [-1,1]. Assume that the physical range of the actual pulse duty cycle is [k min ,k max ], according to s(7) and the physical range [k min ,k max ], the actual pulse duty cycle is: The unit is %; Pulse period: The symbol is s(8), which represents the normalized pulse period value, and its value range is [-1,1]. Assume that the physical range of the real pulse period is [T min ,T ma , according to s(8) and the physical range [T min ,T ma , the actual pulse period is: The unit is ms; The two sensitive status parameters include: Sensitivity, symbolized by s(9), indicates whether the test object is sensitive under the current test waveform, and its value range is [-1,1]. The value range is divided into two intervals: [-1, 0) and [0, 1], which respectively indicate that the test object is sensitive under the current test waveform and the test object is insensitive under the current test waveform; The sensitivity threshold, symbolized by s(10), represents the normalized sensitivity threshold value, and its value range is [-1,1]. Suppose the physical range of the actual sensitivity threshold is [P min ,P max ], according to s(10) and the physical range [P min ,P max ], the true sensitivity threshold is: The unit is dBm.

4. The electromagnetic most sensitive waveform testing method based on reinforcement learning according to claim 1 is characterized in that: The dual-delayed deep deterministic policy gradient network, i.e., the TD3 network, in step S2 includes an online learning network and a target network, each of which contains an actor network and a dual-Q network, wherein the dual-Q network contains two critic networks; Both the actor network and the critic network are fully connected neural networks with two hidden layers, each containing N hidden neurons; the input dimension of the actor network is 10 for input state and the output dimension is 8 for output action, while the input dimension of the critic network is 18 for input state + action and the output dimension is 1 for output Q value; the activation function of the actor network is the hyperbolic tangent function, while the activation function of the critic network is the RELU function.

5. The electromagnetic most sensitive waveform testing method based on reinforcement learning according to claim 1, characterized in that: The step S3 comprises: The output signal of the RF signal source is amplified by a power amplifier. The power-amplified signal is connected to a directional coupler. The through end of the directional coupler is connected to the test product. The signal power is directly injected into the test product. The coupling end of the directional coupler is connected to the signal receiver through an attenuator to measure the actual injected power. The power amplifier is powered by a DC power supply. The test product is fixed by a test bench to improve the repeatability of the experiment. The core of the automatic closed-loop test is a control computer, which obtains the sensitivity status of the test object from the microcontroller and monitors the power at the current test frequency through the signal receiver. Based on the information obtained, it makes intelligent decisions and controls the RF signal source to change the output signal waveform. The sensitivity stimulated by the new waveform is fed back to the control computer in real time to support the next intelligent decision, thus forming a closed loop to search for the most sensitive waveform that minimizes the sensitivity threshold of the test object.

6. The electromagnetic most sensitive waveform testing method based on reinforcement learning according to claim 1 is characterized in that: The step S4 comprises: The reward function is used to guide the agent to achieve the goal. If the test object shows electromagnetic sensitivity, the agent will receive a reward. Considering that most waveforms do not stimulate the electromagnetic sensitivity of the test object, which will lead to the problem of sparse rewards, a reward shaping method is adopted: the total reward is divided into a basic reward r i b and shaping reward r i s to achieve; Basic reward function r i b Will make the agent close to s in the current state min Get a larger reward when s min Indicates the most sensitive waveform state so far; The reward mainly considers the carrier frequency of the most sensitive waveform. min (1), the basic reward function of the i-th step is defined as: Where * indicates non-normalized value, is the sensitivity threshold of the most sensitive waveform so far; The state of the most sensitive waveform is unknown at the initial stage, so the following shaping reward function is designed to guide the agent to find a lower sensitivity threshold: The square term helps the agent search for actions that reduce the maximum sensitivity threshold in one step, thereby accelerating the search for the most sensitive waveform. When the sensitivity threshold of the i-th step is the lowest so far, the agent will be given a huge reward r i s =r i s +100; In addition, there are the following integer reward functions to accelerate convergence: r i s =r i s +1 if s i+1 (9)==1 r i s =r i s -1 if s i+1 (9)==-1 r i s =r i s +5 if s i+1 (9)==1 and s i (9)==-1 r i s =r i s -10 if s i+1 (9)==-1 and s i (9)==1 Finally, the total reward is expressed as: r i =r i b +μr i s Where μ is the reward shaping attenuation factor, which starts from 1 and decays linearly to 0 according to the total step length.

7. The electromagnetic most sensitive waveform testing method based on reinforcement learning according to claim 1 is characterized in that: The step S5 comprises: S501. Start the electromagnetic most sensitive waveform test and synchronously train the constructed reinforcement learning model. The training is divided into N ep Innings, each inning has N s Step 1: The environment needs to be reset at the beginning of each round: the output of the RF signal source is turned off, the state is reset to a random number uniformly distributed between -1 and 1, and the end of the round flag B is set. done Reset to 0; N warm The step is used as a warm-up phase, during which the action is a random number uniformly distributed between -1 and 1, rather than being inferred by the actor network in the TD3 network; in the non-warm-up phase, based on the current state s i Use the actor network in the online network to infer the action; add a mean of 0 and a variance of E to the action n Gaussian noise is used as exploration noise; the current waveform parameter state is based on the obtained action a i To update: s i+1 (1:8)=s i (1:8)+βa i Where β represents the maximum step size of the action, and the updated waveform state s i+1 (1:8) needs to be clipped to the range of [-1,1]; S502. By controlling the RF signal source, the output carrier power is P max The waveform parameter is the state s i+1 (1:8) represents the physical true value. After the signal output stays for 2 seconds, observe the sensitivity of the EUT. If the EUT is not sensitive, set s i+1 (9) = -1 and s i+1 (10) = 1; Otherwise, let s i+1 (9)=1, and in [P min ,P max ] range; each change of carrier power should stay for 2 seconds; use binary search method to change carrier power n times to find the sensitivity threshold with an accuracy of The sensitivity threshold is set by [P min ,P max ] is normalized to [-1,1] and recorded in s i+1 (10) Medium; S503. Calculate reward r according to step S4 i , if the sensitivity threshold is the lowest so far, the current state should be recorded in s min middle; If this round is over, let B done =1, otherwise let B done =0; S504. i 、a i 、r i 、s i+1 and B done Record it in the experience playback buffer and enter the next cycle until each round is finished and the most sensitive waveform state s is obtained min ; During the test, every l rdelay Step 1 rγ Decay learning rate l r , every N En Step γ En Attenuated exploration noise variance E n , in T start After the step, every T train The TD3 network is trained once.

8. The electromagnetic most sensitive waveform testing method based on reinforcement learning according to claim 7 is characterized in that: Every T train The process of training a TD3 network once includes: Get a batch of N from the experience replay buffer batch Samples i 、a i 、r i 、s i+1 , will s i+1 Send it to the actor network of the target network to get a i+1 , will s i+1 and a i+1 The critic network sent to the target network gets Q * 1 and Q * 2. Take Q * 1 and Q * 2 as Q, and the target Q value Q is calculated according to the Bellman formula t =r i +(~B done )γQ, where ~B done Indicates B done Negated, γ represents the discount factor; Will s i and a i Send it to the double Q network of the online network to get Q1 and Q2, and calculate MSE Loss1=Q t -Q1, MSE Loss2 = Q t -Q2, total loss loss = MSE Loss1 + MSE Loss2, using loss as the loss function, and updating the dual-Q network parameters in the online network through the back propagation algorithm; The parameters of the actor network in the online network are updated every T during training. delay The update process is as follows: i Send it to the actor network of the online network and get Will s i and The double Q network is sent to the online network, and only the value of the first critic network is obtained. Its negative value is used as the loss function, and the actor network parameters in the online network are updated through the back-propagation algorithm. The weight parameters of the target network are updated every T during training through a soft update process. delay These parameters are merged with the parameters of the online network in the ratio (1-τ):τ).

Images