PCB mainboard function test strategy dynamic optimization method based on deep reinforcement learning

Abstract

The invention relates to the technical field of mainboard function test, and discloses a PCB mainboard function test strategy dynamic optimization method based on deep reinforcement learning. Comprising the steps of initializing a simulation environment, and obtaining an initial state vector; the actor network of the agent outputs a current action vector based on the state vector; the simulation environment executes the current action vector, updates the test probability of each test item, and simulates the test process; calculating a state vector at the next moment, and calculating a current reward value; storing the experience tuple in an experience playback buffer area; sampling an experience tuple from the experience playback buffer area, and updating parameters of an actor network and a comment network of the intelligent agent by using a near-end strategy optimization algorithm; the above process is repeated, and the test strategy is converged to a multi-target equilibrium solution under the maximum cumulative reward target. According to the method, the problem that an existing strategy cannot respond to the dynamic change of the production line in real time is solved, and multi-target collaborative optimization of the test cost, the defect omission ratio and the production line beat time is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of PCB mainboard function test, and particularly relates to a PCB mainboard function test strategy dynamic optimization method based on deep reinforcement learning. BACKGROUND

[0002] In the field of PCB mainboard production, mainboard function test is a key link at the end of the production line, which refers to the full functional test of the mainboard in the electronic product manufacturing process to ensure the normal operation of each function and component of the mainboard. If a defective mainboard is not detected and flows into the downstream whole machine assembly link, high repair cost (including whole machine disassembly, logistics storage, order delay loss) will be generated. The mainstream test strategy in the industry has significant technical defects, as follows: The existing technical scheme based on static test strategy adopts fixed test items and test probability, which cannot adapt to the dynamic change of defect rate in the production process, resulting in low test efficiency or increased quality risk. The technical scheme based on rule dynamic adjustment adjusts the test strategy through preset rules (such as defect rate threshold), but the response is lagging and cannot achieve global optimization of multiple targets (cost, quality, efficiency). The technical scheme based on mathematical optimization method establishes an optimization model based on historical data, but assumes that the parameters are fixed and cannot adapt to the real-time changing production environment.

[0003] In summary, the existing technologies cannot solve the core problem of "poor dynamic environment adaptability and insufficient cost-quality-efficiency multi-objective balance capability in PCB mainboard function test", and the root cause lies in the lack of adaptive learning mechanism that can respond to the changes of production line state in real time, which is difficult to match the dynamic and complex needs of industrial scenarios. SUMMARY

[0004] To solve the above technical problems, the present application provides a PCB mainboard function test strategy dynamic optimization method based on deep reinforcement learning.

[0005] To solve the above technical problems, the present application adopts the following technical scheme: A PCB mainboard function test strategy dynamic optimization method based on deep reinforcement learning, comprising the following steps: S1, initializing a simulation environment and obtaining an initial state vector; the initial state vector includes test probability and defect rate of each test item, and production line tact time constraint; S2, constructing an intelligent agent using a proximal policy optimization algorithm, including an actor network and a critic network; the actor network of the intelligent agent outputs a current action vector based on the state vector, and the action vector represents the adjustment amount of the test probability of each test item; S3, the simulation environment executes the current action vector, updates the test probability of each test item, and simulates the test process; calculates the state vector at the next moment, and calculates the current reward value based on the average effective cost, the missed detection rate, and the test time timeout condition; S4, the current state vector, action vector, reward value, and state vector at the next moment are stored as experience tuples in the experience replay buffer; S5, experience tuples are sampled from the experience replay buffer, and the parameters of the actor network and the critic network of the agent are updated using the proximal policy optimization algorithm; S6: repeat steps S2 to S5 to converge the test strategy to a multi-objective balanced solution under the goal of maximizing cumulative rewards.

[0006] In one of the embodiments, the state vector includes: a test strategy sub-vector: including the current test probability of M test items, reflecting the current test strategy of the agent; a defect rate sub-vector: including the defect rate of M test items; a macroscopic sub-monitoring vector: including the current average test time and the maximum allowed test time of the production line.

[0007] In one of the embodiments, the calculation of the state vector at the next moment includes: the test probability of the i-th test item in the state vector at the next moment ; the function is used to limit the input value in the range of [0, 1], is the current test probability of the i-th test item, is the adjustment amount of the test probability of the i-th test item in the action vector; the defect rate of the i-th test item in the state vector at the next moment ; is a smoothing factor, is the current defect rate of the i-th test item, is the number of defects detected, represents the actual execution times of the i-th test item in the current batch; the calculation method is "test batch mainboard total number x new test probability of the i-th test item".

[0008] the average test time in the state vector at the next moment is , is the time consumption of the i-th test item; the maximum allowed test time of the production line is adjusted according to the production line queue load .

[0009] In one embodiment, the current reward value is calculated based on the average effective cost, the missed detection rate and the test time timeout condition, and specifically includes: reward value including a main reward term, a constraint penalty term and a quality term: ; main reward term negative reward based on the average effective cost, guiding the test strategy to reduce the cost: ; wherein, the average effective cost of the current test strategy, the state space where the current state vector is located, the historical maximum cost of the initial test strategy; constraint penalty term : ; wherein, the penalty coefficient; the test probability of the i-th test item, the time consumption of the i-th test item, the maximum allowed test time of the production line; quality reward term : ; the quality weight, the industry missed detection rate threshold, the actual missed detection rate.

[0010] In one embodiment, the average effective cost is calculated as follows: ; wherein, the total number of test items, the test probability of the i-th test item, , the unit cost of the i-th test item, the difference between the finished product repair cost and the mainboard repair cost, the defect rate of the i-th test item.

[0011] In one embodiment, the actor network adopts a fully connected neural network, the input is a state vector, passes through three hidden layers, and finally outputs the mean and standard deviation of the Gaussian distribution of each action dimension; The critic network adopts a fully connected neural network, the input is a state vector, passes through three hidden layers, and finally outputs , represents the state value of evaluating the network output, that is, the cumulative reward expectation that the agent can obtain in the future by following the current test strategy to continuously interact with the environment in the current state.

[0012] In one of the embodiments, experience tuples are sampled from the experience replay buffer, and the parameters of the actor network and the critic network of the agent are updated using the proximal policy optimization algorithm, specifically including: A plurality of experience tuples are extracted from the experience replay buffer, the critic network estimates the value of each state vector in the experience tuple, and the value of the current state vector and the value of the state vector at the next time are obtained; the current advantage function is calculated , is a discount factor, is a current reward value; The policy loss of the actor network is: ; is a clipping coefficient, represents the action probability distribution density output by the actor network with the updated parameters , represents the action probability distribution density output by the actor network with the updated parameters , represents the parameters of the updated actor network, including all trainable weight matrices and bias vectors in the actor network, is the parameter of the actor network before updating: by minimizing the weight matrix and the bias vector of the actor network are updated; the function is used to limit the input value in the range of [0, 1]; The value function loss of the critic network ; by minimizing , the weight matrix and the bias vector of the critic network are updated; represents the target value, ; represents the number of reward accumulation steps, which refers to the maximum number of reward accumulation steps from the current time t when the target value is calculated.

[0013] In one of the embodiments, the critic network estimates the value of each state vector in the experience tuple, specifically including: The value of the current state vector is: ; wherein, is the hidden layer output of the critic network, is the weight matrix.

[0014] Compared with the prior art, the beneficial technical effects of the present application are: The present application solves the problem that the existing strategy cannot respond to real-time changes in the production line, including sudden increases in defect rates (such as a test item defect rate rising from 0.3% to 0.6%) and time constraint adjustments (such as the production line cycle time tightening from 200 seconds to 150 seconds), and realizes automatic and rapid iteration of the strategy.

[0015] The present application realizes multi-objective collaborative optimization of test cost, defect omission rate, and production line cycle time, ensures that the average total test cost is more than 15% lower than that of the existing strategy, the omission rate is stably lower than the 0.5% industry threshold, and the average test time meets the dynamic adjustment of the production line cycle constraint (the overtime rate is lower than 2%). BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a schematic diagram of the overall framework of the present application; Figure 2 is a flowchart of the method of the present application. DETAILED DESCRIPTION

[0017] A preferred embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0018] The core of the present application is a sequential closed-loop system of "observation-decision-execution-feedback", and the overall technical framework is as shown in Figure 1

[0019] The present application includes two core modules: An agent using a proximal policy optimization algorithm (PPO agent): plays the role of a production line decision maker, dynamically outputs test strategy adjustment actions based on environmental state through an actor network (strategy generation) and a critic network (value evaluation); A high-fidelity simulation environment: abstracts a PCB functional test production line, executes agent decisions, and calculates and feeds back new states and reward signals.

[0020] The specific process of the present application includes: State observation: the simulation environment is initialized, and the initial state s (including test probability, defect rate, time constraint, etc.) is output to the agent; Strategy decision: the actor network outputs action a (test probability adjustment amount) based on state s; Action execution: the simulation environment executes action a to simulate the test process; Reward feedback: the simulation environment calculates the new state s' and the reward R (fusing cost, quality, and efficiency indicators).​ Experience storage: Store experience tuple (s, A, R, s') into experience replay buffer; Learning update: After experience replay buffer data is qualified, proximal policy optimization (PPO) algorithm evaluates value through critic network, updates actor network parameters, and iteratively optimizes policy.

[0021] 1. Mathematical modeling of the problem.

[0022] (1) Core optimization goal: The method takes the average effective cost as the minimum core goal, and the formula is as follows ; Wherein, is the total number of test items, is the test probability of the i-th test item, , is the unit cost of the i-th test item, is the difference between the finished product repair cost and the mainboard repair cost, is the defect rate of the i-th test item. In this embodiment, there are 30 test items in total.

[0023] (2) Constraint condition: The line cycle time constraint needs to be met: Wherein is the time consumption of the i-th test item, is the dynamically adjusted maximum allowed test time of the production line.

[0024] 2. System modeling four-tuple .

[0025] (1) State space (S) State vector is defined as a 62-dimensional normalized vector: .

[0026] Test strategy sub-vector (30-dimensional): , corresponding to the current test probability of 30 test items, reflecting the current test strategy of the agent.

[0027] Defect rate sub-vector (30-dimensional): , updated in real time through exponential weighted moving average (EWMA), reflecting the defect fluctuation of the production line.

[0028] Macro monitoring sub-vector (2-dimensional): is the current average test time, is the current line cycle constraint, reflecting the efficiency target state.

[0029] (2) Action Space A: The set of all possible action vectors a, which is defined as a 30-dimensional continuous vector: where is the adjustment amount of the test probability of the i-th test item, and the constraint is that the single adjustment amplitude does not exceed 5%, to avoid policy shocks.

[0030] New test probability Calculation: ; The clip function ensures that the effective probability is within the interval [0, 1].

[0031] (3) State Transition (P): State transition function Encapsulates the state update logic after testing: ; Test probability update function : Update the 30-dimensional test probability according to .

[0032] Defect rate update function : Based on the Exponential Weighted Moving Average (EWMA) update of real-time test data: .

[0033] Time calculation function : Recalculate the average test time according to .

[0034] Constraint adjustment function : Dynamically adjust the production line queue load (fluctuation range ± 30%).

[0035] (4) Reward Function (R): Reward function Adopt a three-layer composite reward function to quantify cost, quality, and efficiency goals: ; Main reward item , based on the negative reward of the average effective cost, to guide the strategy to reduce costs: ; where is the average effective cost of the current test strategy, is the historical maximum cost of the initial test strategy (to avoid reward value fluctuations).

[0036] Constraint penalty item , penalize timeout actions, ensure to meet the beat constraints: ; In this embodiment, (1 yuan loss per second of timeout, matching the loss weight in industrial scenarios).

[0037] Quality reward term , reward low false negative rate strategy: ; where the quality weight , industry false negative rate threshold , actual false negative rate .

[0038] 3. Actor-Critic network structure and loss function.

[0039] (1) Actor network (policy generator): Structure: Use a fully connected neural network, input is a 62-dimensional state vector, pass through three hidden layers (256→128→64 neurons, ReLU activation function), finally output two 30-dimensional vectors, representing the mean (Tanh activation) and standard deviation (Softplus activation) of the 30-dimensional action Gaussian distribution.

[0040] The optimization goal is to minimize the clipping policy loss : ; where is the action probability density (Gaussian distribution) of the current test policy, the advantage function is estimated by the Critic network, the clipping coefficient controls the update step size.

[0041] (2) Critic network (value network) Structure: Use a fully connected neural network similar to the Actor network, input 62-dimensional state, pass through three hidden layers (256→128→32 neurons, ReLU activation function), finally output 1-dimensional V(s).

[0042] Optimization goal: Minimize mean square error loss : ; where is the state value estimate of the Critic network, the target value , the discount factor , the value range of [0, 1], in the application, it represents the "influence weight of future reward on current value", the value closer to 1 means more emphasis on long-term reward, and the policy avoids the increase of missed detection rate or cost surge caused by short-sightedness; in the application, ; represents the reward accumulation step, which refers to the maximum number of rewards accumulated from the current time t when calculating the target value .

[0043] Embodiments The method steps of the application are as shown in Figure 2 , specifically including: S1, state initialization: the simulation environment initializes the state vector according to the production batch , including initial , , and state quantities.

[0044] S2, policy decision: the Actor network receives the state vector , and outputs the action vector .

[0045] S3, action execution and feedback: the environment executes the test probability update , obtains and calculates the reward .

[0046] S4, experience storage: store into the experience replay buffer.

[0047] S5, learning update: after the experience replay buffer data is sufficient, the proximal policy optimization (PPO) algorithm is used to update the Actor network and the Critic network for multiple rounds of mini-batch, the minimum clipping policy loss and the minimum mean square error loss are used, and the advantage function is calculated combined with GAE to limit the update amplitude .

[0048] S6, loop iteration convergence: repeat steps S2 to S5, and the test policy converges to a multi-objective balanced solution under the maximum cumulative reward target.

[0049] Data source: based on the historical data of 30 production lines of L generation factories of industry leading enterprises, including of 30 test projects .

[0050] Simulation environment parameters: simulate 1000 mainboard tests for each training episode, and the initial time constraint Defect rate fluctuation ±20%, smoothing factor .

[0051] Proximal Policy Optimization (PPO) Algorithm Parameters: Discount Factor shear coefficient The learning rate is 3e-4 (Adam optimizer), the experience replay buffer capacity is 1024, and the batch update steps are 256.

[0052] Hardware / Software Environment: The software is Python 3.9 (SimPy simulation library, PyTorch 1.12 deep learning framework), and the hardware is CPU i7-12700K and GPU RTX 3090.

[0053] In one embodiment, step S1 specifically includes: Simulation environment parameter loading: Historical datasets from 30 production lines of leading PCB foundries in the industry were read, and basic parameters for 30 test items were extracted, including test time. (Range 0.5~60 seconds, such as memory compatibility testing) =10 seconds, USB interface test =8 seconds), initial defect rate (Range 0.02%~0.45%, such as camera calibration test) ).

[0054] Configure simulation environment dynamic parameters: Set each training episode to simulate the testing process of 1000 motherboards, and the maximum allowable testing time for the initial production line. (Dynamically adjustable by production line queue load ±30%), defect rate fluctuation range ±20% (randomly generated according to normal distribution), Exponentially weighted moving average (EWMA) smoothing factor. .

[0055] Initial state vector construction: generating 30-dimensional test policy sub-vectors Regarding the defect rate Required test items ,right Selected test items Generate a 30-dimensional dynamic defect rate sub-vector. Initialize values ​​based on historical data; generate a 2D macro-monitoring sub-vector; calculate the initial average test time. Determine the initial cycle constraints. The 62-dimensional vector above is normalized (mapping each dimension value to the [0,1] interval, such as the test probability). Direct retention, defect rate Divide by 1% (and the time value by 300 seconds) to obtain the initial state vector. .

[0056] State output to PPO agent: normalized initial state vector Transmitted to the Actor network and Critic network of the PPO agent through the data interface as the input basis for policy decision-making.

[0057] In one embodiment, step S2 specifically includes: State feature extraction: the Actor network inputs a 62-dimensional state vector , and calculates through a first layer of hidden layers (256 neurons, ReLU activation function): wherein, is a 62x256-dimensional weight matrix, and b1 is a 256-dimensional bias vector, to realize basic state feature encoding; calculates through a second layer of hidden layers (128 neurons, ReLU activation function): , is a 256x128-dimensional weight matrix, is a 128-dimensional bias vector, to complete feature dimension reduction and abstraction; calculates through a third layer of hidden layers (64 neurons, ReLU activation function): , is a 128x64-dimensional weight matrix, is a 64-dimensional bias vector, to focus on the core decision-making features of action generation.

[0058] Action distribution parameter output: the mean output layer calculates: , is a 64x30-dimensional weight matrix, is a 30-dimensional bias vector, to output a 30-dimensional action mean (range [-1, 1]).

[0059] The standard deviation output layer calculates: , is a 64x30-dimensional weight matrix, is a 30-dimensional bias vector, to output a 30-dimensional action standard deviation (ensure > 0).

[0060] Continuous action sampling and constraint: based on a Gaussian distribution to generate a 30-dimensional original action ; Boundary clipping is performed on the original action: to ensure that the probability adjustment amplitude of each test item does not exceed ±5%; to output a final action vector .

[0061] In one embodiment, step S3 specifically includes: New test strategy calculation: according to action vector Update test probability of each test item: Such as memory test Generate test execution plan for the current batch of 1000 mainboards: for the i-th test of each mainboard, generate a random number rand in the interval [0, 1], if then execute the test, otherwise skip.

[0062] Test process simulation: simulate mainboard testing block by block: for the executed test item, record the test time consumption (cumulative to the total test time of the mainboard), and according to the current defect rate Randomly generate defect results (such as memory test = 0.45%, about 4 out of 1000 mainboards are detected with defects); batch test data statistics: record the actual execution times, defect detection quantities (such as USB test execution 600 times, 3 defects are detected, then ), total test time of each mainboard, and average test time of the batch.

[0063] In one embodiment, step S4 specifically includes: (1) New state Calculation: Test probability update: compose a new 30-dimensional test strategy sub-vector ; Defect rate dynamic update (EWMA): ; Macroscopic monitoring vector update: calculate new average test time , adjust the cycle constraint according to the production line queue load ; Normalize the new state vector to get .

[0064] (2) Reward value R calculation: main reward item , where is the average effective cost of the current test strategy, is the historical maximum cost of the initial strategy (to avoid reward value fluctuation); constraint penalty item is calculated according to the formula , where (every 1 second overtime is equivalent to 10 yuan loss, matching the loss weight of industrial scene); quality reward item is calculated according to the formula , where the quality weight , and the industry missed detection rate threshold , actual false detection rate ; total reward .

[0065] In one of the embodiments, step S5 specifically comprises: Experience tuple construction: encapsulate the initial state vector , action vector , reward , new state vector into a complete experience tuple .

[0066] Experience replay buffer storage and management: store the experience tuples in an experience replay buffer with a capacity of 1024, using the "first-in, first-out" rule, delete the oldest experience tuples when the experience replay buffer is full; record the current storage quantity of the experience replay buffer, when the quantity reaches a preset threshold (such as 256, i.e. the minimum batch learning sample number), trigger the learning and updating steps of the subsequent PPO agent; randomly shuffle the experience tuples in the experience replay buffer to avoid training bias caused by sample sequence correlation.

[0067] Advantage function (GAE) calculation: extract 256 experience tuples from the experience replay buffer; the Critic network estimates the value of each state: , where is the output of the Critic network hidden layer, is a 32x1-dimensional weight matrix, and is obtained; calculate the advantage function according to the formula . .

[0068] Actor network policy loss calculation and parameter update: calculate the action probability density ratio of the current test policy and the old policy: , where is the output of the Actor network before updating; calculate the policy loss according to the clipping loss formula: . Use the Adam optimizer (learning rate 3e-4) to minimize , update the weight matrix and bias vector of the Actor network.

[0069] Critic network value function loss calculation and parameter update: calculate the target value: ; calculate the value function loss according to the mean square error formula: ; use the Adam optimizer to minimize , update the weight matrix and bias vector of the Critic network.

[0070] Repeat steps S2 to S5 until the number of training steps reaches 1e6 steps or the total reward stabilizes in the range of -0.2±0.02 (convergence threshold). Stop updating parameters and output the current optimal test strategy.

[0071] The advantages of this invention are verified through experimental data, based on historical data from 30 production lines of leading companies in the industry. The experimental results are shown in Table 1, and the specific analysis is as follows.

[0072] (1) Significant cost optimization effect: The average total testing cost of this invention is 12.3 yuan / piece, a 26.4% reduction compared to Factory L's existing strategy (16.7 yuan / piece) and a 15.2% reduction compared to the mathematical optimization strategy (14.5 yuan / piece). The cost reduction stems from: reducing the cost of high-cost, low-defect-rate test items (such as camera calibration testing). Yuan, The test probability was reduced from 20% to 8%, reducing the cost of ineffective testing; for high-defect-rate test items (such as memory compatibility testing), The testing probability dynamically increases from 80% to 92%, reducing the cost of missed detections and rework.

[0073] (2) Quality assurance capabilities meet standards and are stable: The false negative rate of this invention is 0.42%, meeting the industry threshold requirement of 0.5% for the electronics manufacturing industry, a reduction of 38.2% compared to factory L's existing strategy (0.68%); simultaneously, the false negative rate fluctuation is only ±5%, far lower than the mathematical optimization strategy (±40%). The reason for the stable quality is: in the reward function... The real-time guidance strategy keeps the missed detection rate below the threshold; the dynamic defect rate update (EWMA) of the MDP model ensures that the strategy adapts to the defect fluctuations of the production line and avoids quality loss of control due to fixed parameters.

[0074] (3) High efficiency adaptability: The average test time of this invention is 185 seconds, a 22.9% reduction compared to the full test strategy (240 seconds); the timeout rate is only 1.3%, significantly better than the existing strategy (10.7%) and the mathematical optimization strategy (5.3%) in factory L. This efficiency advantage stems from: The quantitative timeout penalty guides the strategy to proactively compress high-time-consuming and low-value test items (such as reducing the probability of a 60-second hard drive read / write test from 50% to 30%); the continuous action space design enables fine-grained control of test time, avoiding timeouts caused by overshooting of strategy adjustments.

[0075] (4) Excellent dynamic adaptability: In scenarios with dynamic fluctuations in the production line, the response speed of this invention far exceeds that of existing strategies: in scenarios with a sudden increase in defect rate (such as USB interface testing)... From 0.3% to 0.6%): The strategy adjustment is completed within 10 minutes (the test probability is increased from 60% to 95%), and the missed detection rate is reduced to 0.48%; the existing strategy of factory L needs manual intervention for 2 hours, during which the missed detection rate is increased to 1.2%; the time constraint is tightened (for example From 200 seconds to 150 seconds): The average test time is stabilized to 148 seconds within 5 test batches; the existing strategy of factory L needs 15 batches, during which the timeout rate is 25%.

[0076] Table 1

[0077] By using the method of the present application, the average test total cost is 12.3 yuan / block, the missed detection rate is 0.42%, the average test time is 185 seconds, and the timeout rate is 1.3%, which meets the design target.

[0078] The terms used herein are merely used to describe specific embodiments, and are not intended to limit the present application. The terms "include", "contain" and the like used herein indicate the existence of the described features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0079] It should be understood that although each step in the flowchart of the drawings is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the drawings can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0080] The technical features of the above embodiments can be combined in any way. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0081] It will be obvious to a person skilled in the art that the application is not limited to the details of the foregoing exemplary embodiments and can be implemented in other concrete forms without departing from the spirit or essential characteristics of the application. The embodiments are considered in all respects to be illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and therefore all changes coming within the meaning and equivalency range of the claims are intended to be embraced therein and no appended claim is to be considered as limiting as to the subject matter recited in that claim.

[0082] Furthermore, it should be understood that although the description is made on the basis of embodiments, not every embodiment contains only one independent technical solution, and the description is made in this way only for the sake of clarity, and a person skilled in the art should consider the description as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by a person skilled in the art.

Claims

1. A method for dynamically optimizing a PCB mainboard function test strategy based on deep reinforcement learning, characterized in that, The method comprises the following steps: S1, initializing a simulation environment, and obtaining an initial state vector; the initial state vector comprises test probabilities and defect rates of each test item, and a production line beat time constraint; S2, constructing an agent using a proximal policy optimization algorithm, including an actor network and a critic network; the actor network of the agent outputs a current action vector based on the state vector, and the action vector represents an adjustment amount of the test probability of each test item; S3, the simulation environment executes the current action vector, updates the test probability of each test item, and simulates a test process; calculating a state vector at a next time, and calculating a current reward value based on an average effective cost, a missed detection rate, and a test time timeout condition; S4, storing the current state vector, the action vector, the reward value, and the state vector at the next time as an experience tuple in an experience replay buffer; S5, sampling experience tuples from the experience replay buffer, and updating parameters of the actor network and the critic network of the agent using the proximal policy optimization algorithm; S6: repeating steps S2 to S5 to converge the test strategy to a multi-objective balanced solution under the maximization of the cumulative reward target.

2. The method of claim 1, wherein the method further comprises: The state vector comprises: a test strategy sub-vector: comprising current test probabilities of M test items, reflecting the current test strategy of the agent; a defect rate sub-vector: comprising defect rates of the M test items; a macroscopic sub-monitoring vector: comprising a current average test time and a maximum allowed test time of the production line.

3. The method of claim 2, wherein the method further comprises: The calculation of the state vector at the next time comprises: the test probability of the i-th test item in the state vector of the next time ; function for limiting the input value to the range [0, 1], is the current test probability of the i-th test item, is the adjustment amount of the test probability of the i-th test item in the action vector; the defect rate of the ith test item in the state vector of the next time ; is a smoothing factor, the current defect rate of the ith test item, is the number of defect detections, denotes the actual number of executions of the ith test item in the current batch; The average test time in the state vector for the next time instant is , is the time spent for the i-th test item; the maximum allowed test time on the line is adjusted according to the line queue load .

4. The method of claim 1, wherein the method further comprises: The calculation of the current reward value based on the average effective cost, the missed detection rate, and the test time timeout condition comprises: reward value comprising a main reward term, a constraint penalty term and a quality term: ； main reward item negative reward based on average effective cost, guiding the testing policy to reduce cost: ； wherein, is the average effective cost of the current test strategy, is the state space in which the current state vector resides, is the historical maximum cost of the initial test strategy; Constraint penalty term : ； wherein, is a penalty coefficient; is a test probability of the i-th test item, is a time consumption of the i-th test item, is a maximum allowed test time of the production line; Quality reward item : ； is a quality weight, is an industry miss rate threshold, is an actual miss rate.

5. The method of claim 1, wherein the method further comprises: The average effective cost is calculated in the following manner: ； wherein, is the total number of test items, is the test probability of the ith test item, , is the unit cost of the ith test item, is the difference between the cost of repairing the finished product and the cost of repairing the mainboard, is the defect rate of the ith test item.

6. The method of claim 1, wherein the method further comprises: The actor network adopts a fully connected neural network, and the input is the state vector, which passes through three hidden layers, and finally outputs the mean and standard deviation of the Gaussian distribution of each action dimension; The comment network adopts a fully connected neural network, the input is a state vector, passes through three hidden layers, and finally outputs , The state value output by the evaluation network represents the expected cumulative reward that the agent can obtain in the future by following the current test strategy to continuously interact with the environment in the current state.

7. The method of claim 1, wherein the method further comprises: determining a test strategy for the PCB based on the test strategy optimization model. sampling experience tuples from the experience replay buffer, and updating parameters of the actor network and the critic network of the agent using the proximal policy optimization algorithm, specifically comprising: The critic network estimates the value of each state vector in the experience tuples to obtain a current state vector value and a value of a next time state vector ; a current advantage function , is a discount factor, is a current reward value; Strategy loss of actor network For: ； is the shear coefficient, denotes the action probability distribution density output by the actor network with the updated parameters denotes the action probability distribution density output by the actor network with the updated parameters denotes the parameters of the updated actor network, is the parameters of the actor network before the update: by minimizing updates the weight matrix and the bias vector of the actor network; the function is used to limit the input value in the range [0, 1];​​ critic network's value function loss by minimizing updating the critic network's weight matrix and bias vector; denotes the target value, ; denotes the reward accumulation step number.

8. The method of claim 1, wherein the method further comprises: determining a test strategy for the PCB based on the test strategy optimization model. The critic network estimates the value of each state vector in the experience tuple, specifically comprising: value of the current state vector is: ； wherein, is the output of the hidden layer of the critic network, is a weight matrix.

Images