Vehicle domain controller EOL test program generation method based on reinforcement learning

Abstract

The invention belongs to the field of automatic generation of EOL offline test programs, and particularly relates to a reinforcement learning-based vehicle domain controller EOL test program generation method. Comprising the following steps: S1, acquiring component original data from a schematic diagram, performing data processing on an original circuit and component connection attributes, and combining the original circuit and the component connection attributes into minimum circuit combination data; s2, constructing an optimal EOL offline test case selection model on the two-dimensional vector data set obtained in the step S1; s3, generating an EOL test program with the highest priority at present by utilizing a Q-Learning algorithm on the basis of the model in S2; and S4, selecting an optimal test case according to the evaluation standard. According to the method, the defects of low detection failure rate and difficulty in convergence of a test result reward application method are overcome; and the test feedback time is long.

Description

Technical Field

[0001] The present invention belongs to the field of automatic generation of EOL offline test programs, and in particular relates to a method for generating an EOL test program for a vehicle domain controller based on reinforcement learning. Technical Background

[0002] At present, all products produced by automobile controller manufacturers need to undergo EOL (end of line) offline testing, but the test equipment and test methods used are different: one way is to design an offline test system control unit based on an embedded platform. A modular approach is used to design the control unit hardware, and then the corresponding underlying driver is designed for each hardware module, and the application layer algorithm is designed to realize the automatic execution of the system workflow; then a wheel speed simulation model is built based on the MATLAB / Simulink platform, and it is coded and transplanted into the embedded program. Another way is to use LabVIEW tools to develop the host computer program and design a simple and intuitive software user interface to realize automated testing. Another way is to develop an offline test system based on virtual instruments to realize functions such as automatic connection of ECU, power supply management, automatic program burning, one-button testing, and intelligent database management.

[0003] The above methods can achieve different degrees of automated EOL testing, but these research directions are mainly focused on the automated execution of the test system during EOL offline testing, and do not involve the issue of improving the efficiency of EOL test program development in the early stages of project development. The development of test programs for each different controller still relies mainly on manual work by engineering and technical personnel. It is unable to solve the tediousness and long cycle of test program development in the early stages of project development.

[0004] Reinforcement learning is a machine learning method that learns how to map from state to behavior in order to maximize the reward. In other words, such an intelligent agent needs to continuously try in the environment and continuously optimize the state-behavior correspondence through the feedback (reward) given by the environment.

[0005] At present, the research on generating test programs based on reinforcement learning in the field of software testing can be roughly divided into two categories: one is to prioritize the test cases that directly fail, that is, to apply rewards to the results. Since test case failures rarely occur, this method has the problems of low failure detection efficiency and difficulty in convergence. The other is to focus on the historical information of test case execution, that is, to apply rewards to the impact of failures. Since the amount of historical information data is huge, more computing resources are required, so it has the disadvantage of long test feedback time.

[0006] At present, the EOL test of vehicle controller production line is based on the test program manually written by engineers and technicians, but in fact, the hardware circuits of the engine controller, transmission controller, body controller, central computing controller and domain controller on the vehicle are very similar, and the hardware offline test methods are also very similar. In the research and development stage of vehicle domain controller, the project cycle is short and the development task is urgent. The development of vehicle domain controller hardware offline test cases brings a lot of workload to the R&D personnel. Therefore, it is very necessary to build an intelligent test system to automatically generate test cases for automotive electronic domain controllers.

[0007] At present, there is no intelligent test program generation method developed specifically for vehicle domain controller EOL end-of-line testing. Therefore, in the field of automotive controller EOL end-of-line testing, an intelligent system is constructed based on the reinforcement learning method to automatically generate automotive controller EOL end-of-line test programs to replace the manual repetitive labor of engineers manually writing a large number of test programs. This can not only improve the efficiency of test program generation, but also improve the functional test coverage, thereby improving the production efficiency of actual industrial applications. Summary of the invention

[0008] In view of the problems existing in the prior art, the purpose of the present invention is to provide a vehicle domain controller EOL test program generation method based on reinforcement learning, so as to provide a feasible method for solving the problems of low efficiency and high error rate of manually developed EOL offline test programs.

[0009] Technical solution of the present invention:

[0010] A method for generating a vehicle domain controller EOL test program based on reinforcement learning comprises the following steps:

[0011] S1: Obtain original component data according to the component list of a front area controller of a certain car, and obtain component connection attribute data according to the connection mode parameters between various components in the schematic diagram, process the original circuit and component connection attributes, and combine them into minimum circuit combination data;

[0012] S2: Based on the minimum circuit combination data generated in step S1, an optimal EOL test case selection model is constructed;

[0013] S3: Based on the S2 model, the Q-table type agent is constructed using the Q-Learning algorithm. It is initialized using the initialization update algorithm according to the minimum circuit combination data, and the Q-table is updated according to the model update algorithm.

[0014] S4: Execute test cases on the circuit model to be tested, generate high-priority test cases based on the reward function and state transition strategy, and select the best test cases based on the proposed evaluation criteria.

[0015] Furthermore, step S1 is specifically as follows:

[0016] First, obtain the original data: The original data of this study is the electronic component information set X{COM x}, where COM x Expressed as (net i ,com z ,net j ), where com z Refers to the name of the component, net i Refers to the network name (net name) on the left side of the component. j Refers to the net name on the right side of the component.

[0017] Given a set of original data (M002, C001, G001), (P001, R001, M002), (M002, R002, N001), ... Among them, the "M" in the parameter "M001" means that the NET end will be connected to a NET end of another component, and "001" refers to the same type of data number of the NET end in the entire data set. One point to note is that the two NET ends connected together have the same number. The second parameter refers to the name and type of the component itself, among which the "R" in the parameter "R001" refers to a resistor-type component, and "001" refers to the same type of data number of this resistor-type component in the entire data set. The "C" in "C001" refers to a capacitor-type component, and "001" refers to the same type of data number of this capacitor-type component in the entire data set. Each parameter represents a component type and number in detail.

[0018] Then comes data extraction: Based on the original data, COMs with the same net name are grouped into a set A{COM1, COM2, COM3, ...COM w}, where w is the number of COMs with the same net name.

[0019] For example, given a set of original data {(M002,C001,G001),(P001,R001,M002),(M002,R002,N001)}

[0020] Data elimination and vector transposition: COM in A is eliminated according to the data with the same net, and then vector transposition is performed to generate a new vector set B.

[0021] For the above example, we can get (M002, C001, G001) T , (P001,R001,M002), (M002,R002,N001)};

[0022] Then comes data combination and offset: After data extraction and data elimination, B is processed through data combination to generate C, that is, {(P001,0,0) T , (R001,0,0) T ,(M002,C001,G003) T (R002,0,0) T , (N001,0,0) T};

[0023] Then comes the grouping process: After the above series of data processing, the original data set X will be divided into many sub-data sets {C q}, these sub-datasets C q Respectively represent a minimum circuit model data.

[0024] Finally, the experimental data set: sub-dataset {C q} constitutes the experimental data set of this study. This sub-dataset is a two-dimensional test vector set.

[0025] Further, step S2 is specifically as follows:

[0026] Use the reinforcement learning model to build the optimal EOL offline test case selection model. The state, action, agent, policy, and reward function in reinforcement learning are defined as follows:

[0027] State: The vector group {C q} is n*m dimensional, where C q The non-zero parameters (types of each interface of the minimum circuit) in the 1st row, 1st column and the nth row, mth column are defined as the State in the reinforcement learning model, and all States constitute the state space.

[0028] 动作(Action) ： 将从状态 s t (当前接口类型)跳转到 s t+1 (下一个接口类型)的抽象的动作定 It means Action in reinforcement learning model. All actions constitute the action space.

[0029] Agent: The two-dimensional test vector set generated in step S1

[0030] Policy: Reinforcement learning ε-greedy policy.

[0031] Reward function: The formula is as follows:

[0032]

[0033] Where y is the total number of historical executions of test case t, r i is the test result of the i-th execution of test case t, represented by 1 or 0, where 1 indicates test failure and 0 indicates test pass. j represents the j-th test cycle, T is the test case set of the current cycle, TS j is a prioritized set of test cases. It is the number of test cases whose execution failed in the current test cycle.

[0034] Furthermore, based on the Q-Learning algorithm model, a Q-table type intelligent agent is constructed. The agent is an n*m dimensional test vector. The initial data constructed in step S1 is initialized through the initialization update algorithm, and then the Q-table is updated according to the model update algorithm to continuously generate the optimal EOL offline test cases.

[0035] Regarding the Q-table initialization part, in some work, the Q-table is often initialized to all 0 values. This method easily causes the Q-table to jump out of order in the early stage of iteration, affecting convergence. The present invention provides an algorithm for initializing the Q-table based on experimental data. The details are as follows:

[0036] First, obtain a set of original experimental data C (a1, a2, ... a n ), and then traverse the adjacent vector data in the original data (a u ,a v ), determine the adjacent data (a u ,a v ) is in the state-action space (s,a), if (a u ,a v ) in the state-action space (s,a), then (a u ,a v ) will increase by 1; if (a u ,a v ) is not in the state-action space (s,a), then (a u ,a v) Add it to the state-action space (s, a) and set the Q value to 1. Then set (a u , a v ) to be traversed one by one. If (a u , a v ) traversal ends, then the Q-table initialization is completed. Otherwise, continue to traverse the vector data (a u , a v ).

[0037] Furthermore, step S3 is specifically as follows:

[0038] After the agent generates a test program, the environment executes the test program, collects the test results, calculates the reward value according to the reward function, and provides it to the agent. The agent updates according to the reward value. After updating the agent, the agent generates the next round of test programs, and iterates until the termination condition is met.

[0039] Among them, the execution of the test program is completed through software, such as the CFI software of Bosch.

[0040] The specific process of agent update is as follows:

[0041] The model update algorithm used is the Monte Carlo update algorithm: Q(s t , a t ) ← Q(s t , a t ) + θ(G t - Q(s t , a t ))

[0042] Among them, G t is the reward value of state s t in this round. The pseudocode of the Monte Carlo update algorithm is as follows:

[0043] Step1: Given the step size parameter θ, the initial ε value, and the maximum number of test case states L;

[0044] Initialize Q(s, a); the maximum value of the test case state is maxL; the counter for the length of a single test case state is len; the counter for the number of test cases co = 0;

[0045] Step2: When co = L, jump to Step3; otherwise, jump to Step9;

[0046] Step3: Given the initial state s0;

[0047] Step4: If s0 is not a termination state and len < maxL, jump to Step5; otherwise, co = co + 1, obtain the test case T, and jump to Step8;

[0048] Step 5: Use the ε-greedy strategy to select action a according to the Q(s,a) value table t ;

[0049] Step 6: Take action a t , get the next state s t +1;

[0050] Step7: Make s t ←s t+1 , jump to Step 4;

[0051] Step 8: Calculate the reward value G for each state in the entire test case T according to the reward function formula R(t) t , and use the formula Q(s t ,a t )←Q(s t ,a t )+θ(G t -Q(s t ,a t )) Update Q(s,a); after the update is completed, jump to Step 2;

[0052] Step 9: Output Q(s,a);

[0053] Among them, the terminal state means that all sub-states of state S are completed by jumping. For example, S has a total of 4 sub-states, S0, S1, S2, and S3. When these four states S are completed, the terminal state is reached.

[0054] The state transition strategy used in the present invention is the ε-greedy strategy. The formula is:

[0055]

[0056] Among them, the action with a large q value is selected with a probability of 1-ε, and the random action is selected with a probability of ε.

[0057] Furthermore, the evaluation criteria in step S4 include:

[0058] NAPFD: Average historical detection rate. The failure distribution is calculated based on the actual detection error rate.

[0059] TFL: Evaluates based on the position of the test case in the test sequence that fails for the first time during test execution.

[0060] Recall: Evaluated based on the ratio of errors actually detected in the test execution sequence to the errors actually existing in the integration cycle.

[0061] Beneficial effects of the present invention:

[0062] The present invention proposes a method for giving reasonable weights to the test results and the adjacent test execution history information, which is applicable to the research problem, and designs a new reward function to solve the defects of the following two methods. The first defect of the method of applying rewards to test results is low failure detection rate and difficulty in convergence; the second defect of the method of applying rewards to test history information is long test feedback time.

[0063] The present invention aims at the problems of different component naming methods and data heterogeneity in hardware schematics of various enterprises in the automotive industry, and achieves the no-sense component naming method by defining a universal data structure, thereby achieving the unification of intelligent system input. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a process flow chart of the method of the present invention;

[0065] Figure 2 This is a general data processing flow chart of step S1 of the method of the present invention;

[0066] Figure 3 This is a flow chart of initializing the Q-table of the method of the present invention;

[0067] Figure 4 This is an input circuit diagram of a front area controller of a certain vehicle according to Embodiment 1 of the present invention;

[0068] Figure 5 This is a flow chart of input circuit data processing according to an embodiment of the present invention;

[0069] Figure 6 This is a Q-table diagram of Embodiment 1 of the present invention;

[0070] Figure 7 A flowchart of optimal test case selection according to Embodiment 1 of the present invention is shown. DETAILED DESCRIPTION

[0071] The technical solution provided by the present application will be further described below in conjunction with specific embodiments and accompanying drawings. The advantages and features of the present application will become more apparent with the following description.

[0072] Embodiment 1

[0073] In an embodiment of the present invention, a method for generating a vehicle front area controller EOL test program based on reinforcement learning is proposed, comprising the following steps: (e.g. Figure 1 )

[0074] S1: Obtain the original data of components according to the component list of a certain automobile front area controller, and obtain the connection attribute data of components according to the connection parameters between the components in the schematic diagram. Figure 2 .

[0075] The input of step S1 is the schematic BOM table, and the specific processing is as follows.

[0076] Specifically, the original data of components and the data of component connection attributes in the original circuit are processed, and the original data of components are combined into minimum circuit combination data through the data processing process.

[0077] The original data of this embodiment uses the BOM data of a certain input circuit of a certain front area controller of a certain car. The circuit diagram is as follows: Figure 4 , the corresponding components are shown in Table 1:

[0078] Table 1

[0079] The left node type of the component Component name The right node type of the component C001 Component Parameters N001 C001 G001 R001 Component Parameters P001 R001 N001 R001 Component Parameters N001 R002 S001 R001 Component Parameters N001 R003 G001

[0080] The "N" in the parameter "N0001" means that the NET end will be connected to a NET end of another component, and "0001" refers to the same type of data number of the NET end in the entire data set. One point to note is that the numbers of the two connected NET ends are the same. The "R" in the parameter "R0001" refers to a resistor type component, and "0001" refers to the same type of data number of this resistor type component in the entire data set. The "C" in the parameter "C0001" refers to a capacitor type component, and "0001" refers to the same type of data number of this capacitor type component in the entire data set. The "G" in the parameter "G001" means that the NET end is connected to the controller's ground. "0001" refers to the same type of data number of this grounding type data in the entire data set, but for the front area controller of this vehicle, there is only one grounding, so the number of all grounding type data in the entire data set is 001. The "S" in the parameter "S001" means that the NET end is directly connected to the analog input port of the chip in the controller. "0001" refers to the data number of the same type of this analog input data in the entire data set.

[0081] The input circuit model is represented by a vector group as follows:

[0082] Y={(P001,R001,N001), (N001,C001,G001), (N001,R002,S001), (N001,R003,G001)}.

[0083] The intermediate process of the transformation of vector group Y is shown in Table 2.

[0084] Table 2

[0085] 0 P001 0 0 0 0 R001 0 0 0 N001 N001 N001 R002 S001 N001 N001 N001 R002 S001 0 C001 R003 0 0 0 G001 G001 0 0

[0086] The final result after the transformation of vector group Y is shown in Table 3.

[0087] Table 3

[0088] 0 P001 0 0 0 0 R001 0 0 0 N001 N001 N001 R002 S001 0 C001 R003 0 0 0 G001 G001 0 0

[0089] Figure 5 The present invention shows an input circuit data processing flow chart of a method for generating a vehicle domain controller EOL test program based on reinforcement learning in the first embodiment of the present invention.

[0090] The vector group Y is processed to generate the minimum circuit model. The specific processing steps are as follows:

[0091] First, after data elimination and vector transposition, the vectors in Y are processed according to the data with the same net to generate a new vector set Q, that is, Q = {(N001, C001, G001) T ,(P001,R001,N001) T ,(S001,R002,N001) T ,(N001,R003,G001) T}

[0092] Then, after data processing such as data combination and offset, and after data extraction and data elimination, Q is processed through data combination to generate R, that is,

[0093] R={(P001,R001,N001,N001,C001,G001) T ,(0,0,N001,N001,R003,G001)T,(0,0,R002,0,0,0) T ,(0,0,S001,0,0,0) T}.

[0094] The pseudo code for data processing from matrix Q to matrix R is as follows:

[0095] Step 1: Given parameter st k (sta k ,stb k ,stc k ), defined as the original element vector. k=1,2,3..., all element vectors st k (sta k ,stb k ,stc k ) forms the matrix Y

[0096] Step 2: ifsta k or stc k=FIND(P)or FIND(G); thenst k (sta k ,stb k ,stc k )=st k (sta k ,stb k ,stc k ) T ; The FIND(G) function indicates that the text contains the character G.

[0097] Step 3: if sta k or stc k =FIND(S)or FIND(S);then st k (sta,stb,stc)=st k (sta,stb,stc).

[0098] Step 4: if sta k and stc k =FIND(S)or FIND(P)or FIND(G)or FIND(N) then k=k+1, jump to Step2.

[0099] Step 5: After the above steps, the matrix Q will be generated

[0100] Step 6: if for the st k (sta k ,stb k ,stc k )and st k+1 (sta k+1 ,stb k+1 ,stc k+1 ),stc k =sta k+1 then st k (sta k ,stb k ,stc k )and st k+1 (sta k+1 ,stb k+1 ,stc k+1 )will merge to st k

[0101] (sta k ,stb k ,stc k ,sta k+1,stb k+1 ,stc k+1 )and then merge to st k (sta k ,stb k ,0,0,stb k+1 ,stc k+1 ).

[0102] Step7: After the above steps, we finally get a matrix R.

[0103] Then comes the data grouping process. After the above series of data processing, the original data set Y will be divided into many sub-data sets {R q}, these sub-datasets R q Respectively represent a minimum circuit model data, and this embodiment has only one set of sub-datasets R, namely

[0104] {(P001,R001,N001,N001,C001,G001) T ,(0,0,N001,N001,R003,G001) T ,(0,0,R002,0,0,0) T ,(0,0,S001,0,0,0) T} is a set of minimum circuit model data representing the minimum circuit.

[0105] The sub-dataset R of this embodiment constitutes the experimental data set of this embodiment, which is a two-dimensional test vector.

[0106] S2: Based on the two-dimensional vector dataset R generated in step S1 (the input of step S2), an optimal EOL offline test case selection model is constructed (the output of step S2).

[0107] Figure 4 A Q-table initialization flow chart in a vehicle domain controller EOL test program generation method based on reinforcement learning of the present invention is shown.

[0108] In this embodiment, a reinforcement learning model is used to construct an optimal EOL offline test case selection model.

[0109] First, the state, action, agent, policy, and reward function in reinforcement learning are defined as follows:

[0110] The four parameters P001, N001, S001, and G001 (types of each interface of the minimum circuit) in the two-dimensional vector data set R generated in step S1 are defined as the State in the reinforcement learning model, and all the States constitute the state space.

[0111] From state s t (Current interface type) Jump to s t+1 The abstract action of (next interface type) is defined as the action in the reinforcement learning model. All actions constitute the action space. In this embodiment, the first action is to jump from state P001 to state N001, the second action is to jump from state N001 to state S001, and the third action is to jump from state S001 to state G001, so there are only three action spaces in this embodiment.

[0112] Then, after step S3, the Q-Learning algorithm is used to generate the EOL test program with the highest current priority based on the test case selection model. The test program is executed to collect test results, and the reward value is calculated based on the reward function and the intelligent test system is updated. The Agent is the test vector set R generated in step S1.

[0113] S3: This embodiment builds a Q-table type agent based on the Q-Learning algorithm model. The agent is a 6*4 dimensional two-dimensional test vector. It is initialized by the initialization update algorithm based on the initial data constructed in step S1, and then the Q-table is updated according to the model update algorithm to continuously generate the optimal EOL offline test cases. For the Q-table initialization part, the Q-table is initialized according to a designed algorithm. First, a set of original experimental data generated in step S1 is used as the experimental data R{(P001, R001, N001, N001, C001, G001) T ,(0,0,N001,N001,R003,G001) T ,(0,0,R002,0,0,0) T ,(0,0,S001,0,0,0) T}, and then traverse the adjacent vector data in the original data U = {(P001, R001, N001, N001, C001, G001) T ,(0,0,N001,N001,R003,G001) T};V={(0,0,N001,N001,R003,G001) T ,(0,0,R002,0,0,0) T}; W = {(0,0,R002,0,0,0) T (0,0,S001,0,0,0) T}.

[0114] Determine whether the adjacent data U, V, W are in the state-action space (s, a). Because the state-action space is relatively small in this embodiment, and the adjacent data U, V, W are not in the state-action space (s, a), U, V, W are added to the state-action space (s, a), and the Q value is set to 1, and then the Q-table is initialized.

[0115] Figure 6 A Q-table diagram of a method for generating a vehicle domain controller EOL test program based on reinforcement learning in Embodiment 1 of the present invention is shown.

[0116] Table 4 shows an initialization Q-table of a Q-table diagram of a vehicle domain controller EOL test program generation method based on reinforcement learning in Embodiment 1 of the present invention.

[0117] Table 4

[0118] Action 1 (N001) Action 2 (P001) Action 3 (G001) Action 4 (S001) Status 1 (N001) 0.03 0.02 0.00 0.04 Status 2 (P001) 0.00 0.00 0.02 0.01 State 3 (G001) 0.00 0.00 0.00 0.00 State 4 (S001) 0.01 0.00 0.00 0.00

[0119] S4: Use the test vector set generated by S1 as the agent of the EOL offline test case selection model, and execute a test case on the circuit model to be tested. After execution, the test result will be generated, and then the reward function will feedback a reward value to the Agent based on the test result. The Agent decides the next action to be performed based on the action executed this time and the reward value obtained, and then repeats the cycle until the test case is executed, and then the best test case is selected according to the proposed evaluation criteria. Figure 7 A flowchart of optimal test case selection for a method for generating a vehicle domain controller EOL test program based on reinforcement learning in Embodiment 1 of the present invention is shown.

[0120] Table 5 shows the intermediate state of the test case set finally generated by a method for generating a vehicle domain controller EOL test program based on reinforcement learning in Embodiment 1 of the present invention (the final test case can be saved in a json file format).

[0121] Table 5

[0122] Step 1 Step 2 Step 3 Step 4 Test1 N001 P001 G001 S001 Test2 N001 G001 S001 P001 ... ... ... ... ... Test n P001 G001 S001 N001

[0123] Table 6 shows a set of test cases finally generated by a method for generating a vehicle domain controller EOL test program based on reinforcement learning in Embodiment 1 of the present invention (the final test cases are saved in a json file format).

[0124] Table 6

[0125]

[0126] Table 7 shows the result indicators after the execution of the test cases finally generated by the method for generating a vehicle domain controller EOL test program based on reinforcement learning in the first embodiment of the present invention.

[0127] Table 7

[0128]

[0129] The state transition strategy used in this embodiment is the ε-greedy strategy. The formula is:

[0130]

[0131] The action with a large q value is selected with a probability of 1-ε, and the random action is selected with a probability of ε. In this embodiment, the ε value is positioned at 0.3.

[0132] The formula of the reward function is:

[0133]

[0134] Where y is the total number of historical executions of test case t, r i is the test result of the jth execution of test case t, represented by 1 or 0, where 1 indicates test failure and 0 indicates test pass. j represents the jth test cycle, T is the test case set of the current cycle, TS j is a prioritized set of test cases. It is the number of test cases whose execution failed in the current test cycle.

[0135] This embodiment uses the Monte Carlo update algorithm to update the model, and the formula is: Q(s t ,a t )←Q(s t ,a t )+θ(G t -Q(s t ,a t ))

[0136] Among them G t is the state s in this round t The reward value of . The algorithm pseudo code is as follows:

[0137] Step 1: Given a step size parameter θ = 1, an initial ε value of ε = 0.3, and a maximum number of test case states of 4;

[0138] After initialization, Q(s,a) is as follows Figure 6As shown; the maximum number of test case status is 4; the length of a single test case status is

[0139] The initial value of the degree counter len is len=0; the test case number counter co=0;

[0140] Step 2: When co=4, jump to Step 3; otherwise jump to Step 9;

[0141] Step 3: Given the initial state P001;

[0142] Step 4: Because P001 is not in the terminal state and len<4, jump to Step 5;

[0143] Step 5: Use the ε-greedy strategy to select action a according to the Q(s,a) value table t :P001->N001;

[0144] Step 6: Take action a t :P001->N001, get the next state N001;

[0145] Step7: Then make s t ←s t+1 , jump to Step 4;

[0146] Step 8: Calculate the reward value G for each state in the entire test case T according to the reward function formula t , and use the formula Q(s t ,a t )←Q(s t ,a t )+θ(G t -Q(s t ,a t )) Update Q(s,a); after the update is completed, jump to Step 2;

[0147] Step 9: Output Q(s,a).

[0148] Embodiment 2

[0149] In order to solve the above-mentioned technical problems existing in the prior art, an embodiment of the present invention also provides an automatic generation method of an EOL test program based on reinforcement learning for a certain output circuit of a certain automobile front area controller. The automatic generation process of the EOL test program of the output circuit can be based on all steps of the EOL test program generation method of the vehicle front area controller based on reinforcement learning.

[0150] The specific steps of the method for automatically generating an EOL test program based on reinforcement learning for an output circuit of a front area controller of a certain automobile and the beneficial effects of generating an EOL test program provided by the embodiment of the present invention are the same as those in Example 1 and will not be described in detail here.

[0151] The above description is only a description of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical contents shall be deemed as equivalent effective embodiments and shall fall within the scope of protection of the technical solution of the present application.

Claims

1. A method for generating a vehicle domain controller EOL test program based on reinforcement learning, characterized in that: It includes the following steps: S1: Obtain the original component data according to the component list of a certain automotive front area controller, and obtain the component connection attribute data according to the connection mode parameters between components in the schematic diagram. Process the original circuit and component connection attributes, and combine them into the minimum circuit combination data; S2: Build an optimal EOL offline test case selection model on the two-dimensional vector data set obtained in step S1; S3: Based on the model in S2, use the Q-Learning algorithm to build an intelligent agent Agent of the Q-table type. According to the minimum circuit combination data, initialize it through the initialization update algorithm, and update the Q-table according to the model update algorithm; S4: Execute test cases on the circuit model to be tested, generate high-priority test cases based on the reward function and state transition strategy, and select the optimal test case according to the established evaluation criteria.

2. The method for generating a vehicle domain controller EOL test program based on reinforcement learning according to claim 1, characterized in that: Step S1 is specifically as follows: Get the original data: The original data is the electronic component information set X{COM x }, where COM x Expressed as (net i ,com z ,net j ), where com z Refers to the name of the component, net i Refers to the network name (net name) on the left side of the component. j It refers to the net name on the right side of the component; Data extraction: Based on the original data, COMs with the same net name are grouped into a set A{COM1, COM2, COM3, ...COM w }, where w is the number of COMs with the same net name; Data elimination and vector transposition: Eliminate the COM in A according to the data with the same net, and then perform vector transposition processing to generate a new vector set B; Data combination and cancellation: After data extraction and data elimination processing, perform data combination processing on B to generate C; Grouping processing: After the above series of data processing, the original data set X is divided into many sub-data sets {C q }, these sub-datasets C q Respectively represent a minimum circuit model data; Get the experimental data set: sub-dataset {C q } constitutes the experimental dataset of this study.

3. The method for generating a vehicle domain controller EOL test program based on reinforcement learning according to claim 1, characterized in that: Step S2 is specifically as follows: Use a reinforcement learning model to build an optimal EOL offline test case selection model. The state State, action Action, intelligent agent Agent, policy Policy, and reward function Reward function in reinforcement learning are defined as follows: State: vector group {C q } is n*m dimensional, where C q The non-zero parameters (types of each interface of the minimum circuit) in the first row, first column and the nth row, mth column are defined as the State in the reinforcement learning model, and all States constitute the state space; Action: Change the current interface type from state to s t Jump to next interface type t+1 The abstract action is defined as the Action in the reinforcement learning model; all actions constitute the action space; Intelligent agent Agent: The two-dimensional test vector set generated in step S1; Policy Policy: The reinforcement learning ε-greedy policy; Reward function Reward function: The formula is as follows: Where y is the total number of historical executions of test case t, r i is the test result of the i-th execution of test case t, represented by 1 or 0, where 1 indicates test failure and 0 indicates test pass. j represents the j-th test cycle, T is the test case set of the current cycle, TS j is a prioritized set of test cases. It is the number of test cases whose execution failed in the current test cycle.

4. The method for generating a vehicle domain controller EOL test program based on reinforcement learning according to claim 1, characterized in that: Step S3 is specifically as follows: After the intelligent agent generates a test program, the environment executes the test program, collects the test results, calculates the reward value according to the reward function and provides it to the intelligent agent, and the intelligent agent updates according to the reward value; After updating the intelligent agent, the intelligent agent generates the next round of test programs, and loops until the termination condition is met; The specific process of updating the intelligent agent is as follows: The model update algorithm used is the Monte Carlo update algorithm: Q(s t ,a t )←Q(s t ,a t )+θ(G t -Q(s t ,a t )) Among them G t is the state s in this round t The pseudo code of the Monte Carlo update algorithm is as follows: Step1: Given the step size parameter θ, the initial ε value, and the maximum number of test case states L; Initialize Q(s,a); The maximum value of the test case state is maxL; The counter for the length of a single test case state is len; The counter for the number of test cases co = 0; Step2: When co = L, jump to Step3; otherwise, jump to Step9; Step3: Given the initial state s0; Step4: If s0 is not a termination state and len < maxL, jump to Step5; otherwise, co = co + 1, obtain the test case T, and jump to Step8; Step 5: Use the ε-greedy strategy to select action a according to the Q(s,a) value table t ; Step 6: Take action a t , get the next state s t +1; Step7: Make s t ←s t+1 , jump to Step 4; Step 8: Calculate the reward value G for each state in the entire test case T according to the reward function formula R(t) t , and use the formula Q(s t ,a t )←Q(s t ,a t )+θ(G t -Q(s t ,a t )) Update Q(s,a); after the update is completed, jump to Step 2; Step9: Output Q(s,a); Among them, the termination state means that all sub-states of state S have been jumped. For example, if S has a total of 4 sub-states, S0, S1, S2, S3, when these four states S have been jumped, the termination state is reached; The state transition strategy is the ε-greedy strategy, and the formula is: Among them, the action with a large q value is selected with a probability of 1-ε, and the random action is selected with a probability of ε.

Images