A reinforcement learning-based method for generating EOL test programs for vehicle domain controllers

Abstract

This invention belongs to the field of automatic generation of EOL (End-of-Life) test programs, specifically involving a reinforcement learning-based method for generating EOL test programs for vehicle domain controllers. It includes the following steps: S1: Obtaining raw component data from the schematic diagram, processing the raw circuit and component connection attributes to form a minimum circuit combination; S2: Constructing an optimal EOL test case selection model on the two-dimensional vector dataset obtained in step S1; S3: Based on the model in S2, generating the highest-priority EOL test program using the Q-Learning algorithm; S4: Selecting the optimal test case according to evaluation criteria. This invention addresses the shortcomings of reward-based test results methods, such as low failure rate, difficulty in convergence, and long test feedback time.

Description

Technical Field

[0001] This invention belongs to the field of automatic generation of EOL (End of Service) test programs, specifically involving a method for generating EOL test programs for vehicle domain controllers based on reinforcement learning. Technical Background

[0002] Currently, all automotive controller manufacturers require end-of-line (EOL) testing for their products, but the testing equipment and methods used vary. One approach involves designing an EOL test system control unit based on an embedded platform. This method employs a modular approach to design the control unit hardware, then designs corresponding low-level drivers for each hardware module, and designs application-level algorithms to automate the system's workflow. A wheel speed simulation model is then built using MATLAB / Simulink, coded, and ported into the embedded program. Another approach uses LabVIEW to develop a host computer program and designs a simple and intuitive software user interface to automate testing. A third approach is to develop an EOL test system based on virtual instruments, enabling automatic ECU connection, intelligent power management, automatic program burning, one-click testing, and intelligent database management.

[0003] The methods described above can achieve varying degrees of automation in EOL testing. However, these research directions primarily focus on automating the execution of test systems during EOL shutdown testing, without addressing the efficiency improvement issues in developing EOL test programs during the early stages of project development. The development of test programs for each different controller still largely relies on manual work by engineering technicians. This fails to resolve the tediousness and long development cycle of test program development in the early stages of project development.

[0004] Reinforcement learning is a machine learning method that learns how to map states to behaviors in order to maximize rewards. In other words, it requires an agent to continuously experiment in the environment, constantly optimizing the state-behavior mapping through feedback (rewards) from the environment.

[0005] Current research on reinforcement learning-based test program generation in the field of software testing can be broadly divided into two categories: one focuses on prioritizing the execution of failed test cases, i.e., rewarding the results. However, since test case failures are relatively rare, this method suffers from low failure detection rates and difficulty in convergence. The other category primarily focuses on the historical information of test case execution, i.e., rewarding the impact of failures. However, due to the massive amount of historical data, more computational resources are required, resulting in long test feedback times.

[0006] Currently, end-of-life (EOL) testing for vehicle controllers on the production line relies on engineers manually writing test programs. However, the hardware circuits of engine controllers, transmission controllers, body controllers, central computing controllers, and domain controllers in vehicles are very similar, and their hardware off-line testing methods are also very similar. During the R&D phase of vehicle domain controllers, projects often have short timelines and urgent development tasks. Developing test cases for vehicle domain controller hardware off-line testing places a significant workload on R&D personnel. Therefore, building an intelligent testing system to automatically generate test cases for automotive electronic domain controllers is essential.

[0007] Currently, there is no intelligent test program generation method specifically developed for EOL (End-of-Life) testing of vehicle domain controllers. Therefore, in the field of automotive controller EOL testing, a method based on reinforcement learning is proposed to build an intelligent system that automatically generates automotive controller EOL test programs to replace the repetitive manual labor of engineers writing a large number of test programs. This can not only improve the efficiency of test program generation but also increase the functional test coverage, thereby improving the production efficiency of actual industrial applications. Summary of the Invention

[0008] To address the problems existing in the prior art, the purpose of this invention is to provide a reinforcement learning-based method for generating EOL test programs for vehicle domain controllers, thus offering a feasible solution to the problems of low efficiency and high error rate in manually developing EOL offline test programs.

[0009] Technical solution of the present invention:

[0010] A method for generating an EOL test program for a vehicle domain controller based on reinforcement learning includes the following steps:

[0011] S1: Obtain the original data of the components according to the component list of a certain car front area controller, and obtain the component connection attribute data according to the connection method parameters between each component in the schematic diagram. Process the original circuit and component connection attributes to form the minimum circuit combination data.

[0012] S2: Based on the minimum circuit combination data generated in step S1, construct an optimal EOL offline test case selection model;

[0013] S3: Based on the S2 model, a Q-table type agent is constructed using the Q-Learning algorithm. The agent is initialized using the minimum circuit combination data and the Q-table is updated using the initialization update algorithm.

[0014] S4: Execute test cases on the circuit model under test, generate high-priority test cases based on the reward function and state transition strategy, and select the optimal test cases according to the established evaluation criteria.

[0015] Furthermore, step S1 is detailed as follows:

[0016] First, obtain the raw data: The raw data for this study is the set of electronic component information X{COM from the schematic diagram of the vehicle domain controller. x}, where COM x Represented as (net) i ,com z net j ), where com z This refers to the name of the component, net i This refers to the network name (net name) on the left side of the component. j This refers to the net name on the right side of the component.

[0017] Given a set of raw data (M002, C001, G001), (P001, R001, M002), (M002, R002, N001), ... In parameter "M001", "M" indicates that this NET terminal will be connected to another NET terminal of a component, and "001" indicates the data number of this NET terminal of the same type in the entire dataset. It is important to note that two connected NET terminals have the same number. The second parameter refers to the component's name and type. In parameter "R001", "R" indicates a resistive component, and "001" indicates the data number of this type of resistive component of the same type in the entire dataset. In "C001", "C" indicates a capacitive component, and "001" indicates the data number of this type of capacitive component of the same type in the entire dataset. Each parameter specifically represents a component type and its number.

[0018] Next is data extraction: Based on the original data, COMs with the same net name are grouped into a set A{COM1,COM2,COM3,...COM}. w}, where w is the number of COMs with the same net name.

[0019] For example, given a set of original data {(M002,C001,G001), (P001,R001,M002), (M002,R002,N001)}

[0020] Data removal and vector transpose: Remove COM data from A based on data with the same net, and then transpose the vectors to generate a new vector set B.

[0021] For the example above, we get (M002, C001, G001). T , (P001,R001,M002), (M002,R002,N001)};

[0022] Next is data combination and cancellation: After data extraction and data removal, B is combined to generate C, i.e., {(P001,0,0)}. T (R001,0,0) T (M002,C001,G003) T (R002,0,0) T (N001,0,0) T};

[0023] Next is grouping: After the above series of data processing steps, the original dataset X will be divided into many subsets {C}. q These subsets C q Each represents a minimum circuit model data.

[0024] Finally, the experimental dataset: Subdataset {C} q This constitutes the experimental dataset for this study. This subset of the dataset is a two-dimensional test vector set.

[0025] Furthermore, step S2 is detailed as follows:

[0026] A reinforcement learning model is used to construct an optimal EOL (Extreme Online Performance) test case selection model. The state, action, agent, policy, and reward function in reinforcement learning are defined as follows:

[0027] State: The vector set {C} generated in the data processing part of step S1. q} is of n*m dimensionality, where C q The non-zero parameters (types of each interface of the minimum circuit) in the first row, first column, nth row, and mth column are defined as the State in the reinforcement learning model, and all the States constitute the state space.

[0028] 动作(Action) ： 将从状态 s t (当前接口类型)跳转到 s t+1 (下一个接口类型)的抽象的动作定 In reinforcement learning models, Action refers to the action space. All Actions together form the action space.

[0029] Agent: The two-dimensional test vector set generated in step S1

[0030] Policy: ε-greedy policy in reinforcement learning.

[0031] Reward function: The formula is as follows:

[0032]

[0033] Where y is the total number of historical executions of test case t, and r i is the result of the i-th execution of test case t, represented by 1 or 0, where 1 indicates test failure and 0 indicates test success. j represents the j-th test cycle, T is the set of test cases for the current cycle, and TS j It is the set of test cases after priority sorting. This represents the number of test cases that failed during the current test cycle.

[0034] Furthermore, based on the Q-Learning algorithm model, a Q-table type agent is constructed. This agent is an n*m dimensional test vector. It is initialized using the initial data constructed in step S1 through an initialization update algorithm, and then the Q-table is updated according to the model update algorithm to continuously generate optimal EOL offline test cases.

[0035] In the Q-table initialization part, some works often initialize the Q-table to all zeros. This method can easily cause unordered jumps in the Q-table during the early stages of iteration, affecting convergence. This invention provides an algorithm for initializing the Q-table based on experimental data. Details are as follows:

[0036] First, obtain a set of raw experimental data C(a1,a2,...a...). n Then iterate through the adjacent vector data (a) in the original data. u ,a v ), determine the adjacent data (a) u ,a v Is it in the state-action space (s, a)? If (a) u ,a v In the state-action space (s, a), then (a) u ,a v The Q value of (a) will increase by 1; if (a) u ,a v If (a) is not in the state-action space (s, a), then (a) u ,a v) Add it to the state-action space (s, a) and set the Q value to 1. Then set (a u , a v ) to be traversed completely. If (a u , a v ) has been traversed, then the Q-table initialization is completed; otherwise, continue to traverse the vector data (a u , a v ).

[0037] Furthermore, step S3 is specifically as follows:

[0038] After the agent generates a test program, the environment executes the test program, collects the test results, calculates the reward value according to the reward function, and provides it to the agent. The agent updates according to the reward value. After updating the agent, the agent generates the next round of test programs, and iterates until the termination condition is met.

[0039] Among them, the execution of the test program is completed through software, such as the CFI software of Bosch.

[0040] The specific process of agent update is as follows:

[0041] The model update algorithm used is the Monte Carlo update algorithm: Q(s t , a t ) ← Q(s t , a t ) + θ(G t - Q(s t , a t ))

[0042] Among them, G t is the reward value of the state s t in this round. The pseudocode of the Monte Carlo update algorithm is as follows:

[0043] Step1: Given the step size parameter θ, the initial ε value, and the maximum number of test case states L;

[0044] Initialize Q(s, a); the maximum value of the test case state is maxL; the counter for the length of a single test case state is len; the counter for the number of test cases co = 0;

[0045] Step2: When co = L, jump to Step3; otherwise, jump to Step9;

[0046] Step3: Given the initial state s0;

[0047] Step4: If s0 is not a termination state and len < maxL, jump to Step5; otherwise, co = co + 1, obtain the test case T, and jump to Step8;

[0048] Step 5: Use the ε-greedy strategy to select action a based on the Q(s,a) value table. t ;

[0049] Step 6: Take action a t , obtain the next state s t +1;

[0050] Step 7: Make s t ←s t+1 Jump to Step 4;

[0051] Step 8: Calculate the reward value G for each state in the entire test case T based on the reward function formula R(t). t And using the formula Q(s) t ,a t )←Q(s t ,a t )+θ(G t -Q(s t ,a t Update Q(s,a); after the update is complete, proceed to Step 2.

[0052] Step 9: Output Q(s,a);

[0053] The terminating state is when all the sub-states of state S have been transitioned to. For example, if S has four sub-states, S0, S1, S2, and S3, the terminating state is reached when all four sub-states of S have been transitioned to.

[0054] The state transition strategy used in this invention is the ε-greedy strategy. The formula is:

[0055]

[0056] Among them, the action with a large q value is selected with a probability of 1-ε, and a random action is selected with a probability of ε.

[0057] Furthermore, the evaluation criteria in step S4 include:

[0058] NAPFD: Average Historical Detection Rate. The failure distribution is calculated based on the actual detection error rate.

[0059] TFL: Evaluation is based on the position of the test case that fails the first time it is executed during test execution within the test sequence.

[0060] Recall: Evaluation is based on the ratio of errors actually detected in the test execution sequence to errors actually existing during the integration cycle.

[0061] The beneficial effects of this invention are:

[0062] This invention proposes a method for assigning reasonable weights to test results and adjacent test execution history information, applicable to the research problem, and designs a new reward function to address the shortcomings of the following two methods. The first method, which applies rewards to test results, suffers from low detection failure rate and difficulty in convergence; the second method, which applies rewards to test history information, suffers from long test feedback time.

[0063] This invention addresses the problem of inconsistent component naming methods and heterogeneous data in the hardware schematics of various companies in the automotive industry. It achieves seamless integration of component naming methods by defining a universal data structure, thereby unifying the input of intelligent systems. Attached Figure Description

[0064] Figure 1 This is a flowchart of the method of the present invention;

[0065] Figure 2 This is a general data processing flowchart for step S1 of the method of the present invention;

[0066] Figure 3 This is a flowchart of the Q-table initialization process of the method of the present invention;

[0067] Figure 4 This is an input circuit diagram of a car front area controller according to Embodiment 1 of the present invention;

[0068] Figure 5 This is a flowchart of the input circuit data processing in Embodiment 1 of the present invention;

[0069] Figure 6 This is a Q-table diagram from Embodiment 1 of the present invention;

[0070] Figure 7 The flowchart for selecting the optimal test cases in Embodiment 1 of the present invention is shown. Detailed Implementation

[0071] The technical solutions provided in this application will be further described below with reference to specific embodiments and accompanying drawings. The advantages and features of this application will become clearer from the following description.

[0072] Example 1

[0073] In this embodiment of the invention, a method for generating an EOL test program for a vehicle front area controller based on reinforcement learning is proposed, including the following steps: (e.g.) Figure 1 )

[0074] S1: Obtain the original component data from the component list of a certain automotive front area controller, and obtain the component connection attribute data based on the connection parameters between each component in the schematic diagram. For example... Figure 2 .

[0075] The input for step S1 is the schematic BOM table, and the specific processing is as follows.

[0076] Specifically, the original component data and component connection attribute data in the original circuit are processed. From the original component data, through the data processing process, the original data is combined into the minimum circuit combination data.

[0077] The raw data used in this embodiment is the BOM data of an input circuit of a certain automotive front area controller. The circuit diagram is as follows. Figure 4 The corresponding components are shown in Table 1:

[0078] Table 1

[0079] Left node type of components Component Name Component right-side node type C001 component parameters N001 C001 G001 Parameters of R001 component P001 R001 N001 Parameters of R001 component N001 R002 S001 Parameters of R001 component N001 R003 G001

[0080] In parameter "N0001", "N" indicates that this NET terminal will be connected to another NET terminal of a component, and "0001" refers to the data number of this NET terminal of the same type in the entire dataset. It's important to note that two connected NET terminals have the same number. In parameter "R0001", "R" indicates a resistive component, and "0001" refers to the data number of this type of resistive component of the same type in the entire dataset. In parameter "C0001", "C" indicates a capacitive component, and "0001" refers to the data number of this type of capacitive component of the same type in the entire dataset. In parameter "G001", "G" indicates that this NET terminal is connected to the controller's ground. "0001" refers to the data number of this grounding type of data of the same type in the entire dataset; however, for this vehicle's front area controller, there is only one ground, therefore, the number of all grounding type data in the entire dataset is 001. In parameter "S001", "S" indicates that this NET terminal is directly connected to the analog input port of the chip in the controller. "0001" refers to the data number of the same type in the entire dataset for this type of simulated input data.

[0081] The input circuit model is represented by a vector group as follows:

[0082] Y={(P001,R001,N001), (N001,C001,G001), (N001,R002,S001), (N001,R003,G001)}.

[0083] The intermediate process of transforming vector group Y is shown in Table 2.

[0084] Table 2

[0085] 0 P001 0 0 0 0 R001 0 0 0 N001 N001 N001 R002 S001 N001 N001 N001 R002 S001 0 C001 R003 0 0 0 G001 G001 0 0

[0086] The final result after the transformation of vector group Y is shown in Table 3.

[0087] Table 3

[0088] 0 P001 0 0 0 0 R001 0 0 0 N001 N001 N001 R002 S001 0 C001 R003 0 0 0 G001 G001 0 0

[0089] Figure 5 The diagram shows the input circuit data processing flowchart of a reinforcement learning-based vehicle domain controller EOL test program generation method according to Embodiment 1 of the present invention.

[0090] The vector group Y undergoes data processing to ultimately generate a minimum circuit model. The specific processing steps are as follows:

[0091] First, after data removal and vector transposition, the vectors in Y are processed by removing data with the same net and transposing the vectors to generate a new vector set Q, i.e., Q = {(N001, C001, G001)}. T (P001,R001,N001) T (S001,R002,N001) T (N001,R003,G001) T}

[0092] After data processing including data combination and offsetting, and data extraction and elimination, Q is then processed through data combination to generate R, i.e.

[0093] R={(P001,R001,N001,N001,C001,G001) T ,(0,0,N001,N001,R003,G001)T,(0,0,R002,0,0,0) T (0,0,S001,0,0,0) T}

[0094] The pseudocode for data processing from matrix Q to matrix R is as follows:

[0095] Step 1: Given the parameter st k (sta k stb k stc k ), defined as the original element vector. k = 1, 2, 3..., all element vectors st k (sta k stb k stc k ) Form matrix Y

[0096] Step 2: ifsta k or stc k=FIND(P)or FIND(G); then k (sta k stb k stc k )=st k (sta k stb k stc k ) T The FIND(G) function indicates that the text contains the character G.

[0097] Step 3: if sta k or stc k =FIND(S)or FIND(S);then st k (sta, stb, stc) = st k (sta,stb,stc).

[0098] Step 4: if sta k and stc k =FIND(S)or FIND(P)or FIND(G)or FIND(N) then k=k+1, jump to Step2.

[0099] Step 5: After the above steps, matrix Q will be generated.

[0100] Step 6: if for the st k (sta k stb k stc k )and st k+1 (sta k+1 stb k+1 stc k+1 ), stc k =sta k+1 then st k (sta k stb k stc k )and st k+1 (sta k+1 stb k+1 stc k+1 )will merge to st k

[0101] (sta k stb k stc k ,sta k+1stb k+1 stc k+1 )and then merge to st k (sta k stb k ,0,0,stb k+1 stc k+1 ).

[0102] Step 7: After the above steps, we finally obtain a matrix R.

[0103] Next comes data grouping. After the above series of data processing steps, the original dataset Y will be divided into many subsets {R}. q These subsets R q Each represents a minimum circuit model data set. In this embodiment, there is only one subset R, i.e.

[0104] {(P001,R001,N001,N001,C001,G001) T ,(0,0,N001,N001,R003,G001) T ,(0,0,R002,0,0,0) T (0,0,S001,0,0,0) T} represents a set of minimum circuit model data.

[0105] The subset R in this embodiment constitutes the experimental dataset of this embodiment, which is a two-dimensional test vector.

[0106] S2: Based on the two-dimensional vector dataset R generated in step S1 (the input of step S2), construct an optimal EOL offline test case selection model (the output of step S2).

[0107] Figure 4 The flowchart of Q-table initialization in a reinforcement learning-based vehicle domain controller EOL test program generation method of the present invention is shown.

[0108] In this embodiment, a reinforcement learning model is used to construct an optimal EOL (End of Service) test case selection model.

[0109] First, the states, actions, agents, policies, and reward functions in reinforcement learning are defined as follows:

[0110] The four parameters P001, N001, S001, and G001 (the types of each interface of the minimum circuit) in the two-dimensional vector dataset R generated in step S1 are defined as the State in the reinforcement learning model, and all the States constitute the state space.

[0111] From state s t (Current interface type) Jump to s t+1 The abstract action of (the next interface type) is defined as an Action in the reinforcement learning model. All Actions constitute the action space. In this embodiment, the first Action is a jump from state P001 to state N001, the second Action is a jump from state N001 to state S001, and the third Action is a jump from state S001 to state G001. Therefore, the action space in this embodiment has only three actions.

[0112] Then, in step S3, the Q-Learning algorithm is used to generate the highest-priority EOL test program based on the test case selection model. The test program is executed, test results are collected, reward values ​​are calculated according to the reward function, and the intelligent testing system is updated. The Agent is the test vector set R generated in step S1.

[0113] S3: This embodiment constructs a Q-table type agent based on the Q-Learning algorithm model. This agent is a 6*4 dimensional two-dimensional test vector. It is initialized using the initial data constructed in step S1 through an initialization update algorithm. Then, the Q-table is updated according to the model update algorithm, continuously generating optimal EOL (End-of-Life) test cases. For the Q-table initialization part, the Q-table is initialized according to a pre-designed algorithm. First, a set of original experimental data generated in step S1 is used as the experimental data R{(P001,R001,N001,N001,C001,G001) T ,(0,0,N001,N001,R003,G001) T ,(0,0,R002,0,0,0) T (0,0,S001,0,0,0) T Then iterate through the adjacent vector data U = {(P001,R001,N001,N001,C001,G001) in the original data. T ,(0,0,N001,N001,R003,G001) T};V={(0,0,N001,N001,R003,G001) T ,(0,0,R002,0,0,0) T};W={(0,0,R002,0,0,0) T (0,0,S001,0,0,0) T}

[0114] Determine whether the adjacent data U, V, W are in the state-action space (s, a). Since the state-action space is relatively small in this embodiment, and the adjacent data U, V, W are not in the state-action space (s, a), add U, V, W to the state-action space (s, a), set the Q value to 1, and then complete the Q-table initialization.

[0115] Figure 6 The diagram shows a Q-table of a reinforcement learning-based method for generating an EOL test program for a vehicle domain controller, as described in Embodiment 1 of the present invention.

[0116] Table 4 shows the initial Q-table of the Q-table graph of a reinforcement learning-based vehicle domain controller EOL test program generation method in Embodiment 1 of the present invention.

[0117] Table 4

[0118] Action 1 (N001) Action 2 (P001) Action 3 (G001) Action 4 (S001) State 1 (N001) 0.03 0.02 0.00 0.04 State 2 (P001) 0.00 0.00 0.02 0.01 State 3 (G001) 0.00 0.00 0.00 0.00 State 4 (S001) 0.01 0.00 0.00 0.00

[0119] S4: The test vector set generated in S1 is used as the Agent of the EOL offline test case selection model. A test case is executed on the circuit model under test. After execution, a test result is generated. Then, the reward function will feed back a reward value to the Agent based on the test result. The Agent decides the next action to be executed based on the action executed and the reward value obtained. This process is repeated until all test cases are executed. Then, the optimal test case is selected according to the established evaluation criteria. Figure 7 The flowchart illustrating the optimal test case selection process of a reinforcement learning-based vehicle domain controller EOL test program generation method in Embodiment 1 of the present invention is shown.

[0120] Table 5 shows the intermediate state of the final test case set generated by the reinforcement learning-based vehicle domain controller EOL test program generation method in Embodiment 1 of the present invention (the final test cases can be saved as JSON files).

[0121] Table 5

[0122] Step 1 Step 2 Step 3 Step 4 Test1 N001 P001 G001 S001 Test2 N001 G001 S001 P001 ... ... ... ... ... Test n P001 G001 S001 N001

[0123] Table 6 shows the final set of test cases generated by the reinforcement learning-based EOL test program generation method for vehicle domain controllers in Embodiment 1 of the present invention (the final test cases are saved in JSON file format).

[0124] Table 6

[0125]

[0126] Table 7 shows the result metrics after the execution of the final generated test cases of the reinforcement learning-based vehicle domain controller EOL test program generation method in Embodiment 1 of the present invention.

[0127] Table 7

[0128]

[0129] The state transition strategy used in this embodiment is the ε-greedy strategy. The formula is:

[0130]

[0131] In this embodiment, the action with the largest q value is selected with a probability of 1-ε, and a random action is selected with a probability of ε. In this embodiment, the ε value is set to 0.3.

[0132] The formula for the reward function is:

[0133]

[0134] Where y is the total number of historical executions of test case t, and r i This represents the result of the j-th execution of test case t, indicated by 1 or 0, where 1 indicates test failure and 0 indicates test success. j represents the j-th test cycle, T is the set of test cases for the current cycle, and TS... j It is the set of test cases after priority sorting. This represents the number of test cases that failed during the current test cycle.

[0135] This embodiment uses the Monte Carlo update algorithm for model updating, with the formula: Q(s) t ,a t )←Q(s t ,a t )+θ(G t -Q(s t ,a t ))

[0136] Among them G t State s in this round t The reward value. The algorithm pseudocode is as follows:

[0137] Step 1: Given a step size parameter θ = 1, an initial ε value of ε = 0.3, and a maximum number of test case states of 4;

[0138] The initialized Q(s,a) is as follows Figure 6As shown; the maximum value of a test case state is 4; the length of a single test case state is...

[0139] The initial value of the degree counter len is len = 0; the test case count counter co = 0;

[0140] Step 2: If co = 4, proceed to Step 3; otherwise, proceed to Step 9.

[0141] Step 3: Given the initial state P001;

[0142] Step 4: Since P001 is not in a terminated state and len < 4, proceed to Step 5;

[0143] Step 5: Use the ε-greedy strategy to select action a based on the Q(s,a) value table. t : P001->N001;

[0144] Step 6: Take action a t : P001->N001, obtain the next state N001;

[0145] Step 7: Then make s t ←s t+1 Jump to Step 4;

[0146] Step 8: Calculate the reward value G for each state in the entire test case T according to the reward function formula. t And using the formula Q(s) t ,a t )←Q(s t ,a t )+θ(G t -Q(s t ,a t Update Q(s,a); after the update is complete, proceed to Step 2.

[0147] Step 9: Output Q(s,a).

[0148] Example 2

[0149] To address the aforementioned technical problems in the prior art, this invention also provides an automatic generation method for EOL test programs based on reinforcement learning for an output circuit of a vehicle front area controller. The automatic generation process of the EOL test program for this output circuit can be based on all steps of the reinforcement learning-based method for generating EOL test programs for vehicle front area controllers.

[0150] The specific steps of the reinforcement learning-based automatic generation method for EOL test programs based on a certain output circuit of a certain automotive front area controller, as well as the beneficial effects of using the EOL test program generation method provided by the embodiments of the present invention, are the same as those in Embodiment 1, and will not be repeated here.

[0151] The above description is merely a description of preferred embodiments of this application and is not intended to limit the scope of this application in any way. Any changes or modifications made by those skilled in the art based on the above-disclosed technical content should be considered as equivalent and valid embodiments and fall within the scope of protection of the technical solution of this application.

Claims

1. A method for generating an EOL test program for a vehicle domain controller based on reinforcement learning, characterized in that, Includes the following steps: S1: Obtain the original data of the components according to the component list of a certain car front area controller, and obtain the component connection attribute data according to the connection method parameters between each component in the schematic diagram. Process the original circuit and component connection attributes to form the minimum circuit combination data. S2: Construct an optimal EOL (Electronic Online Test Case) selection model on the two-dimensional vector dataset obtained in step S1; S3: Based on the S2 model, construct a Q-table type agent using the Q-Learning algorithm. Initialize the agent using the minimum circuit combination data and update it using the initialization update algorithm. Update the Q-table using the model update algorithm. S4: Execute test cases on the circuit model under test. Generate high-priority test cases based on the reward function and state transition strategy. Select the optimal test case according to the established evaluation criteria. Step S2 is as follows: A reinforcement learning model is used to construct an optimal EOL (Extreme Online Time) test case selection model. The State, Action, Agent, Policy, and Reward function in reinforcement learning are defined as follows: State: The vector group generated in the data processing part of step S1 { } is of n*m dimensionality, where, The non-zero parameters in the first row, first column, and nth row, mth column are defined as the State in the reinforcement learning model. All the States constitute the state space, and this parameter represents the type of each interface of the minimum circuit. Action: From the current interface type of the current state Jump to the next interface type An abstract action is defined as an Action in a reinforcement learning model; all Actions constitute the action space. Agent: The two-dimensional test vector set generated in step S1; Policy: ε-greedy policy in reinforcement learning; Reward function: The formula is as follows: R(t) + Where y is the total number of historical executions of test case t. This represents the test result of the i-th execution of test case t, indicated by 1 or 0, where 1 indicates test failure and 0 indicates test success; j represents the j-th test cycle, and T is the set of test cases for the current cycle. It is the set of test cases after priority sorting. This represents the number of test cases that failed to execute during the current test cycle. Step S3 is as follows: After the agent generates the test program, the environment executes the test program and collects the test results. The reward value is calculated based on the reward function and provided to the agent. The agent updates itself based on the reward value. After updating the agent, the agent generates the next round of test programs. The process iterates until the termination condition is met. The agent update process is as follows: The model update algorithm used is the Monte Carlo update algorithm: Q( , ) ← Q( , )+θ( -Q( , )) in State during this round The reward value; The pseudocode for the Monte Carlo update algorithm is as follows: Step 1: Given the step size parameter θ, the initial ε value, and the maximum number of test case states L; Initialized Q(s, a); maximum value of test case state is maxL; single test case state length counter len; test case count counter co=0; Step 2: If co = L, proceed to Step 3; otherwise, proceed to Step 9. Step 3: Given the initial state ; Step4: If it is not in the termination state and len < maxL, jump to Step5; otherwise, co = co + 1, obtain the test case T, and jump to Step8; Step 5: Use the ε-greedy strategy to select actions based on the Q(s, a) value table. ; Step 6: Take Action To obtain the next state +1; Step 7: Make ← Jump to Step 4; Step 8: Calculate the reward value for each state in the entire test case T based on the reward function formula R(t). And using formula Q( , ) ← Q( , )+θ( -Q( , Update Q(s, a); after the update is complete, proceed to Step 2. Step 9: Output Q(s, a); The terminating state is when all the sub-states of state S have been transitioned to. For example, if S has four sub-states, S0, S1, S2, and S3, the terminating state is reached when all four sub-states of S have been transitioned to. The state transition strategy is an ε-greedy strategy, expressed by the following formula: Among them, the action with a large q value is selected with a probability of 1-ε, and a random action is selected with a probability of ε.

2. The method for generating an EOL test program for a vehicle domain controller based on reinforcement learning according to claim 1, characterized in that, Step S1 is as follows: Obtaining raw data: Raw data is a collection of electronic component information from the schematic diagram of the vehicle domain controller. ,in, Represented as ( , , ),in, This refers to the name of the component. This refers to the net name on the left side of the component. This refers to the net name on the right side of the component; Data extraction: Based on the original data, COMs with the same net name are grouped into set A{ , , ,... }, where w is the number of COMs with the same net name; Data removal and vector transpose: Remove COMs in A based on data with the same net, and then transpose the vectors to generate a new vector set B; Data combination and elimination: After data extraction and data removal, B is combined to generate C. Grouping: After the above series of data processing steps, the original dataset X is divided into many subsets { These subsets Each represents a minimum circuit model data; Obtain the experimental dataset: sub-dataset { This constitutes the experimental dataset for this study.

Images