Vehicle-mounted CAN bus intelligent fuzzy test method and system
By collecting and preprocessing CAN bus messages in real time, building a test case evaluation model and using large models to generate test cases, the problem of low testing efficiency in the existing technology is solved and efficient vulnerability detection is achieved.
Patent Information
- Application Number
- CN202510585400.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2025-07-18
AI Technical Summary
The existing fuzzy testing methods of on-board CAN bus protocol rely on known protocols or interface specifications, resulting in low testing efficiency and ineffective detection of vulnerabilities in complex on-board environments.
By collecting and preprocessing CAN bus messages in real time, building a test case evaluation model for on-board CAN bus messages is built, using the big model to generate test cases, and fuzzing is performed based on the generated use cases, and finally updating the evaluation model to reduce dependence on experience and protocol knowledge.
It realizes intelligent generation of test cases, improves testing efficiency and vulnerability detection effect, and reduces dependence on experience and protocol knowledge.
Smart Images

Figure CN120342926A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of intelligent networked vehicle cybersecurity, and particularly relates to an intelligent fuzz testing method and system for in-vehicle CAN bus. Background Art
[0002] With the development of intelligent networked vehicles, the originally relatively closed and independent in-vehicle network has been exposed to great information security threats. The higher the degree of vehicle networking, the more attack interfaces the vehicle exposes to the outside, and the more attack paths there will be. The electronic control units (ECUs) in the vehicle are used to measure and control the vehicle, and most ECUs communicate with each other through the CAN bus. Therefore, the communication security of the CAN bus directly affects the safety of the driver. Due to the vulnerability of the inherent security mechanism of the in-vehicle CAN bus, vehicle information security faces greater challenges.
[0003] As a security testing method for CAN bus communication, fuzz testing technology monitors the abnormal state of the network and discovers network security vulnerabilities by sending random or mutated unexpected message data to the in-vehicle CAN communication network. However, most of the existing fuzz testing methods for in-vehicle CAN bus protocols are modeled based on known protocols or interface specifications to generate test cases. Therefore, how to reduce the dependence on protocol knowledge and experience and improve testing efficiency is an urgent problem to be solved by those skilled in the art. Summary of the Invention
[0004] In order to make up for the deficiencies of the prior art, the purpose of the present invention is to provide an intelligent fuzz testing method and system for in-vehicle CAN bus, so as to effectively detect CAN bus vulnerabilities in a complex in-vehicle environment.
[0005] The technical effects achieved by the present invention can be realized through the following specific technical solutions:
[0006] On the one hand, the present invention provides an intelligent fuzz testing method for in-vehicle CAN bus, including the following steps:
[0007] S1. Real-time collection and preprocessing of in-vehicle CAN bus messages, and standardizing the CAN bus message fields;
[0008] S2. Constructing an evaluation model for in-vehicle CAN bus message test cases;
[0009] S3. Generating in-vehicle CAN bus message test cases based on a large model;
[0010] S4. Conducting in-vehicle CAN bus fuzz testing based on the generated test cases;
[0011] S5. Update of the Evaluation Model for On-vehicle CAN Bus Message Test Cases.
[0012] Further, the specific process of step S1 is as follows:
[0013] S11. Use a CAN bus analysis tool to collect CAN messages during vehicle operation, obtain all CAN messages within the t time period, and form an on-vehicle CAN bus data packet U, which includes a timestamp (TimeStamp), a message identifier (ID), a message length (DataLen), and a message field (Data);
[0014] S12. Perform standardization processing on the Data field, pad the insufficient low bits with 0s to ensure that the Data field is in a 16-bit format, and separate it in groups of two bits with spaces;
[0015] S13. Use a non-zero nibble comparison method to compare the messages in the on-vehicle CAN bus data packet U with the messages N for which control information has been analyzed, and obtain the requirement coverage rate M of the data packet U u :
[0016]
[0017] Among them, |N| represents the total number of messages N, |U| represents the number of test cases, K j represents the number of non-zero nibbles in the j-th message for which control information has been analyzed, and δ (i,j,k) represents an indicator function, which is 0 when the k-th non-zero nibble of the i-th test case does not match the k-th non-zero nibble of the j-th message for which control information has been analyzed, and is 1 when all K j non-zero nibbles are the same.
[0018] Further, the specific process of step S2 is as follows:
[0019] S21. Initialize the Q-Table, where the Q-Table includes a state space S, an action space A, and a reward R;
[0020] S22. Initialize the state space S, divide the state space S based on the requirement coverage rate, and each requirement coverage rate range corresponds to a state S i , and the state space division formula is as follows:
[0021] S = {S1, S2,..., S k}
[0022]
[0023] Among them, k represents the number of divisions of the state space, and M maxRepresents the maximum value of the demand coverage rate. The status index i of the data packet is determined by its demand coverage rate M, and the calculation formula is as follows:
[0024]
[0025] S23. Initialize the action space A. The action space A is composed of mutation behaviors, and each mutation behavior corresponds to an action A i ;
[0026] S24. Initialize the reward R. The initial reward value R is all 0.
[0027] Furthermore, the step S3 includes the following steps:
[0028] S31. Randomly select a state S i , input it into the in-vehicle system using a CAN bus analysis tool, and capture the CAN data packet T at the time of input for calculating the demand coverage rate M of the state S i ; i ;
[0029] S32. Select the action A using the ε-greedy algorithm i . When the randomly generated value ε' is greater than the threshold ε, select the action A with the maximum return i , otherwise randomly select the action A i . The calculation formula is as follows:
[0030]
[0031] Among them, ε represents the probability of randomly selecting an action; argmax(a) represents the action with the maximum return value, and a represents the selected action;
[0032] S33. Call the large model API and input the state S i and the action A i into the large model; The prompt words are as follows:
[0033] Input requirements: Please generate a new test case according to the data packet of the state S i and the mutation rule of the action A i ;
[0034] Specific prompt: Require the large model to mutate according to the data packet of the state S i using the mutation rule of the action A i to generate a test case L;
[0035] Output format: The generated test case L should be in the same format as S i , including the time stamp (TimeStamp), message identifier (ID), message length (DataLen), and message field (Data);
[0036] S34. The large model generates test case L according to the prompt words. After generation, format verification of test case L is performed to ensure that the formats of the following fields are correct:
[0037] Time Stamp: An integer in milliseconds;
[0038] Message Identifier (ID): A 3-digit hexadecimal number (e.g., 0xXXX);
[0039] Message Length (DataLen): An integer of 1 byte, not exceeding 8;
[0040] Message Field (Data): A 16-digit hexadecimal number, with each two digits grouped and separated by a space (e.g., xx xx xx xx xxxx xx xx).
[0041] Furthermore, step S4 includes the following steps:
[0042] S41. Use a CAN bus analysis tool to input test case L into the in-vehicle system, capture the CAN data packet T' during input, and calculate the requirement coverage rate M i ';
[0043] S42. Analyze the in-vehicle system operation status and data packet T', detect whether there are abnormal behaviors (such as restart, crash, etc.). If a problem is found, segment the data and send only a part each time until the attack data is located, and perform field analysis on the attack data to check whether the data value of each field exceeds the normal range, the data type does not conform to the regulation, the data content is meaningless or destructive.
[0044] Furthermore, step S5 includes the following steps:
[0045] S51. If a message of new control information is found, add the message to the message N of the analyzed control information; if a new problem is found, record the new problem;
[0046] S52. Calculate the reward value R according to the reward rule, calculate the reward value R according to the change in requirement coverage rate and the discovery of vulnerabilities. The formula for the reward value R is as follows:
[0047] R = w1(M' i - M i ) + w2N pro + w3N new_pro + w4N new_func
[0048] Where N pro represents the number of discovered problems, N new_pro represents the number of newly discovered problems, Nnew_func The number of packets representing newly discovered control information, w1 represents the weight of the change in demand coverage rate, and w2, w3, and w4 respectively correspond to the weights of rewards;
[0049] S53. Update the Q-Table, and add the newly generated test case L to the corresponding state set S according to the demand coverage rate. The Q-Table update formula is as follows:
[0050] Q(s,a)←Q(s,a)+α[r+γmaxQ(s′,a')-Q(s,a)]
[0051] Among them, α is the learning rate, which controls the influence degree of new information on the old Q value; γ is the discount factor, which represents the importance of future rewards; r represents the reward of s, and s' and a' respectively represent the new state and the new action.
[0052] On the other hand, the present invention also provides an intelligent fuzzy testing system for in-vehicle CAN bus, which is used to execute the above-mentioned intelligent fuzzy testing method for in-vehicle CAN bus, including a packet acquisition unit, a packet mutation unit, a model construction and update unit, and a detection unit. Among them,
[0053] The packet acquisition unit is used to collect CAN bus packets, and generate test cases through the packet mutation unit for the collected CAN bus packets;
[0054] The model construction and update unit is used to construct an evaluation model for in-vehicle CAN bus packet test cases and update it in combination with the results obtained by the detection unit;
[0055] The detection unit sends the test cases into the vehicle-mounted system through a CAN bus analysis tool, and updates the evaluation model for in-vehicle CAN bus packet test cases according to the reaction of the vehicle-mounted system.
[0056] Compared with the prior art, the present invention has the following advantages:
[0057] The intelligent fuzzy testing method for in-vehicle CAN bus proposed by the present invention realizes intelligent generation of test cases, improves test efficiency, enhances the vulnerability detection effect, reduces the dependence on experience and protocol knowledge, and improves test efficiency by collecting and preprocessing in-vehicle CAN bus packets in real time, constructing an evaluation model for in-vehicle CAN bus packet test cases, generating in-vehicle CAN bus packet test cases based on a large model, performing fuzzy testing on the in-vehicle CAN bus based on the generated test cases, and finally updating the evaluation model for in-vehicle CAN bus packet test cases. Description of the Drawings
[0058] Figure 1 It is a flow chart of the intelligent fuzzy testing method for in-vehicle CAN bus of the present invention;
[0059] Figure 2 This is the flowchart for evaluating test cases of in-vehicle CAN bus messages based on Q-Learning for the present invention. Detailed implementation manners
[0060] To make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0061] Embodiment 1
[0062] As Figure 1 and Figure 2 shown, this embodiment provides an intelligent fuzz testing method for in-vehicle CAN bus, and the specific content includes:
[0063] (1) Real-time collection and preprocessing of in-vehicle CAN bus messages, and standardizing the CAN bus message fields.
[0064] The specific process of this step is as follows:
[0065] ① Use a CAN bus analysis tool to collect CAN messages during vehicle operation, obtain all CAN messages within one minute, and form a data packet U, which includes a timestamp (TimeStamp), a message identifier (ID), a message length (DataLen), and a message field (Data).
[0066] For example: Use the CANalyst-II tool to capture CAN bus messages. The CAN messages are as follows:
[0067]
[0068] ② Standardize the Data field. Pad the insufficient lower bits with 0 to ensure that the Data field is in a 16-bit format, and separate it into groups of two bits with spaces (e.g., xx xx xx xx xx xx xx xx).
[0069] For example:
[0070]
[0071] ③ Compare the messages in the data packet U with the messages N whose control information has been analyzed (such as the control of the right rear door closing, the control of the multimedia volume, and the control of the windshield wiper at the first gear, etc.) in a non-zero half-byte comparison manner to obtain the requirement coverage rate M of the data packet U u :
[0072]
[0073] Wherein, |N| represents the total number of messages N, |U| represents the number of test cases, and K j represents the number of non-zero half-bytes in the j-th message where control information has been analyzed, and δ (i,j,k) represents an indicator function that is 0 when the k-th non-zero half-byte of the i-th test case does not match the k-th non-zero half-byte of the j-th message where control information has been analyzed, and is 1 when all K j non-zero half-bytes are the same.
[0074] For example: Data packet U: 10674275 0x1C2 8 BB BB BB BB BB BB BB BB, messages N where control information has been analyzed: 1C2 B0 00 00 00 00 00 00 00, 1C2 0B 00 00 00 00 00 00, 133 00 B100 00 00 00 00 00. Since the messages in U have the same controlled ID as these two messages, and the first and second bits of the Data field of the messages in U are the same as the first and second bits of the Data fields of the two messages in N respectively, it can be concluded that U covers 2 functional requirements, and the functional requirement coverage rate is
[0075] (2) Construction of an evaluation model for in-vehicle CAN bus message test cases.
[0076] The specific process of this step is as follows:
[0077] ① Initialize the Q-Table, where the Q-Table includes the state space S, the action space A, and the reward R.
[0078] ② Initialize the state space S, divide the state space S based on the requirement coverage rate, and each requirement coverage rate range corresponds to a state S i , and the state space division formula is as follows:
[0079] S = {S1, S2, …, S k}
[0080]
[0081] max represents the maximum value of the requirement coverage rate. The state index i of the data packet is determined by its requirement coverage rate M, and the calculation formula is as follows:
[0082]
[0083] For example: if k is 5, then the state space S = {(0, 0.2), (0.2, 0.4), (0.4, 0.6), (0.6, 0.8), (0.8, 1.0)}.
[0084] ③ Initialize the action space A. The action space A consists of mutation behaviors, and each mutation behavior corresponds to an action A i .
[0085] ④ Initialize the reward R. The initial reward value R is 0 for all.
[0086] (3) Generation of on-vehicle CAN bus message test cases based on large models.
[0087] The specific process of this step is as follows:
[0088] ① Randomly select a state S i , input it into the in-vehicle system using a CAN bus analysis tool, and capture the CAN data packet T at the time of input for calculating the requirement coverage rate M i of the state S i .
[0089] ② Select the action A i adopted by the ε-greedy algorithm. When the randomly generated value ε' is greater than the threshold ε, select the action A with the maximum return i , otherwise randomly select the action A i . The calculation formula is as follows:
[0090]
[0091] where ε represents the probability of randomly selecting an action; argmax(a) represents the action with the maximum return value, and a represents the selected action.
[0092] ③ Call the large model API and input the state S i and the action A i into the large model.
[0093] The prompt is as follows:
[0094] Input requirement: Please generate new test cases according to the data packet of the state S i and the mutation rule of the action A i ;
[0095] Specific prompt: Require the large model to mutate the data packet of the state S i using the mutation rule of the action A i to generate the test case L;
[0096] Output format: The generated test case L should be the same as S iThe format is the same, including the time stamp (TimeStamp), message identifier (ID), message length (DataLen), and message field (Data);
[0097] ④ The large model generates test case L according to the prompt words. After generation, format verification is performed on test case L to ensure that the formats of the following fields are correct:
[0098] Time stamp (TimeStamp): An integer in milliseconds;
[0099] Message identifier (ID): A 3-digit hexadecimal number (e.g., 0xXXX);
[0100] Message length (DataLen): An integer of 1 byte, not exceeding 8;
[0101] Message field (Data): A 16-digit hexadecimal number, with each two digits separated by a space (e.g., xx xx xx xx xxxx xx xx).
[0102] For example: ε' is 0.1 which is less than 0.15, randomly select an action bit to flip: randomly select one bit to flip. For data packet U: 10665655 0x000001C2 8 0B 00 00 00 00 00 8F 70, after flipping one bit, the obtained test case L is: 10665655 0x000001C2 8 04 00 00 00 00 00 8F 70
[0103] (4) Vehicle-mounted CAN bus fuzz testing based on generated test cases.
[0104] The specific process of this step is as follows:
[0105] ① Use a CAN bus analysis tool to input test case L into the in-vehicle system, capture the CAN data packet T' during input, and calculate the requirement coverage rate M i '.
[0106] For example: For test case L: 10674275 0x1C2 8 04 00 00 00 00 00 00 00, the analyzed message N of the control information is: 1C2 B0 00 00 00 00 00 00 00, 1C2 0B 00 00 00 00 00 00 00, 133 00 B100 00 00 00 00 00. Since the ID of the message in L is the same as that of these two messages, but the sum of the Data fields of the message in L does not match the Data fields of the two messages in N, it can be concluded that L covers 0 functional requirements, and the functional requirement coverage rate is 0.
[0107] ② Analyze the operating state of the in-vehicle computer and data packet T', and detect whether there are abnormal behaviors (such as restart, crash, etc.). If a problem is found, segment the data and send only a part each time until the attack data is located, and perform field analysis on the attack data to check whether the data value of each field exceeds the normal range, the data type does not conform to the regulations, the data content is meaningless or destructive.
[0108] (5) Update the evaluation model of in-vehicle CAN bus message test cases.
[0109] The specific process of this step is as follows:
[0110] ① If a message with new control information is found, add the message to the message N of the analyzed control information; if a new problem is found, record the new problem.
[0111] ② Calculate the reward value R according to the reward rules, and calculate the reward value R according to the change in requirement coverage rate and the discovery of vulnerabilities. The calculation formula is as follows:
[0112] R = w1(M' i - M i ) + w2N pro + w3N new_pro + w4N new_func
[0113] Among them, N pro represents the number of problems found, N new_pro represents the number of newly discovered problems, N new_func represents the number of messages with newly discovered control information, w1 represents the weight of the change in requirement coverage rate, and w2, w3, w4 respectively correspond to the weights of rewards.
[0114] For example: M' i is 0, M i is N pro is 0, N new_pro is 0, N new_func is 0, and w1, w2, w3, w4 are 5, 8, 10, 3 respectively. Calculate that R is
[0115] ③ Update the Q-Table, and add the newly generated test case L to the corresponding state set S according to the requirement coverage rate. The Q-Table update formula is as follows:
[0116] Q(s,a) ← Q(s,a) + α[r + γmaxQ(s',a') - Q(s,a)]
[0117] Among them, α is the learning rate, which controls the influence degree of new information on the old Q value; γ is the discount factor, which represents the importance of future rewards; r represents the reward of s, and s' and a' respectively represent the new state and the new action.
[0118] Embodiment 2
[0119] This embodiment provides an in-vehicle CAN bus intelligent fuzz testing system for implementing the in-vehicle CAN bus intelligent fuzz testing method described in Embodiment 1, including a message acquisition unit, a message mutation unit, a model construction and update unit, and a detection unit. Among them, the message acquisition unit is used to acquire CAN bus messages and generate test cases through the message mutation unit for the acquired CAN bus messages; the model construction and update unit is used to construct an in-vehicle CAN bus message test case evaluation model and update it in combination with the results obtained by the detection unit; the detection unit sends the test cases into the vehicle system through a CAN bus analysis tool and updates the in-vehicle CAN bus message test case evaluation model according to the reaction of the vehicle system.
[0120] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. An intelligent fuzzy testing method for in-vehicle CAN bus, characterized in that It includes the following steps: S1. Real-time collection and preprocessing of in-vehicle CAN bus messages, and standardizing the CAN bus message fields; S2. Construction of an evaluation model for in-vehicle CAN bus message test cases; S3. Generation of in-vehicle CAN bus message test cases based on a large model; S4. Fuzz testing of in-vehicle CAN bus messages based on the generated test cases; S5. Update of the evaluation model for in-vehicle CAN bus message test cases.
2. The intelligent fuzzy testing method for in-vehicle CAN bus according to claim 1, characterized in that The specific process of step S1 is as follows: S11. Use a CAN bus analysis tool to collect CAN messages during vehicle operation, obtain all CAN messages within the t time period, and form an in-vehicle CAN bus data packet U, which includes a timestamp, a message identifier, a message length, and message fields; S12. Standardize the Data field, pad the insufficient lower bits with 0s to ensure that the Data field is in a 16-bit format, and separate it in groups of two digits with spaces; S13. Compare the message of in-vehicle CAN bus data packet U with the message N with analyzed control information in a non-zero nibble comparison manner to obtain the requirement coverage rate M of data packet U u : Among them, |N| represents the total number of messages N, |U| represents the number of test cases, and K j represents the number of non-zero nibbles in the j-th message for which control information has been analyzed, and δ (i,j,k) represents an indicator function that is 0 when the k-th non-zero nibble of the i-th test case does not match the k-th non-zero nibble of the j-th message for which control information has been analyzed, and is 1 when all K j non-zero nibbles are the same.
3. The intelligent fuzzy testing method for in-vehicle CAN bus according to claim 1, characterized in that, The specific process of step S2 is as follows: S21. Initialize the Q-Table, where the Q-Table includes a state space S, an action space A, and a reward R; S22. Initialize the state space S and partition the state space S based on the requirement coverage rate, where each requirement coverage rate range corresponds to a state S i , and the state space partitioning formula is as follows: S = {S1, S2, …, S k} where k represents the number of partitions of the state space, and M max represents the maximum value of the demand coverage rate. The state index i of the data packet is determined by its demand coverage rate M, and the calculation formula is as follows: S23. Initialize the action space A. The action space A is composed of mutation behaviors, and each mutation behavior corresponds to an action A i ; S24. Initialize the reward R, and the initial reward value R is 0 for all.
4. The intelligent fuzzy testing method for in-vehicle CAN bus according to claim 1, characterized in that, Step S3 includes the following steps: S31. Randomly select a state S i , input it into the in-vehicle system using a CAN bus analysis tool, and capture the CAN data packet T during input for calculating the requirement coverage rate M i of state S i ; Action A selected using the ε-greedy algorithm i , when the randomly generated value ε' is greater than the threshold ε, select the action A with the maximum reward i , otherwise randomly select action A i , and the calculation formula is as follows: Among them, ε represents the probability of randomly selecting an action; argmax(a) represents the action with the maximum return value, and a represents the selected action; S33. Call the large model API and input the status S i and the action A i into the large model. The prompt is as follows: Input requirements: Please generate new test cases according to the data packets of status S i and action A i 's mutation rules; Specific hint: Require the large model to generate test case L according to the state S i Mutate the data packet using action A i according to the mutation rule, and generate test case L; Output format: The generated test case L should be the same as S i in format, including the timestamp, message identifier, message length, and message fields; S34. The large model generates a test case L according to the prompt words. After generation, perform format verification on the test case L to ensure that the formats of the following fields are correct: Timestamp: An integer in milliseconds; Message identifier: A 3-digit hexadecimal number; Message length: An integer of 1 byte, not exceeding 8; Message field: A 16-digit hexadecimal number, separated in groups of two digits with spaces.
5. The intelligent fuzzy testing method for in-vehicle CAN bus according to claim 1, wherein Step S4 includes the following steps: S41. Use a CAN bus analysis tool to input test case L into the in-vehicle system, capture the CAN data packet T' at the time of input, and calculate the requirement coverage rate M i '; S42. Analyze the operating state of the in-vehicle unit and the data packet T', detect whether there are abnormal behaviors. If problems are found, segment the data and send only a part each time until the attack data is located, and perform field analysis on the attack data to check whether the data values of each field exceed the normal range, the data types do not conform to the regulations, the data content is meaningless or destructive.
6. The intelligent fuzzy testing method for in-vehicle CAN bus according to claim 1, characterized in that Step S5 includes the following steps: S51. If a message with new control information is found, add this message to the message N of the analyzed control information; if a newly emerged problem is found, record the newly emerged problem; S52. Calculate the reward value R according to the reward rules, calculate the reward value R based on the change in requirement coverage and the discovery of vulnerabilities. The formula for calculating the reward value R is as follows: R = w1(M′ i - M i ) + w2N pro + w3N new_pro + w4N new_func Among them, N pro represents the number of problems found, and N new_pro represents the number of newly discovered problems, and N new_func represents the number of packets of newly discovered control information. w1 represents the weight of the change in demand coverage rate, and w2, w3, and w4 respectively correspond to the weights of rewards; S53. Update the Q-Table, and add the newly generated test case L to the corresponding state set S according to the requirement coverage. The Q-Table update formula is as follows: Q(s,a)←Q(s,a)+α[r+γmaxQ(s',a')-Q(s,a)] Among them, α is the learning rate, which controls the influence degree of new information on the old Q value; γ is the discount factor, which represents the importance of future rewards; r represents the reward of s, and s' and a' represent the new state and the new action respectively.
7. An in-vehicle CAN bus intelligent fuzz testing system for implementing an in-vehicle CAN bus intelligent fuzz testing method according to any one of claims 1-6, characterized in that, It includes a message collection unit, a message mutation unit, a model construction and update unit, and a detection unit. Among them, The message acquisition unit is used to acquire CAN bus messages and generate test cases from the acquired CAN bus messages through the message mutation unit; The model construction and update unit is used to construct an evaluation model for on-vehicle CAN bus message test cases and update it in combination with the results obtained by the detection unit; The detection unit sends the test cases into the in-vehicle system through a CAN bus analysis tool and updates the evaluation model for on-vehicle CAN bus message test cases according to the reaction of the in-vehicle system.