Automobile data testing method and system based on large model and knowledge graph

Abstract

The application discloses a kind of automobile data test method and system based on large model and knowledge graph.The method constructs and maintains knowledge graph, generates structured field context based on graph subgraph retrieval and reasoning;Context and test requirement are input into large language model to generate structured test case;The effectiveness of the case is pre-evaluated and coverage analysis is mapped to the graph;Based on graph topology and historical data, multi-dimensional priority is calculated, and a sorted recommendation list is obtained;Execute the use case in the recommended list, and dynamically adjust the execution sequence and parameters according to real-time feedback using a hybrid framework of rule engine and reinforcement learning model;Collect result data to reconstruct attack path or extract knowledge triples or labels for failure root cause diagnosis, and update the knowledge graph.The application realizes intelligent collaboration and knowledge automatic closed loop in the whole test process, and improves the accuracy of automobile data security testing.

Description

Technical Field

[0001] This invention belongs to the technical field of automotive data testing methods, specifically relating to an automotive data testing method and system based on large models and knowledge graphs. Background Technology

[0002] With the centralization and networking of the electronic and electrical architecture of intelligent connected vehicles, vehicles integrate a large number of sensors, computing units, and peripheral interfaces, making the data security threats they face increasingly insidious and complex. As a crucial link in ensuring vehicle safety, automotive data security testing is facing challenges in terms of testing efficiency, coverage depth, and accuracy.

[0003] Currently, similar automotive data security testing technologies mainly focus on point-by-point optimization of specific stages of the testing process. Their practical application status and main shortcomings are as follows: Existing technologies often achieve centralized storage and task scheduling of test resources, test cases, and scripts by building cloud-based or local test management platforms. However, the construction and updating of test case libraries on such platforms heavily rely on manual import and maintenance. The retrieval, verification, and execution of test cases are fragmented, and there is a lack of data linkage between tools, resulting in functional silos. Retrieved test cases are executed without being evaluated for effectiveness and coverage against the current testing objectives, leading to blind testing that is ill-suited to rapidly evolving automotive safety threats.

[0004] Some existing technologies attempt to use large language models to learn known attack patterns to generate specific test cases, or supplement them with static vulnerability databases. However, the generated test cases are often one-off, not deeply integrated with the enterprise's test assets and historical execution data, and lack a tight logical connection with the specific vehicle architecture context. Furthermore, vehicle architecture knowledge, vulnerability information, and historical test results are isolated from each other, failing to form a network of connections that can be understood and reasoned about by machines.

[0005] Existing test execution processes largely rely on manual configuration and linear script replay by security experts, resulting in rigid execution strategies. When faced with complex, combined security threats, the system cannot dynamically adjust the test direction and parameters based on real-time feedback from the system under test, lacking exploratory testing capabilities. Furthermore, test results typically only produce discrete pass / fail conclusions, failing to conduct in-depth root cause analysis and knowledge extraction. The system cannot autonomously learn and evolve from historical test data, lacking the ability to predict risks and dynamically optimize test strategies.

[0006] The existing technology framework makes it difficult to achieve intelligent generation, accurate recommendation, adaptive execution, and system-level self-learning optimization of test cases, resulting in low collaborative efficiency throughout the entire test lifecycle and an inability to effectively address increasingly complex automotive data security threats. Summary of the Invention

[0007] The technical problem to be solved by this invention is to overcome the shortcomings of insufficient efficiency in existing automotive data security testing methods, thereby providing an automotive data testing method and system based on large models and knowledge graphs.

[0008] A vehicle data testing method based on large models and knowledge graphs includes the following steps: Construct and maintain a knowledge graph containing entities and relationships in automotive cybersecurity, receive test requests, perform subgraph retrieval and reasoning based on the knowledge graph, and generate structured domain context; The structured domain context and the test requirements are input into the large language model, and the prompt word template is used to guide the large language model to generate structured test cases; The test cases are subjected to a validity pre-evaluation and coverage analysis based on the knowledge graph, and the validated test cases are mapped to the knowledge graph. Based on the topology of the knowledge graph and historical execution data, the multi-dimensional priority of the test cases is calculated, and a recommended list of test cases sorted by priority is obtained according to the multi-dimensional priority. The test cases in the recommended list are executed, real-time feedback data of the system under test is collected, and the execution sequence and parameters of the test cases are dynamically adjusted based on the real-time feedback data using a hybrid decision framework that includes a rule engine and a reinforcement learning model. Collect test execution result data, reconstruct attack paths or diagnose failure root causes from the result data to extract knowledge triples or root cause labels, and update the knowledge graph based on the knowledge triples or root cause labels for subsequent test requirements.

[0009] Furthermore, the step of generating a structured domain context by performing subgraph retrieval and reasoning based on the knowledge graph includes: The test requirements are parsed to extract key entities, and the key entities are linked to the corresponding standardized entity nodes in the knowledge graph. Using the standardized entity nodes as anchor points, a multi-hop traversal is performed along the relation edges in the knowledge graph to retrieve associated component, vulnerability, attack technology and protection measure nodes, forming a knowledge subgraph; Based on the topology of the knowledge subgraph and predefined reasoning rules, a structured domain context is assembled, which includes target architecture details, known threat intelligence, attack path reasoning, and historical test references.

[0010] Furthermore, the step of using prompt word templates to guide the large language model in generating structured test cases includes: Expert roles are defined using the prompt word template, and the structured domain context provided by the knowledge graph is injected. The large language model is guided by the logical chain of attack path reasoning in the structured domain context, and a structured output format including test case identifiers, preconditions, operation sequences and expected results is specified. The large language model generates test cases that conform to the test syntax specification based on the structured domain context and the prompt word template.

[0011] Furthermore, the step of the effectiveness pre-evaluation based on the knowledge graph includes: Obtain the target object information intended to be executed by the test case, and extract a detailed subgraph of the target object from the knowledge graph, wherein the detailed subgraph includes hardware model, software version and known protection measures; The attack techniques and target components intended to be used in the test cases are analyzed. The connection paths between the attack techniques, the target components and the corresponding vulnerabilities are queried in the knowledge graph. The security mechanisms that defend against the attack techniques are also checked on the target components. The pre-evaluation confidence score is calculated based on the graph path connectivity results and the security mechanism check results. When the pre-evaluation confidence score is lower than a set threshold, the test case is marked as invalid and filtered.

[0012] Furthermore, the coverage analysis steps include: The knowledge graph extracts the security requirement nodes and attack surface nodes associated with the current test task, and maps the test cases to the attack surfaces and security requirements they cover. Calculate the difference between the set of covered nodes of the test cases and the set already covered by the current test task to identify a list of uncovered coverage blind spots; The step of feeding back the coverage blind spot list to generate test cases serves as the trigger input for generating new test cases.

[0013] Furthermore, the step of calculating the multi-dimensional priority of the test cases includes: Starting from the component node in the test target, perform an N-hop traversal of the knowledge graph to retrieve the candidate test case set; For each candidate use case in the candidate use case set, calculate the shortest semantic path length to the test target component to evaluate the correlation tightness, combine the historical detection rate and time decay factor to evaluate historical validity, evaluate the execution cost based on execution time and resource consumption, and calculate the new set of coverage nodes it can bring to evaluate coverage novelty. The candidate use cases are comprehensively scored based on the correlation tightness, historical validity, execution cost, and coverage novelty, using a linear weighted or learning-based ranking model. The recommendation list is then generated by sorting the candidate use cases in descending order according to the comprehensive score.

[0014] Furthermore, the step of dynamically adjusting the execution sequence and parameters of test cases based on the real-time feedback data using a hybrid decision-making framework that includes a rule engine and a reinforcement learning model includes: Monitor real-time feedback data during the execution process, extract feature vectors, and input them into the rule engine for condition matching; When the rule engine successfully matches, it executes the corresponding deterministic action, which includes generating a derivative test case and inserting it into the priority position of the current test queue, reducing the priority of consecutive unresponsive test cases, or triggering a resource protection mechanism. When the rule engine fails to match, the current state features are input into the trained reinforcement learning model to calculate the expected value of each optional action, select the action with the highest expected value to execute, and store the state, action, and reward sequence of this decision into the replay buffer for incremental training of the reinforcement learning model.

[0015] Furthermore, the step of reconstructing the attack path or diagnosing the root cause of failure on the result data to extract knowledge triples or root cause labels includes: When the result data indicates that a vulnerability has been successfully discovered, the protocol event sequence is extracted from the execution log, and a legal behavior baseline is established using a predefined protocol state machine model. By processing the protocol event sequence one by one to detect abnormal state transition points, and starting from the abnormal state transition points, forward backtracking is performed to find the leading message and backward tracing is performed to find the abnormal consequence sequence, which are then spliced ​​together to form a candidate attack chain. The candidate attack chains are compared and verified with the attack techniques declared in the test cases. Knowledge triples containing attack techniques, exploit relationships, and vulnerability components are extracted from the verified attack chains.

[0016] Furthermore, the step of updating the knowledge graph based on the knowledge triples or root cause labels includes: When the knowledge triple is extracted and the vulnerability node it points to does not exist in the knowledge graph, a new node is created in the knowledge graph, and an association edge is established between the test case, the attack technique and the new node, and the weight of the association edge is increased. When the result data indicates a test failure, the root cause label is classified based on log features. If the root cause label indicates that the protection is effective, the confidence attribute of the corresponding protection measure node is updated and the weight of the corresponding attack path is reduced. If the root cause label indicates that the attack path is ineffective, negative samples are recorded on the corresponding attack technology node to reduce the priority of subsequent recommendations. The updated node attributes and weight changes of the knowledge graph are fed back to the priority sorting step and the test case generation step.

[0017] A vehicle data testing system based on large models and knowledge graphs, comprising: The knowledge graph construction and management module is used to construct and maintain the knowledge graph, and to generate structured domain context based on the knowledge graph through subgraph retrieval and reasoning. The large model-driven intelligent test case generation module is used to receive the structured domain context and test requirements, and use prompt word templates to guide the large language model to generate structured test cases. The test case multidimensional verification and evaluation module is used to perform validity pre-evaluation and coverage analysis on the test cases based on the knowledge graph, and to map the verified test cases to the knowledge graph; The intelligent recommendation and priority ranking module is used to calculate the multi-dimensional priority of the test cases based on the topology of the knowledge graph and historical execution data, and obtain a recommended list of test cases ranked by priority. An adaptive test execution engine is used to execute test cases in the recommendation list. It utilizes a hybrid decision framework that includes a rule engine and a reinforcement learning model to dynamically adjust the execution sequence and parameters of test cases based on real-time feedback data. The dynamic evaluation and knowledge update module is used to collect result data to reconstruct attack paths or diagnose root causes of failures in order to extract knowledge triples or root cause labels, and update the knowledge graph based on the knowledge triples or root cause labels, so as to drive the iteration of the knowledge graph construction and management module, the large model-driven intelligent test case generation module, and the intelligent recommendation and priority ranking module.

[0018] Beneficial Effects: This invention provides a method and system for testing automotive data based on a large model and knowledge graph. By performing subgraph retrieval and reasoning within the knowledge graph, a structured domain context closely related to the testing objective is extracted. This context is then input into a large language model, providing it with clear domain knowledge and logical threads. This avoids generating unrealistic and invalid test cases, making the generated structured test cases more aligned with actual testing needs.

[0019] Before test cases are executed, a knowledge graph is used to pre-assess their effectiveness, eliminating test cases that are bound to fail due to architecture or protection mismatches. Simultaneously, coverage analysis is used to calculate security requirements and attack surface blind spots not currently covered by the tests. This avoids wasting testing resources and ensures that testing efforts are focused on the truly necessary coverage areas.

[0020] Based on the knowledge graph's topology and historical execution data, the system comprehensively calculates the priority of test cases across multiple dimensions, including relevance, historical validity, execution cost, and coverage novelty. A recommendation list is generated through quantitative scoring, replacing the traditional selection method that relies on human experience, ensuring that the highest-value test cases are executed first with limited resources.

[0021] During the execution phase, a hybrid decision-making framework combining a rule engine and a reinforcement learning model is employed. When the system under test generates real-time feedback, the rule engine can quickly execute deterministic actions such as generating derived test cases, while the reinforcement learning model handles complex state decisions. This allows testing to move beyond fixed scripts, dynamically adjusting execution sequences and parameters based on feedback, thereby increasing the probability of discovering deep vulnerabilities.

[0022] After testing, the system automatically extracts knowledge triples or root cause tags by reconstructing attack paths or diagnosing root causes of failures from the results data, and updates the node and relationship weights in the knowledge graph accordingly. This allows the system's knowledge base to automatically expand and correct as testing progresses, continuously improving the quality of handling subsequent test requirements and reducing reliance on manual maintenance. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the main method steps of the present invention; Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0025] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0026] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0027] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0028] Example 1: Reference Figure 1 As shown, this embodiment provides a vehicle data testing method based on large models and knowledge graphs, including the following steps: Step S1: Construct and maintain a knowledge graph containing automotive cybersecurity entities and relationships, receive test requirements, perform subgraph retrieval and reasoning based on the knowledge graph, and generate a structured domain context.

[0029] Specifically, the knowledge graph constructed and maintained in this step is implemented in the following way: First, a graph pattern for the automotive cybersecurity domain is defined, specifying node types including vehicle models, electronic control units (ECUs), software components, vulnerabilities, attack techniques, protective measures, test cases, and security requirements; and edge relationship types including "equipped," "existing," "exploited," "impacted," "protected," and "verified." Next, multi-source data is classified and extracted: For structured data such as CVE / NVD vulnerability databases and supplier-provided software bills of materials (SBOMs), parsing scripts are written to directly map fields in the data tables to corresponding entity nodes and their attributes; for vehicle engineering description files such as ARXML, XML parsing tools are used to extract ECU names, network interfaces, and signal interaction relationships from the vehicle's electronic and electrical architecture, creating ECU entity nodes and topological edge relationships; for semi-structured or unstructured text such as security standard documents and historical penetration test reports, pre-trained language models are used to perform named entity recognition and relationship extraction tasks. Then, entity alignment and fusion are performed, merging entities pointing to the same real-world object based on entity attribute matching or a pre-defined thesaurus. Finally, the fused entities and relationships are stored in the graph database in the form of triples. During system operation, when new vulnerability intelligence or test result data is received, the above data extraction and fusion process is automatically triggered to update the graph with new nodes and edges incrementally.

[0030] Upon receiving a test request described in natural language, the system performs subgraph retrieval and reasoning. First, the input request is parsed, and key entities are extracted using named entity recognition technology. Then, these textual entities are linked to standardized entity nodes defined in the graph. After successful linking, using these core entity nodes as anchors, a multi-hop traversal query is performed along predefined relational edges in the graph to collect all strongly related nodes and relationships, forming a knowledge subgraph. The traversal and expansion include retrieving connected operating systems, middleware, and other components and assets; retrieving associated known vulnerabilities and weaknesses; retrieving attack technology nodes; retrieving security mechanisms deployed on components and their configuration nodes; and retrieving historical test case nodes and their execution results.

[0031] Finally, based on the topology of the knowledge subgraph, predefined inference rules are used to automatically assemble descriptions of the most threatening potential attack scenarios. These predefined inference rules are specifically represented as a mapping between a sequence of walk constraints defined on the graph's edge relationships and a natural language generation template. The system executes a path search algorithm on the knowledge subgraph to find connected paths from external entry entities to core data asset entities. For example, an inference rule template might be defined as: "[External application entity] -> (exploitation) -> [Operating system privilege vulnerability entity] -> (impact) -> [Application sandbox mechanism entity] -> (cause) -> [target data asset entity]". When the inference engine finds a node chain matching this topology sequence in the subgraph, it identifies the path as a potential attack path. Subsequently, the system extracts the attribute information of each node on this path and substitutes it into the corresponding natural language template to generate an attack hypothesis description. The final formatted output includes target architecture details, known threat intelligence, attack path inference, and a structured domain context with historical test references, providing accurate domain knowledge for subsequent large-scale model generation.

[0032] Step S2: Input the structured domain context and the test requirements into the large language model, and use prompt word templates to guide the large language model to generate structured test cases.

[0033] When the test case generation module is triggered, its core task is to initiate an intelligent query to the knowledge graph to obtain the structured domain context necessary for generating test cases. After obtaining this context, it organically combines task instructions, domain knowledge, and the generation capabilities of a large model through carefully designed prompt word templates.

[0034] As a preferred embodiment of this invention, a specific example of prompt word construction is as follows: "You are a professional automotive cybersecurity penetration testing expert, specializing in designing accurate and executable security testing solutions."

[0035]

Task

[0036] [Test Objective] Target System: {System description extracted from requirements, such as: intelligent cockpit infotainment system for vehicle model X} Test focus: {Test focus extracted from requirements, such as: data isolation mechanism with third-party applications} [Domain Knowledge Context] (Provided by Knowledge Graph) Target architecture details: ECUs involved: {List of ECUs retrieved from the graph}; Operating system and middleware: {Software environment retrieved from the graph}; Related data assets: {List of data assets retrieved from the graph} Known Threat Intelligence: Related Vulnerabilities: {List of CVEs retrieved from the vulnerability graph and brief descriptions}; Associated Attack Techniques: {List of attack techniques retrieved from the vulnerability graph, such as ATT&CK Tactics / Techniques}; Existing Protective Measures: {Known Security Mechanisms retrieved from the vulnerability graph} Attack path inference: {1-2 descriptions of the most likely potential attack scenarios generated by the graph inference engine} Historical test reference: {1-2 relevant historical test case IDs retrieved from the graph and brief conclusions} [Generation Requirements] Please strictly follow the following structure to generate test cases: Test Case ID and Title: Define a unique ID and a descriptive title.

[0037] Test objective: To reiterate the specific security attributes to be verified or the types of vulnerabilities to be discovered.

[0038] Prerequisites: List the environment, tools, permissions, etc. required to perform this test.

[0039] Test Procedure: Provide a clear, orderly, and actionable sequence of steps. This includes: a. Test preparation (e.g., tool configuration, test payload construction); b. Attack execution (specific injection points, sent packets / requests, and operation sequence); c. Observation points (system logs, network traffic, and process behavior to be monitored).

[0040] Expected results: Describe the abnormal phenomena that the system should exhibit if the vulnerability exists (such as specific error codes, unexpected data access, service crashes).

[0041] Result verification method: Explain how to confirm the test was successful (e.g., which log entries to check, and which data to verify was tampered with or leaked).

[0042] Note: May include risk level, precautions, etc.

[0043] Please begin generating based on the above knowledge. The instructions clearly define the expert role of the large model, specifying its domain and style of expertise in response. The context provided by the knowledge graph is injected in modular sections, resulting in clear information presentation. The intentionally added attack path reasoning section not only provides information but also demonstrates a logical thinking paradigm, strongly guiding the large model to generate specific test steps along this logical chain and avoiding invalid generation. Strict generation requirements clearly define the components and level of detail of test cases, ensuring that the output can be directly used for subsequent automated verification and execution. Based on the above structured injection and output format constraints, the large model generates test cases that conform to test syntax specifications and closely adhere to requirements, improving the engineering level of the output.

[0044] Step S3: Perform a validity pre-evaluation and coverage analysis on the test cases based on the knowledge graph, and map the verified test cases to the knowledge graph.

[0045] Before test cases are added to the availability library or executed, they undergo automated, multi-layered filtering and evaluation, implementing a four-stage automated quality control pipeline. Only test cases that pass all four stages of verification are marked as available and injected into the knowledge graph library.

[0046] The first phase is security verification, which leverages existing technologies to ensure that the test cases themselves do not contain malicious code. Static code analysis is performed on the test case script files, using regular expressions or abstract syntax trees to parse and match known malicious code patterns. The Static Application Security Testing (SAST) tool rule set is integrated, and dynamic behavior sandbox monitoring is executed with least privileges in a fully isolated containerized environment to intercept unauthorized access attempts.

[0047] The second phase involves verifying the grammatical and logical correctness of the test cases, utilizing existing technologies to ensure that the test cases are technically parsable and executable correctly. A dedicated syntax checking sandbox is used to invoke the interpreter for syntax checks, verifying the usability of external libraries, tools, and commands. Test cases are then executed in a lightweight vehicle bus simulation environment or the target ECU's software simulator. The process is monitored for runtime errors such as exceptions, timeouts, and infinite loops, and the generated test messages or instructions are checked to ensure they conform to the basic communication protocol format.

[0048] The third stage is a knowledge graph-based effectiveness pre-assessment, which predicts whether the test case is likely to be effective in the current specific test target context. This involves obtaining information about the target object to be executed by the test case and extracting a detailed subgraph of the target from the knowledge graph, including its hardware model, software version, network interface, and known protection measures. Natural language processing (NLP) techniques are used to parse and extract the intended exploitable vulnerabilities, attack techniques, and expected impact on target components from the test case description. The knowledge graph is then queried to determine if the attack technique can exploit vulnerabilities and whether these vulnerabilities affect target components of the target vehicle model. Further queries are conducted to determine if security mechanisms capable of defending against or mitigating this attack technique have been deployed on the target component. A pre-assessment baseline score (e.g., 100 points) is set. If the path between the attack technique, vulnerability, and target component is completely connected in the knowledge graph, the baseline score is retained; if the path is not connected (e.g., the vulnerability does not affect the component version), it is directly rejected. If the knowledge graph records that the target component has a corresponding protection mechanism, points are deducted according to the confidence level of that protection mechanism. Finally, a pre-assessment confidence score is calculated. When the score is higher than a set threshold (e.g., 60 points), the use case is considered potentially valid and proceeds to the next stage; otherwise, it is marked as potentially invalid and returned, thus eliminating use cases that are bound to fail in advance and avoiding waste of resources.

[0049] The fourth phase is coverage analysis, which quantitatively evaluates the contribution of the test case to the security requirements of the current testing task. Security requirements and attack surface nodes associated with the current testing task are extracted from the knowledge graph. The test case is mapped to the attack surface it involves and the security requirements it can cover, based on the attack techniques it verifies. The system calculates which security requirements or attack surfaces previously not covered by other test cases are newly covered after the test case is executed, and identifies a list of coverage blind spots that are currently not covered by any test cases through set operations. The test case is tagged with a coverage label, and the list of coverage blind spots is fed back to the large model intelligent generation module as one of the inputs to trigger the generation of new test cases, thereby driving the self-improvement of the test design.

[0050] Step S4: Based on the topology of the knowledge graph and historical execution data, calculate the multi-dimensional priority of the test cases, and obtain a recommended list of test cases sorted by priority according to the multi-dimensional priority.

[0051] Once the test task is initiated and the test objective is set, an N-hop traversal (e.g., N=3) is performed on the knowledge graph, starting from the component nodes in the test objective, to retrieve all directly or indirectly related test cases, forming an initial candidate test case set. Four core factors are then calculated for each test case in the candidate test case set.

[0052] Factor A represents the target association tightness, calculated based on the knowledge graph's topology, using the shortest phrase semantic path length (measured by relation hops) from candidate use case nodes to each node in the test target component node set. Shorter paths indicate more direct associations and higher scores. An example formula is RS. i = 1 / (1 + average shortest path length), a score close to 1 is considered a perfect score.

[0053] Factor B represents historical effectiveness. Based on the historical detection rate and most recent execution time of each use case recorded in the historical database, it quantifies the lethality of a use case and introduces time decay to give higher weight to recently successful use cases. An example formula is HES. i =HHR i * λ^(T current - T last_success ), where λ is the decay factor (e.g., 0.95), and this score will decrease for use cases that have not been successful for a long time.

[0054] Factor C represents execution cost, converting time cost and resource consumption into a profitability score. An example formula is ICS. i =1 / (AET i + α * Resource_Weight i ), where AET_ i This is the average time consumed, Resource_Weight i It is the estimated resource consumption (such as whether special hardware is required), and α is the adjustment coefficient. Use cases with short time consumption and low resource requirements score higher.

[0055] Factor D represents coverage novelty. The system maintains the target coverage set for the current test task. It determines the set of security requirement nodes covered by candidate test cases in the knowledge graph, calculates the difference between this set and the current coverage set, and obtains the new set of coverage nodes that the test case can bring. An example formula is CNS. i = ω * |Novel_Cover i | / |Cover i | where ω is the novelty weighting coefficient. The more new coverage a use case can bring to the current task, the higher the score. If all coverage has been satisfied, the score is 0 or very low.

[0056] The final score can be calculated using a linear weighting method, i.e., Final_Score. i = w1 * RS i + w2 * HES i +w3 * ICS i + w4 * CNS iHere, w1 to w4 are configurable weight parameters, summing to 1, and can be dynamically adjusted according to the testing phase. Alternatively, a learning-based ranking model (LTR), such as LambdaMART, can be used, training the model with the above four factors and more features as feature vectors. All candidate use cases are sorted in descending order based on the comprehensive score, generating a priority recommendation list which is submitted to the execution engine to ensure that the highest-value use cases are executed first with limited resources.

[0057] Step S5: Execute the test cases in the recommended list, collect real-time feedback data of the system under test, and dynamically adjust the execution sequence and parameters of the test cases based on the real-time feedback data using a hybrid decision framework that includes a rule engine and a reinforcement learning model.

[0058] The engine receives a recommendation list and drives the testing tools to execute in real or simulated vehicle environments. Feedback data from the system under test (SUT), such as CAN / LIN / Ethernet messages, ECU logs, diagnostic responses, and network traffic, is collected in real time via the test toolchain's APIs and probes. This data is processed in real time and extracted into feature vectors (e.g., whether abnormal responses have occurred, bus load rate). These feature vectors are then input into a hybrid decision framework for dynamic decision-making.

[0059] The framework's rule engine is used for rapid response and solidification of expert knowledge. It pre-defines a series of condition-action decision rules to handle common and well-defined feedback patterns, achieving millisecond-level response and possessing the highest priority. For example, rule R1 (Success Derivation): If the current test case triggers an unprecedented exception type (such as a new DTC error code), the variant generator is immediately invoked. Based on the ID, data field, and ECU response status of the current exception message, it uses mutation algorithms such as bit flipping, boundary value replacement, or timing delay to fine-tune the message payload that triggered the exception, automatically generating 3-5 derived test cases and inserting them into the current test queue for priority execution. Rule R2 (Failure Transfer): If the same attack technique targeting the same ECU has been executed N times consecutively (e.g., N=10) without any exception response, the priority of this type of test case is automatically reduced, and the recommendation module is triggered to immediately recommend a test case targeting a different attack surface. Rule R3 (Resource Protection): If the CPU utilization of the target ECU is detected to be consistently higher than the threshold or the bus load exceeds the safety limit, the current high-intensity test is paused, and a sleep interval is inserted to prevent system crashes.

[0060] If the rule engine fails to match, the current state features are input into a trained reinforcement learning model (such as using DQN or PPO algorithms) for long-term policy optimization. The model's state represents the feature vector of the current test progress and environment, including the sequence of executed test cases, the success / failure rates of various attack techniques, real-time status indicators of the current target ECU (such as response latency and error code distribution), and the set of covered and uncovered attack surfaces. The model's actions include executing the next recommended test case, skipping a test case, initiating a round of derivative tests targeting the current successful pattern, and completely switching the test attack surface. The model's reward provides immediate feedback based on the test results; for example, discovering a new vulnerability rewards +10, triggering a known anomaly rewards +2, completing a test case without discovery rewards -0.1, and causing an abnormal test environment rewards -5. The model outputs the expected value (Q-value) of each selectable action in the current state, and the engine selects the action with the highest expected value to execute. After execution, the results are collected, the immediate reward is calculated, and the sequence of decisions (S-value) is updated. t A t , R t , S t+1 This new experiential data is stored as an experience buffer. During system idle periods or periodically, this new experiential data is used to incrementally train the reinforcement learning model, enabling it to continuously adapt to new testing objectives and environments. This hybrid framework achieves a synergy between millisecond-level rapid response and long-term policy optimization, enhancing the depth of exploratory testing.

[0061] Step S6: Collect the result data of the test execution, reconstruct the attack path or diagnose the root cause of failure on the result data to extract knowledge triples or root cause labels, and update the knowledge graph based on the knowledge triples or root cause labels for subsequent test requirements.

[0062] The engine receives standardized test event packets from the execution process, which include test case metadata, execution result tags (SUCCESS, FAILURE, ERROR), full execution logs (timestamp-aligned CAN / Ethernet message sequences, ECU diagnostic responses, system logs), and environment snapshots.

[0063] When the result label is SUCCESS, a deep analysis subprocess is initiated to reconstruct the attack path. A formalized, standardized state machine model is pre-built for the communication protocols involved in the system under test (such as the UDS diagnostic protocol, DoIP, etc.), explicitly defining all legal states (e.g., UDS's DefaultSession), legal state transition conditions, and allowed legal service requests, and storing them in a knowledge graph. Messages related to the protocol under test are filtered from the full execution log, and fields such as timestamp, source address, destination address, service ID, sub-function, data payload, and response code are extracted and arranged in chronological order to form a protocol event sequence. A virtual state machine instance is initialized, its state is set to the initial state, and then each message in the protocol event sequence is processed one by one to establish a baseline for legal behavior. For each request message, it is checked whether the current state allows the service; if not, it is recorded as a potential anomaly trigger point. For each response message, it is checked whether its response code matches the expectation; if abnormal, it is also recorded as a potential anomaly signal. When encountering illegal service requests, unexpected state changes, or protocol violations (such as incorrect data length or CRC check failure), an anomaly transition point is marked, and the timestamp of the occurrence and the message that triggered the anomaly are recorded. Starting from the anomaly transition point, a sliding window of a preset size (e.g., the first 50 messages) is traced backwards, searching for all leading messages that could potentially cause the current anomaly. Simultaneously, subsequent messages are traced backwards to observe whether unexpected behavior occurs, such as the ECU returning an unexpected error code or a large number of retransmission anomaly messages appearing on the bus. The sequence of leading messages found forwards is concatenated with the sequence of anomaly consequences traced backwards to form a complete candidate attack chain. The candidate attack chain is compared and verified against the attack techniques declared in the test cases. If the starting part matches the attack technique description, the attack chain is confirmed to be valid. Triples in the form of (attack technique, exploit relationship, vulnerability / component) are automatically extracted from the verified attack chains, for example, extracting ("diagnostic session obfuscation", "exploitation", "memory check defect in firmware F").

[0064] When the result label is FAILURE, the root cause diagnostic classifier is activated. Features are extracted from the logs, such as the response code distribution of the target ECU, the presence of protective packet loss, log records of specific security events, and the degree of matching between the packet interaction sequence in network traffic and predefined attack packet feature sequences. A rule-based classifier makes decisions: if the feature includes "firewall drop record" or "intrusion detection system alarm," it is classified as protection effective; if the feature shows "protocol stack response fully conforms to specifications" or "injection packet is silently ignored," it is classified as an invalid attack path; if the feature shows "communication timeout" and is unrelated to the network layer, it is classified as an environment mismatch. This assigns a clear root cause label to each failed use case.

[0065] Knowledge fusion and graph updates are then performed. In scenarios where a new attack pattern is successfully discovered, if the vulnerability / component in the extracted triplet points to a weakness not present in the graph, the graph management API is invoked to create a new vulnerability or attack pattern node. Related edges such as [Test Case]-(Verification)->[Attack Technique], [Attack Technique]-(Exploitation)->[New Vulnerability Node], and [New Vulnerability Node]-(Impact)->[Target ECU] are created or strengthened, increasing the weight of these edges and recording the timestamp of the successful verification as evidence. In scenarios where the failure is due to ineffective protection, the corresponding protection measure node (e.g., "Firewall Policy P") is found in the knowledge graph, its attributes are updated (confidence += increment, last_verified_time = current time), and the weight of the edge [Attack Technique]-(Blocked)->[Protection Measures] is increased, while the weight of the edge [Attack Technique]-(Exploitation)->[Vulnerability] is decreased. In scenarios where the failure is due to an invalid attack path, a negative sample containing the test context is recorded on the corresponding [Attack Technique] node. When the same technique is recommended again, its priority will be lowered based on the historical failure rate.

[0066] After knowledge updates, optimization instructions are proactively generated and pushed to relevant modules. A message is sent to the intelligent recommendation and ranking module, informing them that threat weights have been updated or the confidence level of protective measures has increased. The recommendation module receives this message in real time and recalculates the priority scores of relevant test cases, enabling dynamic adjustments to the testing strategy. Simultaneously, feedback is fed back to the large model-driven intelligent test case generation module, which uses failed test cases and their root cause labels, as well as newly discovered successful attack patterns, as high-quality learning samples. These samples are injected into the module's prompt word fine-tuning dataset or context example library, allowing the large model to reference successful patterns in subsequent generation, avoiding repeated ineffective attempts. Through this closed loop, the system possesses the autonomous evolutionary capability to continuously learn from historical tests and optimize its knowledge base and testing strategies.

[0067] Example 2: Reference Figure 2 As shown, Embodiment 2 provides an automotive data testing system based on a large model and knowledge graph. This system corresponds to the method in Embodiment 1 and includes the following functional modules: The knowledge graph construction and management module is used to build and maintain a knowledge graph containing entities and relationships in automotive cybersecurity. This module extracts entity relationships from structured data, engineering documents, and unstructured text using script mapping, XML parsing, and natural language processing models, respectively, by defining graph schemas. After entity alignment and fusion, the extracted relationships are stored in a graph database. Based on the knowledge graph, subgraph retrieval and reasoning are performed to generate a structured domain context containing target architecture details, known threat intelligence, attack path reasoning, and historical test references. This module is responsible for maintaining graph nodes, multi-hop traversal queries, and attack scenario reasoning and assembly based on subgraph topology. It generates attack hypotheses by matching predefined traversal constraint sequences using path search algorithms and filling node attributes into natural language templates.

[0068] The large-model-driven intelligent test case generation module receives the structured domain context and test requirements. It then uses prompt word templates, including role settings, structured knowledge injection, and reasoning guidance, to guide the large language model in generating structured test cases containing test case identifiers, preconditions, operation sequences, and expected results. This module uses prompt word templates to guide the large model in generating test cases that closely adhere to domain knowledge and logical chains, conforming to test syntax specifications.

[0069] The test case multidimensional verification and evaluation module is used to sequentially perform security verification, syntax and logic correctness verification, validity pre-evaluation based on the knowledge graph, and coverage analysis on newly generated or imported test cases. Security verification and syntax verification are implemented using existing technical tools. Validity pre-evaluation extracts test case intent through named entity recognition, queries the graph path connectivity and protection measure check results, and calculates a pre-evaluation confidence score based on a base score deduction mechanism for filtering. Coverage analysis identifies a list of coverage blind spots by calculating the difference between the sets of covered nodes. Finally, the verified test cases are mapped to the knowledge graph.

[0070] The intelligent recommendation and priority ranking module, based on the topology and historical execution data of the knowledge graph, performs an N-hop traversal starting from component nodes to calculate the relevance, historical validity, execution cost, and coverage novelty of candidate test case sets. It then uses a linear weighted or learning-based ranking model for comprehensive scoring to obtain a priority-ranked list of recommended test cases. This module achieves a quantitative decision-making process that maximizes risk coverage and optimizes testing efficiency.

[0071] An adaptive test execution engine executes test cases from the recommended list and collects real-time feedback data from the system under test. Utilizing a hybrid decision framework incorporating a rule engine and a reinforcement learning model, it dynamically adjusts the execution sequence and parameters of test cases based on the real-time feedback data. The rule engine performs deterministic actions such as generating derivative test cases based on bit flipping or boundary value replacement, lowering priorities, or triggering resource protection. The reinforcement learning model calculates the expected value of each optional action based on test progress and environmental state characteristics, using state, action, and reward factors, and performs incremental training using a replay buffer. This module enables dynamic adaptive adjustment of exploratory testing.

[0072] The dynamic evaluation and knowledge update module is used to collect result data for attack path reconstruction or failure root cause diagnosis to extract knowledge triples or root cause labels. For successful results, the attack chain is reconstructed and triples are extracted using anomaly transition detection and forward / backward tracing methods based on a protocol state machine model. For failed results, a rule classifier is used to output root cause labels based on log features and traffic matching degree. The knowledge graph is updated based on the knowledge triples or root cause labels, including creating new nodes, adjusting edge weights, updating protection confidence attributes, and recording negative samples. This drives the iteration of the knowledge graph construction and management module, the large model-driven intelligent test case generation module, and the intelligent recommendation and priority ranking module, enabling the system's autonomous evolution.

[0073] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A vehicle data testing method based on large models and knowledge graphs, characterized in that, Includes the following steps: Construct and maintain a knowledge graph containing entities and relationships in automotive cybersecurity, receive test requests, perform subgraph retrieval and reasoning based on the knowledge graph, and generate structured domain context; The structured domain context and the test requirements are input into the large language model, and the prompt word template is used to guide the large language model to generate structured test cases; The test cases are subjected to a validity pre-evaluation and coverage analysis based on the knowledge graph, and the validated test cases are mapped to the knowledge graph. The steps of the effectiveness pre-evaluation based on the knowledge graph include: Obtain the target object information intended to be executed by the test case, and extract a detailed subgraph of the target object from the knowledge graph, wherein the detailed subgraph includes hardware model, software version and known protection measures; The attack techniques and target components intended to be used in the test cases are analyzed. The connection paths between the attack techniques, the target components and the corresponding vulnerabilities are queried in the knowledge graph. The security mechanisms that defend against the attack techniques are also checked on the target components. The pre-evaluation confidence score is calculated based on the network connectivity results and the security mechanism check results. When the pre-evaluation confidence score is lower than a set threshold, the test case is marked as invalid and filtered. The steps of the coverage analysis include: The knowledge graph extracts the security requirement nodes and attack surface nodes associated with the current test task, and maps the test cases to the attack surfaces and security requirements they cover. Calculate the difference between the set of covered nodes of the test cases and the set already covered by the current test task to identify a list of uncovered coverage blind spots; The list of coverage blind spots is fed back to the step of generating test cases as the trigger input for generating new test cases; Based on the topology of the knowledge graph and historical execution data, the multi-dimensional priority of the test cases is calculated, and a recommended list of test cases sorted by priority is obtained according to the multi-dimensional priority. The test cases in the recommended list are executed, real-time feedback data of the system under test is collected, and the execution sequence and parameters of the test cases are dynamically adjusted based on the real-time feedback data using a hybrid decision framework that includes a rule engine and a reinforcement learning model. Collect test execution result data, reconstruct attack paths or diagnose failure root causes from the result data to extract knowledge triples or root cause labels, and update the knowledge graph based on the knowledge triples or root cause labels for subsequent test requirements.

2. The vehicle data testing method based on large models and knowledge graphs according to claim 1, characterized in that, The steps of performing subgraph retrieval and reasoning based on the knowledge graph to generate structured domain context include: The test requirements are parsed to extract key entities, and the key entities are linked to the corresponding standardized entity nodes in the knowledge graph. Using the standardized entity nodes as anchor points, a multi-hop traversal is performed along the relation edges in the knowledge graph to retrieve associated component, vulnerability, attack technology and protection measure nodes, forming a knowledge subgraph; Based on the topology of the knowledge subgraph and predefined reasoning rules, a structured domain context is assembled, which includes target architecture details, known threat intelligence, attack path reasoning, and historical test references.

3. The vehicle data testing method based on large models and knowledge graphs according to claim 1, characterized in that, The steps of using prompt word templates to guide the large language model in generating structured test cases include: Expert roles are defined using the prompt word template, and the structured domain context provided by the knowledge graph is injected. The large language model is guided by the logical chain of attack path reasoning in the structured domain context, and a structured output format including test case identifiers, preconditions, operation sequences and expected results is specified. The large language model generates test cases that conform to the test syntax specification based on the structured domain context and the prompt word template.

4. The vehicle data testing method based on large models and knowledge graphs according to claim 1, characterized in that, The steps for calculating the multi-dimensional priority of the test cases include: Starting from the component node in the test target, perform an N-hop traversal of the knowledge graph to retrieve the candidate test case set; For each candidate use case in the candidate use case set, calculate the shortest semantic path length to the test target component to evaluate the correlation tightness, combine the historical detection rate and time decay factor to evaluate historical validity, evaluate the execution cost based on execution time and resource consumption, and calculate the new set of coverage nodes it can bring to evaluate coverage novelty. The candidate use cases are comprehensively scored based on the correlation tightness, historical validity, execution cost, and coverage novelty, using a linear weighted or learning-based ranking model. The recommendation list is then generated by sorting the candidate use cases in descending order according to the comprehensive score.

5. The vehicle data testing method based on large models and knowledge graphs according to claim 1, characterized in that, The steps of dynamically adjusting the execution sequence and parameters of test cases based on the real-time feedback data using a hybrid decision framework that includes a rule engine and a reinforcement learning model include: Monitor real-time feedback data during the execution process, extract feature vectors, and input them into the rule engine for condition matching; When the rule engine successfully matches, it executes the corresponding deterministic action, which includes generating a derivative test case and inserting it into the priority position of the current test queue, reducing the priority of consecutive unresponsive test cases, or triggering a resource protection mechanism. When the rule engine fails to match, the current state features are input into the trained reinforcement learning model to calculate the expected value of each optional action, select the action with the highest expected value to execute, and store the state, action, and reward sequence of this decision into the replay buffer for incremental training of the reinforcement learning model.

6. The vehicle data testing method based on large models and knowledge graphs according to claim 1, characterized in that, The step of reconstructing attack paths or diagnosing root causes of failures in the result data to extract knowledge triples or root cause labels includes: When the result data indicates that a vulnerability has been successfully discovered, the protocol event sequence is extracted from the execution log, and a legal behavior baseline is established using a predefined protocol state machine model. By processing the protocol event sequence one by one to detect abnormal state transition points, and starting from the abnormal state transition points, forward backtracking is performed to find the leading message and backward tracing is performed to find the abnormal consequence sequence, which are then spliced ​​together to form a candidate attack chain. The candidate attack chains are compared and verified with the attack techniques declared in the test cases. Knowledge triples containing attack techniques, exploit relationships, and vulnerability components are extracted from the verified attack chains.

7. The vehicle data testing method based on large models and knowledge graphs according to claim 6, characterized in that, The step of updating the knowledge graph based on the knowledge triples or root cause labels includes: When the knowledge triple is extracted and the vulnerability node it points to does not exist in the knowledge graph, a new node is created in the knowledge graph, and an association edge is established between the test case, the attack technique and the new node, and the weight of the association edge is increased. When the result data indicates a test failure, the root cause label is classified based on log features. If the root cause label indicates that the protection is effective, the confidence attribute of the corresponding protection measure node is updated and the weight of the corresponding attack path is reduced. If the root cause label indicates that the attack path is ineffective, negative samples are recorded on the corresponding attack technology node to reduce the priority of subsequent recommendations. The updated node attributes and weight changes of the knowledge graph are fed back to the priority sorting step and the test case generation step.

8. A vehicle data testing system based on large models and knowledge graphs, characterized in that, include: The knowledge graph construction and management module is used to construct and maintain the knowledge graph, and to generate structured domain context based on the knowledge graph through subgraph retrieval and reasoning. The large model-driven intelligent test case generation module is used to receive the structured domain context and test requirements, and use prompt word templates to guide the large language model to generate structured test cases. The test case multidimensional verification and evaluation module is used to perform validity pre-evaluation and coverage analysis on the test cases based on the knowledge graph, and to map the verified test cases to the knowledge graph; The steps of the effectiveness pre-evaluation based on the knowledge graph include: Obtain the target object information intended to be executed by the test case, and extract a detailed subgraph of the target object from the knowledge graph, wherein the detailed subgraph includes hardware model, software version and known protection measures; The attack techniques and target components intended to be used in the test cases are analyzed. The connection paths between the attack techniques, the target components and the corresponding vulnerabilities are queried in the knowledge graph. The security mechanisms that defend against the attack techniques are also checked on the target components. The pre-evaluation confidence score is calculated based on the network connectivity results and the security mechanism check results. When the pre-evaluation confidence score is lower than a set threshold, the test case is marked as invalid and filtered. The steps of the coverage analysis include: The knowledge graph extracts the security requirement nodes and attack surface nodes associated with the current test task, and maps the test cases to the attack surfaces and security requirements they cover. Calculate the difference between the set of covered nodes of the test cases and the set already covered by the current test task to identify a list of uncovered coverage blind spots; The list of coverage blind spots is fed back to the step of generating test cases as the trigger input for generating new test cases; The intelligent recommendation and priority ranking module is used to calculate the multi-dimensional priority of the test cases based on the topology of the knowledge graph and historical execution data, and obtain a recommended list of test cases ranked by priority. An adaptive test execution engine is used to execute test cases in the recommendation list. It utilizes a hybrid decision framework that includes a rule engine and a reinforcement learning model to dynamically adjust the execution sequence and parameters of test cases based on real-time feedback data. The dynamic evaluation and knowledge update module is used to collect result data to reconstruct attack paths or diagnose root causes of failures in order to extract knowledge triples or root cause labels, and update the knowledge graph based on the knowledge triples or root cause labels, so as to drive the iteration of the knowledge graph construction and management module, the large model-driven intelligent test case generation module, and the intelligent recommendation and priority ranking module.

Images