Method and system for predicting one or more attributes of test cases

Abstract

To provide a computer-implemented method, a computing system, a computer program, a machine-readable storage medium, and a data carrier signal for prediction of one or more attributes of test cases.SOLUTION: A method includes receiving a plurality of test cases from a test case library. Each test case includes text descriptions of one or more executable test steps of the test case. The method further includes: generating a corpus-level graph representative of the plurality of test cases; and feeding the corpus-level graph into a trained temporal graph neural network to predict one or more attributes of each test case that affect a testing report of software, hardware, and / or system testing using the test case.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure generally relates to software, hardware, and / or system testing, and more specifically to the representation and prediction of one or more attributes of test cases for software, hardware, and / or system testing by generating test reports.

[0002] Cross - Reference to Related Applications This application claims the benefit of GB2319457.4, filed on December 19, 2023, entitled "Method and System for Predicting One or More Attributes of Test Cases", which is hereby incorporated by reference in its entirety.

Background Art

[0003] In the field of software - based testing, machine learning techniques such as classification, regression, clustering, and reinforcement learning have been proposed for prioritizing test cases. However, such techniques typically require access to source code for software testing. In system testing, access to source code is usually not permitted, and there is no straightforward way to transfer the proposed solutions directly from software testing to system testing. Existing methods also do not consider the behavior and interactions among hardware components that may exist during system testing.

[0004] In the case of system testing, current methods rely on test managers to create or design test cases and thus rely heavily on the intuition and experience of test managers. Also, there is a lack of an approach that can guide the decision - making process of test managers and ensure the quality of the generated test plans.

[0005] Existing methods for automatically generating test plans include the use of evolutionary algorithms and support vector machines. Existing methods that use an evolutionary algorithm (EA) generate a test plan by using a belief model based on the heuristic performance of test cases to recommend the test cases to use and the number of cycles each test case should be executed. However, such methods cannot handle new test cases without previous performance history or test cases that have never been seen during the training phase after deployment, and the evaluation of such test cases and test plans cannot be generalized to test cases that have never been seen before. Existing methods that use a support vector machine (SVM) apply natural language processing (NLP) techniques to test case descriptions and metadata before using the SVM to classify test cases as important or unimportant. However, such methods assume that test case descriptions are always available, which is not always applicable. In addition, such methods also assume that test case descriptions are consistent across the industry, which may not be the case since test case descriptions are usually non-experiential and can vary from user to user. SUMMARY OF THE INVENTION

[0006] An object of the present disclosure is to provide a method for predicting attributes of test cases that affect test reports of hardware, software, and / or system tests by providing the subject matter of the independent claims using such test cases.

[0007] The object of the present disclosure is solved by the subject matter of the independent claims, and further embodiments are incorporated in the dependent claims.

[0008] All embodiments of the present disclosure related to the method may be executed in the order of the steps described, but it should be noted that this is not necessarily the only and essential order of the steps of the method. The methods presented herein can be executed in an order different from the disclosed steps without departing from each method embodiment, unless the contrary is explicitly stated hereinafter.

[0009] To solve the above technical problems, the present disclosure provides a computer-implemented method. This computer-implemented method receiving a plurality of test cases from a test case library, each test case including a text description of one or more executable test steps of the test case, generating a corpus-level graph representing the plurality of test cases, feeding the corpus-level graph into a trained time-series graph neural network to predict one or more attributes of each test case that affect a test report of software, hardware, and / or system tests using the test case and including.

[0010] The computer-implemented method of the present disclosure is advantageous over known methods in that the disclosed method can predict test case attributes that affect the test reports of software, hardware, and / or system tests using test cases without a previous performance history and without executing test cases. The prediction uses a corpus-level graph that represents and stores information about the complex interactions and relationships (including temporal relationships) of different components, characteristics, and attributes of test cases using nodes and edges, thereby potentially revealing patterns, trends, and insights that may be difficult to achieve using other methods. The corpus-level graph is graph-structured data that is more advantageous than conventional tabular data structures because it has higher interpretability due to the easy visualization of relationships and paths within the data. The graph-structured data is also advantageous because it provides context information to the machine by showing how entities are connected to each other and facilitates training a machine learning model as compared to conventional tabular data. Additionally, the use of a trained time-series graph neural network may enable more accurate prediction of attributes because the trained time-series graph neural network takes into account the temporal relationships present in the corpus-level graph. The trained time-series graph neural network is also advantageous for sequential processing in the present disclosure because it is independent of the order of words.

[0011] A preferred method of the present disclosure is the computer-implemented method described above, wherein the corpus-level graph is a plurality of word nodes, each representing one or more words within a text description, a plurality of test step nodes, each representing a test step, a plurality of undirected edges connecting the word nodes to their associated test step nodes, and a plurality of directed edges connecting the test step nodes imposing an ordering relationship that specifies the ordering between executable test steps represented by the test step nodes and including.

[0012] The above-described aspects of the present disclosure have the advantage that the specific topology of the corpus-level graph represents, stores information regarding the complex interactions and relationships (including temporal relationships) of the different components, characteristics, and attributes of the test case, thereby potentially better predicting any attribute of the test case. The topology of the corpus-level graph is also a consistent representation of test cases and test case libraries that can potentially be more easily integrated or fed into downstream applications. The specific topology is also advantageous because the learned information is embedded at the word node level, so predictions generated by a trained time series graph neural network can be easily adapted to executable test steps and test cases that have not been seen before.

[0013] A preferred method of the present disclosure is the computer-implemented method described above or as preferred above, in which one or more words represented by word nodes are identified from the text description using tokenization and / or text cleaning.

[0014] The above-described aspects of the present disclosure have the advantage that text preprocessing techniques such as tokenization and text cleaning remove stop words, punctuation, and other irrelevant text from the text description, thereby identifying and retaining important features, parameters, or components present within the text description to be represented in the corpus-level graph. Tokenization is particularly advantageous in facilitating more efficient and structured text analysis by splitting long sentences or text into individual words or tokens and enabling the model to learn embeddings representing each of the token units, where each of the token units can then be combined into meaningful representations of sentences or graphs in the text description. Tokenization also enables normalization of the text, which allows techniques such as stemming and stop word removal to be more easily applied. The tokenizer also removes irrelevant text from the text description, thereby enabling integration of text descriptions from various different sources that may have inconsistent languages. Further, the use of the tokenizer may enable easier training of a time-series graph neural network as only important words or features are identified and considered.

[0015] A preferred method of the present disclosure is the computer-implemented method described above or described as preferred, and generating a corpus-level graph representing a plurality of test cases is generating a test case graph for each test case of the plurality of test cases, and constructing a test step graph for each test step of the test case, wherein each test step graph includes a test step node connected to a plurality of word nodes having undirected edges, constructing the test step graph. Connecting test step graphs of each test case to generate a test case graph, wherein the test step nodes of the test step graph are connected by directed edges based on an ordering relationship, and word nodes common to two or more executable test steps are merged, and connecting the test step graphs Connecting test case graphs to generate a corpus-level graph, wherein test step nodes common to two or more test cases are merged, and connecting the test case graphs including

[0016] In the above-described aspect of the present disclosure, information about each test case (e.g., an attribute such as a failure rate) is used to train test step nodes and word nodes (i.e., through trickle-down of information), so that when a new test case or test step appears, the information learned by the word nodes is used to construct embeddings for the new executable test steps and new test cases, and inferences or predictions (e.g., prediction of failure rate) can be made about such new test steps or test cases.

[0017] A preferred method of the present disclosure is the computer-implemented method described above or described as preferred, wherein a test case further includes a text description of one or more preconditions, a corpus-level graph further includes one or more precondition nodes, each precondition node represents a precondition and is connected to one or more word nodes by an undirected edge, and when there are two or more precondition nodes, the precondition nodes are connected by a directed edge imposing an ordering relationship specifying an ordering between the preconditions represented by the precondition nodes, and finally the ordered precondition nodes are connected by a directed edge to the first-ordered test step node of the test case.

[0018] The above-described aspects of the present disclosure have the advantage of providing additional context to the model, such as information regarding what preconditions to include or consider, how test cases are to be set up, and which modules / components a test case may be related to, thereby enabling the model to better infer test cases, associate that test case with other test cases, or provide more useful information that can be used by the model to generalize the test case.

[0019] A preferred method of the present disclosure is the computer-implemented method described above or described as preferred above, wherein a test case further includes a text description of the expected result, the corpus-level graph further includes one or more expected result nodes, each expected result node represents an expected result, is connected to one or more word nodes by an undirected edge, and when there are two or more expected result nodes, the expected result nodes are connected by a directed edge that imposes an ordering relationship specifying the ordering between the expected results represented by the expected result nodes, and the first ordered expected result node is connected by a directed edge to the last ordered test step node of the test case.

[0020] The above-described aspects of the present disclosure have the advantage of providing additional context to the model, such as information regarding how different modules / components should react after a test case is executed, thereby enabling the model to better infer test cases, associate that test case with other test cases, or provide more useful information that can be used by the model to generalize the test case.

[0021] A preferred method of the present disclosure is the computer-implemented method described above or described as preferred above, and this method associating each test case with its predicted one or more attributes, and Generating a test plan for testing software, hardware, and / or a system based on a test case having one or more predicted attributes; Executing one or more test cases of the test plan to generate test results; Identifying one or more defects in the software, hardware, and / or system based on the generated test results; including; Executing one or more test cases of the test plan preferably includes testing software, hardware, and / or a system on a vehicle and generating test results using signals received from one or more vehicle components.

[0022] The above-described aspects of the present disclosure may better identify defects in software, hardware, and / or a system using the results of a test plan generated based on a test case having predicted attributes, and thus have the advantage of assisting in creating software, hardware, and / or a system with fewer defects. Executing at least one test case by testing on a vehicle and generating test results using signals received from one or more vehicle components also ensures that the software, hardware, and / or system is safe for implementation on a vehicle.

[0023] A preferred method of the present invention is the computer-implemented method described above or as preferred above, wherein executing one or more test plans is preferably automatically executed using one or more automation scripts.

[0024] The above-described aspects of the present disclosure have the advantage that the implementation of the test plan is automatically executed without human intervention, and thus can enhance the effectiveness of the implementation of the test plan.

[0025] The above-described advantageous aspects of the computer-implemented method of the present disclosure also apply to all aspects of the computing system of the present disclosure described hereinafter. All of the advantageous aspects of the computing system of the present disclosure described hereinafter also apply to all aspects of the computer-implemented method of the present disclosure described above.

[0026] The present disclosure also relates to a computing system comprising one or more processors and a memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for performing the computer-implemented method according to any one of the preceding claims.

[0027] A preferred method of the present disclosure is the above-described computing system configured to be connected to a vehicle and receive signals from one or more vehicle components.

[0028] The above-described aspects of the present disclosure have the advantage that a test plan may be implemented on software, hardware, and / or systems for vehicle applications and may be tested and evaluated based on signals from vehicle components, thus ensuring that the software, hardware, and / or systems for vehicle applications are safely implemented within the vehicle.

[0029] The above-described advantageous aspects of the computer-implemented method or computing system of the present disclosure also apply to all aspects of the use of the computer-implemented method of the present disclosure described hereinafter. All of the advantageous aspects of the use of the computer-implemented method of the present disclosure described hereinafter also apply to all aspects of the computer-implemented method or computing system of the present disclosure described above.

[0030] The present disclosure also relates to the use of the computer-implemented method of the present disclosure.

[0031] The above-described advantageous aspects of the computer-implemented method, computing system, or use of a computer-implemented method of the present disclosure also apply to all aspects of the computer program, machine-readable storage medium, or data carrier signal of the present disclosure described hereinafter. All of the advantageous aspects of the computer program, machine-readable storage medium, or data carrier signal of the present disclosure described hereinafter also apply to all aspects of the computer-implemented method, computing system, or use of a computer-implemented method of the present disclosure described above.

[0032] The present disclosure also relates to a computer program, machine-readable storage medium, or data carrier signal that, when executed on a data processing device and / or control unit, includes instructions that cause the data processing device and / or control unit to perform the steps of the computer-implemented method according to the present disclosure. A machine-readable medium can include any medium and / or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). A machine-readable medium can be, for example, a read-only memory (ROM), random access memory (RAM), universal serial bus (USB) stick, compact disc (CD), digital video disc (DVD), data storage device, hard disk, any medium such as an electrical, acoustic, optical, or other form of propagated signal (e.g., a digital signal, data carrier signal, carrier wave), or any other medium in which the program elements described above can be transmitted and / or stored.

[0033] As used in this summary, the following description, the following claims, and the accompanying drawings, the term "test case" refers to a predefined test procedure for testing some of the functions of the software being tested. An example of a predefined test procedure is the procedure used to test whether a vehicle's seat belt detection system is operating.

[0034] As used in this summary, the following description, the following claims, and the accompanying drawings, the term "component" refers to any software or hardware that is part of or an element of the system being tested. For example, if the system being tested is a vehicle, a component may refer to any software or hardware that is part of or an element of the vehicle and may be part of one or more systems within the vehicle. Examples of vehicle components include engines, batteries, alternators, brakes, radiators, transmissions, shock absorbers, converters, steering, electronic control units, suspensions, sensors, and the like.

[0035] These and other features, aspects, and advantages will be better understood with reference to the following description, the appended claims, and the accompanying drawings.

Brief Description of the Drawings

[0036]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

[0037] In the drawings, like parts are designated by like reference numerals.

[0038] It should be understood by those skilled in the art that any block diagram in this specification represents a conceptual diagram of an exemplary system embodying the principles of the present subject matter. Similarly, any flowchart, flow diagram, state transition diagram, pseudocode, etc., represents various processes that can be substantially represented in a computer-readable medium and executed by such a computer or processor, whether or not a computer or processor is explicitly shown.

[0039] In the above summary, this description, the following claims, and the accompanying drawings, specific features of the present disclosure (including method steps) are referred to. It should be understood that the disclosure herein includes all possible combinations of such specific features. For example, if a particular feature is disclosed in the context of a particular aspect or embodiment of the present disclosure, or a particular claim, that feature can also be combined, to the extent possible, with other particular aspects and embodiments of the present disclosure, and / or in that context, and generally used in the present disclosure.

[0040] As used herein, the word "exemplary" is used to mean "serving as an example, instance, or illustration" in this specification. Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0041] Although the present disclosure admits of various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will be described in detail below. It should be understood, however, that the present disclosure is not intended to be limited to the disclosed form, but on the contrary, is intended to cover all modifications, equivalents, and alternatives falling within the scope of the present disclosure.

[0042] The present disclosure is directed to a computer-implemented method, a computing system, use of a computer-implemented method, a computer program, a machine-readable storage medium, and a data carrier signal for generating graphical representations and predicting attributes of test cases. Attributes that affect test reports or results of hardware, software, and / or system testing are predicted using a trained time-series graph neural network, and the predicted attributes can be associated with their respective test cases. Test cases having the predicted attributes can then be used to generate a test plan that can be implemented to identify defects in software, hardware, and / or systems.

[0043] FIG. 1 is a schematic diagram of a computer-implemented method 100 for predicting attributes of test cases and identifying defects in software, hardware, and / or systems according to an embodiment of the present disclosure. The method 100 for detecting defects in software, hardware, and / or systems can be implemented by a data processing device on any architecture and / or computing system. For example, various architectures using, for example, multiple integrated circuit (IC) chips and / or packages, and / or various computing devices and / or consumer electronic (CE) devices such as multifunctional devices, tablets, smartphones, etc., can implement the techniques and / or configurations described herein. When executed on a data processing device and / or control unit, the method 100 can be stored as executable instructions that cause the data processing device and / or control unit to execute the steps of the method 100.

[0044] According to some embodiments, a method 100 for diagnosing software, hardware, and / or system defects may include a step 108 in which a plurality of test cases are received from a test case library. The plurality of test cases may include test cases for evaluating software. In some embodiments, the plurality of test cases may be defined based on the software to be evaluated. As an example, the plurality of test cases may include test cases for testing automotive functions for software used in an automotive or vehicle application. In some embodiments, the plurality of test cases may be defined based on test objectives, characteristics and / or functions of the software, hardware, or system being tested, and / or the stage or phase of software, hardware, or system development. The test objective can be any test objective including technical objectives and / or business objectives. The test cases can be any type of test cases including functional test cases, performance test cases, unit test cases, user interface test cases, security test cases, integration test cases, database test cases, usability test cases, user acceptance test cases, regression test cases, life cycle tests, stress tests, or any combination thereof. The test case library may include test cases executed on previous versions of the software, hardware, or system, test cases previously executed on similar software, hardware, or systems, newly defined test cases, and / or general-purpose test cases. Newly defined test cases can be test cases that are newly created, designed, or defined for testing software, hardware, or systems and have not yet been implemented and / or executed. In situations where there are new requirements or features and the test cases in the test case library do not cover such new requirements, new test cases may be defined and added to the test case library.A general-purpose test case may be a test case defined for features that may be common across different software, applications, and / or product lines and can be reused and / or re-adapted with minimal effort. Examples of general-purpose test cases include test cases designed to test features such as backup battery, Bluetooth, emergency call, etc. In some embodiments, the test case library may be stored on a data storage device and may be retrieved when implementing method 100.

[0045] According to some embodiments, each test case may include a textual description of one or more executable test steps of the test case. Table 1 below shows an example of a test case having the associated textual description of the executable test steps for each test case. As shown in Table 1, each test case may be associated with a textual description of one or more executable test steps. Table 1 TIFF2025106212000002.tif100170

[0046] According to some embodiments, each test case may further include a textual description of one or more preconditions and / or one or more expected results. Table 2 below shows an example of a test case having the associated textual description of one or more preconditions, one or more executable test steps or variations, and one or more expected results. In some embodiments, the textual descriptions of one or more preconditions, one or more executable test steps or variations, and one or more expected results may be concatenated in a single column (see Table 3 below). It is emphasized that the examples provided merely enumerate preconditions, executable test steps, and expected results as test case parameters, but there may be other test case parameters that can be associated with a test case and represented within a corpus-level graph. Table 2 TIFF2025106212000003.tif Table 3 of 168170 TIFF2025106212000004.tif of 191170

[0047] According to some embodiments, method 100 may include step 116 in which a corpus-level graph representing a plurality of test cases is generated. The corpus-level graph represents all of the test cases of the plurality of test cases and may include one or more nodes and one or more edges connecting the one or more nodes.

[0048] FIG. 2 is a schematic diagram of an example of a corpus-level graph according to an embodiment of the present disclosure. In particular, FIG. 2 shows a corpus-level graph 200 representing two test case IDs listed in Table 1. As shown in FIG. 2, the corpus-level graph 200 may include a plurality of word nodes 208 (shown as white circles), a plurality of test step nodes 216 (shown as black circles), a plurality of undirected edges 224 (shown as lines), and a plurality of directed edges 232 (shown as arrows).

[0049] According to some embodiments, each word node 208 represents one or more words within the text description of a test case. In some embodiments, the one or more words represented by word node 208 can be components listed within the text description. In some embodiments, the one or more words represented by word node 208 can be identified from the text description using text preprocessing. In some embodiments, the one or more words represented by word node 208 can be identified from the text description using a spacer and text cleaning. Preferably, the one or more words represented by word node 208 are identified from the text description using tokenization and / or text cleaning. A tokenizer decomposes the text stream into tokens by splitting phrases, sentences, or paragraphs into smaller units (called tokens). Tokenization is performed using a tokenizer library. Examples of tokenizer libraries include the NLTK tokenizer available at https: / / www.nltk.org / api / nltk.tokenize.html, and the spaCy tokenizer available at https: / / spacy.io / api / tokenizer. Text cleaning involves processing the tokens generated by the tokenizer and includes techniques such as stop word removal, lowercasing, punctuation removal, verb stemming (jumping->jump), whitespace trimming, and / or expansion of contractions (e.g., "on’t->do not").

[0050] According to some embodiments, each test step node 216 represents a test step. In some embodiments, a plurality of undirected edges 224 connect word nodes 208 to their associated test step nodes 216. Each word node 208 can be connected to one or more test step nodes 216 via an undirected edge 224. For example, the first test step of test ID TC_ATP_01 in Table 1 is the state of "DCM is in RUN mode". As shown in FIG. 2, one or more words identified by tokenization and / or text cleaning are "DCM" and "Run mode" represented by word nodes 208a and 208b respectively, and word nodes 208a and 208b are connected to a test step node 216a representing the test step "DCM is in RUN mode" having undirected edges 224a and 224b respectively.

[0051] According to some embodiments, a plurality of directed edges 232 connect test step nodes 216, and the directed edges 232 impose an ordering relationship that specifies the ordering between executable test steps represented by the test step nodes 216. For example, as shown in FIG. 2, there may be five test step nodes 216a - 216e. Based on the directed edges 232, there is a first ordering of executable test steps representing test case ID TC_ATP_01 from test step node 216a (marked as TS1), test step node 216b (marked as TS2) → test step node 216c (marked as TS3) → test step node 216d (marked as TS4), and a second ordering of executable test steps representing test case TC_ATP_01 from test step node 216a (marked as TS1) → test step node 216d (marked as TS4) → test step node 216e (marked as TS5) for test case ID TC_ATP_04.

[0052] FIG. 3 is a simplified test case graph representing a test case that includes text descriptions of preconditions, executable test steps, and expected results according to an embodiment of the present disclosure. In particular, FIG. 3 shows a simplified test case graph 300 representing test case ID TC_ATP_01 listed in Tables 2 and 3. Word nodes are not shown in the illustration for ease of visualization.

[0053] In some embodiments, a graph 300 representing a test case that includes a text description of a precondition may further include one or more precondition nodes 308 (shown as circles with a horizontal line pattern), where each precondition node 308 represents a precondition associated with the test case. In the case of test case ID TC_ATP_01, four preconditions are listed, and thus there are four precondition nodes 308a, 308b, 308c, and 308d, each representing a precondition associated with test case ID TC_ATP_01. The precondition nodes 308 are connected by directed edges 316 that impose an ordering relationship specifying the ordering among the preconditions represented by the precondition nodes 308. Based on the directed edges, there is an ordering of the preconditions associated with test case ID TC_ATP_01 from precondition node 308a (marked as PC1) → precondition node 308b (marked as PC2) → precondition node 308c (marked as PC3) → precondition node 308d (marked as PC4). In some embodiments, the last ordered precondition node 308 is connected to the first ordered test step node 324 of the test case. As shown in FIG. 3, there is an ordering of executable test steps associated with test case ID TC_ATP_01 from test step node 216a (marked as TS1) → test step node 216b (marked as TS2) → test step node 216c (marked as TS3) → test step node 216d (marked as TS4), and the last ordered precondition node 308d is connected to the first ordered test case node 324a via a directed edge 316a.

[0054] In some embodiments, a graph 300 representing a test case that includes a textual description of an expected result may further include one or more expected result nodes 332 (shown as circles with dot patterns), where each expected result node 332 represents an expected result associated with the test case. For test case ID TC_ATP_01, there is a single expected result, and thus there is a single expected result node 332 that represents the expected result associated with test case ID TC_ATP_01. If there are two or more expected results, the expected result nodes 332 may be connected by directed edges (not shown) that impose an ordering relationship that specifies the ordering among the expected results represented by the expected result nodes 332. In some embodiments, the first ordered expected result node 332 is connected to the last ordered test step node 324. For test case ID TC_ATP_01 shown in FIG. 3, there is only one expected result node 332, and thus it is connected to the last ordered test step node 324d (marked as TS4) via the directed edge 316b. FIGS. 2 and 3 merely illustrate nodes for preconditions, executable test steps, and / or expected results as test case parameters, but it is emphasized that there may be other test case parameters that can be represented within the corpus-level graph as nodes associated with the test case and connected by directed edges.

[0055] FIG. 4A and FIG. 4B are schematic diagrams of a method 400 for generating a corpus-level graph representing a plurality of test cases according to an embodiment of the present disclosure. In particular, FIGS. 4A and 4B are schematic diagrams of a method for generating a corpus-level graph representing the test cases of Table 1. According to some embodiments, method 400 may include step 408 in which a test case graph 416 is generated for each test case of the plurality of test cases. In some embodiments, step 408 may include step 424 in which a test step graph 432 is constructed for each test step of the test case, and step 440 in which the test step graphs 432 of each test case are connected to generate a test case graph 416, the test step nodes of the test step graphs being connected by directed edges based on an ordering relationship, and word nodes common to two or more executable test steps being merged.

[0056] Figure 4A shows how a test case graph 416 is generated for test case ID TC_ATP_01. Test case ID TC_ATP_01 includes text descriptions of four executable test steps. In step 424, a test step graph 432 is generated for each test step. The first test step graph 432a generated for the first test step describes "DCM is in RUN mode", the second test step graph 432b generated for the second test step describes "Trigger an SOS call", the third test step graph 432c generated for the third test step describes "End the SOS call using the debug console", and the fourth test step graph 432d generated for the fourth test step describes "Turn off IGN and ACC". The four test step graphs 432a - 432d are then merged in step 440 to generate the test case graph 416. The test step nodes 448a - 448d are connected by directed edges 456a - 456c based on the ordering relationship of the executable test steps represented by the test step nodes 448a - 448d. Word nodes common to two or more executable test steps are also merged in step 440. For example, the word node 464a connected to the test step node 448b and the word node 464b connected to the test step node 448c both represent the word "SOS call". Thus, the two word nodes 464a and 464b are merged in step 440 to form a word node 464c connected to the test step nodes 448b and 448c within the test case graph 416.

[0057] According to some embodiments, method 400 may include step 464 of connecting test case graph 416 to generate corpus-level graph 472, where test step nodes 448 / 466 common to two or more test cases 416 are merged. FIG. 4B shows how corpus-level graph 472 represents two test cases (test case ID TC_ATP_01 and test case ID TC_ATP_04). For example, test step node 466a of test case graph 416a and test step node 446b of test case graph 416b are both connected to word nodes representing the words "RUN mode" and "DCM", and thus, the two test step nodes 466a and 466b are merged in step 464 to form test step node 466c. FIG. 4B describes the generation of the corpus-level graph for two cases, but it is emphasized that this method is scalable and uses the same steps / principle to generate a corpus-level graph representing any number of test cases.

[0058] Referring to FIG. 1, method 100 may include step 124 where a corpus-level graph is supplied to a trained time-series graph neural network to predict one or more attributes of each test case that may affect the test report or result of software, hardware, and / or system testing using the test case. The one or more attributes can be any attribute that may affect the test report or result of software, hardware, and / or system testing using the test case. Examples of attributes include failure rate, test coverage rate, test penetration, affected modules / components, test duration, and / or test type. In some embodiments, the trained time-series graph neural network can be any sequential model or time-based model, such as a recurrent neural network (RNN), long-short term memory model (LSTM), or Transformer, connected to any graph neural network. Preferably, the trained time-series graph neural network is an improved Graph-BERT. Graph-BERT is a graph-based BERT (Bidirectional Encoder Representations from Transformers) machine learning framework disclosed in "Graph-BERT: Only Attention is Needed for Learning Graph Representations" by Zhang et al. (arXiv:2001.05140v2), which trains nodes sampled from the input large-sized graph data along with their contexts (also called "linkless subgraphs") such that the representation learning is purely based on the attention mechanism. Graph-BERT generally includes several parts, namely, (1) linkless subgraph batching, (2) node input vector embedding, (3) graph transformer-based encoder, (4) representation fusion, and (5) functional components.Each sampled linkless subgraph covers both the target node and the surrounding context nodes. In the present disclosure, a linkless subgraph may correspond to a test step graph, the target node may correspond to a test step node, and the connected word nodes may correspond to the surrounding context nodes. With respect to the functional components, the improved Graph-BERT can be trained for a prediction task, particularly the prediction of one or more test case attributes.

[0059] In some embodiments, the architecture and training mechanism for the time-series graph neural network may correspond to the architecture and training mechanism disclosed in the paper by Zhang et al. with the following modifications. · Data labels modified from LongTensor to FloatTensor for class one-hot encoding · Training loss modified from cross-entropy loss (used for classification) to mean squared error (MSE) (used for regression) · Output size of the prediction modified from [number of classes] to [1].

[0060] In some embodiments, the time-series graph neural network can be trained using a batch of subgraphs of a test case graph labeled with one or more attributes (e.g., failure rate, test coverage rate, test penetration, affected modules / components, test period, and / or test type).

[0061] Referring to FIG. 1, method 100 may include step 132 where each test case is associated with its related one or more attributes predicted at step 124. In some embodiments, any known encoding method such as one-hot encoding may be used to associate the test cases with their predicted one or more attributes.

[0062] According to some embodiments, method 100 may include step 140 of generating a test plan for testing software, hardware, or a system based on a test case having one or more predicted attributes output from step 132. The test plan may be generated using any known optimization algorithm such as a Markov decision process and reinforcement learning. For example, the test plan may be generated using the automated test plan generation framework disclosed in "An Automatic Test Plan Generation Approach for Automotive Software Testing" by Cao et al.

[0063] According to some embodiments, method 100 may include step 148 of executing one or more test cases of the generated test plan to generate test results. In some embodiments, the one or more test cases may be executed on software, hardware, and / or a system. In some embodiments, the one or more test cases may be executed manually. In some embodiments, the one or more test cases may be automatically executed, preferably using one or more automation scripts. Examples of test cases that may be automatically executed include BUB (backup battery test), FOTA (firmware over air), ATB (engine cranking, voice call), and AFT (diagnostic DTC). In some embodiments, when the software, hardware, or system is for an automotive or vehicle application, the execution of at least one test case may include embedding and / or testing the software, hardware, or system on the vehicle. In such embodiments, the test results may be generated using signals received from one or more vehicle components.

[0064] According to some embodiments, method 100 may include step 156 in which one or more defects are identified in software, hardware, and / or the system based on the test results generated in step 148. For example, when running a BUB test, if the test results indicate that the backup battery is not actively supplying power or is not deactivated when the power is turned off, a defect may be identified in the software backup battery module (i.e., the module that controls the backup battery). Implementations of steps 148 and 156 are described in further detail in connection with FIG. 5.

[0065] FIG. 5 is a schematic diagram of a system that can be used for implementing one or more test cases according to an embodiment of the present disclosure. System 500 can be used to implement one or more steps of method 100. In some embodiments, system 500 can include one or more processors 508 configured to execute step 148 of method 100. In some embodiments, system 500 can be installed within vehicle 516 to test software, hardware, or a system to be tested and execute one or more test cases on such software, hardware, or system on the vehicle. In some embodiments, executing one or more test cases on such software, hardware, or system on the vehicle can include controlling one or more vehicle components 524 and / or receiving signals from one or more vehicle components 524. An example of a test case that can be executed on the vehicle is a test for an emergency call (safety system), where an airbag pulse width modulation (PWM) signal and a control area network (CAN) signal are collected and used by a data communication module (DCM) to make a determination of a collision detection and trigger an advanced collision notification call function (ECall). Another example of a test case that can be executed on the vehicle is a test for a backup battery (BUB), where an Arduino platform is used to control the data communication module (DCM) by coding, and a system on chip (SOC) and a vehicle microcontroller (VUC) are used to monitor DCM activity. Prerequisites for such a BUB test are that the backup battery is connected, the DCM power is on, the ignition (IGN) is on, and the ACC (accessory) is on. Steps of such a BUB test are as follows. 1. Check that the serial port is not occupied by other communication channels other than the automated test platform (ATP) (to avoid problems with occupied ports). 2. Turn off the DCM power supply (IGN and ACC remain on). 3. ATP waits for 10 seconds. 4. Check that BUB is actively powered. 5. ATP waits for 5 seconds. 6. Turn on the DCM power supply (IGN and ACC remain on). 7. Check that BUB is inactive. If any of the checks in step 4 followed by step 7 fail, a problem in the BUB module of the software being tested may be identified. In some embodiments, the problem may be identified with further attention by the tester. The problem may exist in the software, hardware, and / or system.

[0066] FIG. 6 shows an example of a computing system according to an embodiment of the present disclosure. Computing system 600 can be used, for example, for one or more steps of method 100. Computing system 600 can be used for one or more of the components of system 500 in FIG. 5. System 600 can be a computer connected to a network. System 600 can be a client or a server. As shown in FIG. 6, system 600 can be any suitable type of processor-based system, such as a personal computer, a workstation, a server, a handheld computing device (portable electronic device) such as a phone or a tablet, or an embedded system or other dedicated device. System 600 can include, for example, one or more of input device 620, output device 630, one or more processors 610, storage 640, and communication device 660. Input device 620 and output device 630 can generally be either connectable to or integrated with computing system 600. In some embodiments, storage 640 can store the generated test case library and / or the corpus-level graph.

[0067] Input device 620 can be any suitable device that provides input, such as a touch screen, a keyboard or keypad, a mouse, a gesture recognition component of a virtual / augmented reality system, or a voice recognition device. Output device 630 can be or include any suitable device that provides output, such as a display, a touch screen, a tactile device, a virtual / augmented reality display, or a speaker.

[0068] Storage 640 can be any suitable device that provides storage, such as electrical, magnetic, or optical memory, including RAM, cache, hard drive, removable storage disk, or other non-transitory computer-readable media. Communication device 660 can include any suitable device that can transmit and receive signals over a network, such as a network interface chip or device. The components of computing system 600 can be connected in any suitable way, either via a physical bus or wirelessly.

[0069] Processor 610 can be any suitable processor or combination of processors, including any one of a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), and an application specific integrated circuit (ASIC), or any combination thereof. Software 650 stored in storage 640 and executable by one or more processors 610 can include, for example, programming that implements the functions or a portion of the functions of the present disclosure (e.g., embodied in the devices described above). For example, software 650 can include one or more programs executable by one or more processors 610 to perform one or more of the steps of method 100.

[0070] Software 650 can also be stored in and / or transported within any non-transitory computer-readable storage medium for use by, or in connection with, the instruction execution systems, apparatus, or devices described above, which can read the software-related instructions therefrom and execute those instructions. In the context of the present disclosure, a computer-readable storage medium can be any medium such as storage 640 that can contain or store programming for use by, or in connection with, an instruction execution system, apparatus, or device.

[0071] Software 650 can also be propagated within any transport medium for use by, or in connection with, the instruction execution systems, devices, or apparatuses described above, and these instruction execution systems, devices, or apparatuses can read out the instructions related to the software therefrom and execute those instructions. In the context of the present disclosure, the transport medium can be any medium that can communicate, propagate, or transport programming for use by, or in connection with, the instruction execution systems, devices, or apparatuses. The transport computer-readable medium can include, but is not limited to, wired or wireless propagation media by electronic, magnetic, optical, electromagnetic, or infrared.

[0072] System 600 can be connected to a network that can be any suitable type of interconnected communication system. The network can implement any suitable communication protocol and can be protected by any suitable security protocol. The network can include any suitable configuration of network links capable of transmitting and receiving network signals, such as wireless network connections, T1 or T3 lines, cable networks, DSL, or telephone lines.

[0073] System 600 can implement any operating system suitable for operating on a network. Software 650 can be described in any suitable programming language such as C, C++, Java, or Python. In various embodiments, the application software embodying the functions of the present disclosure can be deployed in different configurations, for example, in a client / server configuration or as a web-based application or web service via a web browser or the like.

[0074] Finally, the language used in this specification has been chosen primarily for readability and for the purpose of instruction, and may not have been chosen to delineate or limit the subject matter of the invention. Accordingly, the scope of the invention is intended to be limited not by this detailed description, but rather by any claims that issue on an application based on this specification. Accordingly, the embodiments of this disclosure are illustrative of the scope of the invention as set forth in the following claims and are not intended to limit it.

Claims

1. A computer-implemented method for automating the testing of hardware products and / or software products for an industry, the method comprising: receiving (108), by one or more processors, a plurality of test cases stored in and retrievable from a test case library, each test case including a textual description of one or more executable test steps of the test case; generating (116), by the one or more processors, a corpus-level graph representing the plurality of test cases; feeding (124) the generated corpus-level graph to a trained time-series graph neural network; and the method further comprising: generating, by the time-series graph neural network, the test report including a result predicting one or more attributes of at least one test result of each test case affecting the test report; wherein the at least one test result included in the test report is selected from the group consisting of a failure rate test result, a test coverage rate, a test penetration rate, an affected module / component, a test period, and / or a test type, the computer-implemented method.

2. The corpus-level graph comprises: a plurality of word nodes (208), each representing one or more words within the textual description; a plurality of test step nodes (216), each representing a test step; a plurality of undirected edges (224) connecting the word nodes to their associated test step nodes; and a plurality of directed edges (232) connecting the test step nodes imposing an ordering relationship specifying an ordering between executable test steps represented by the test step nodes; The computer-implemented method according to claim 1.

3. The one or more words represented by the word nodes are identified from the textual description using tokenization and / or text cleaning. The computer-implemented method according to claim 2.

4. Generating the corpus-level graph representing the plurality of test cases comprises: generating (408) a test case graph for each test case of the plurality of test cases, Constructing (424) a test step graph for each test step of the test case, each test step graph including a test step node connected to a plurality of word nodes having undirected edges, constructing (424) the test step graph, and Connecting (440) the test step graphs of each test case to generate the test case graph, wherein the test step nodes of the test step graphs are connected by directed edges based on the ordering relationship, and word nodes common to two or more executable test steps are merged, connecting (440) the test step graph Including, generating (408) a test case graph; Connecting (464) the test case graphs to generate the corpus level graph, wherein test step nodes common to two or more test cases are merged, connecting (464) the test case graphs; and The computer-implemented method according to claim 2.

5. The test case further includes a text description of one or more preconditions, the corpus level graph further includes one or more precondition nodes (308), each precondition node represents a precondition and is connected to one or more word nodes by an undirected edge, and when there are two or more precondition nodes, the precondition nodes are connected by a directed edge imposing an ordering relationship specifying an ordering between the preconditions represented by the precondition nodes, and the last ordered precondition node is connected by a directed edge to the first ordered test step node of the test case. The computer-implemented method according to claim 2.

6. The test case further includes a textual description of the expected result, the corpus-level graph further includes one or more expected result nodes (332), each expected result node represents an expected result, is connected to one or more word nodes by an undirected edge, and when there are two or more expected result nodes, the expected result nodes are connected by a directed edge imposing an ordering relationship specifying the ordering between the expected results represented by the expected result nodes, and the first ordered expected result node is connected by a directed edge to the last ordered test step node of the test case. The computer-implemented method according to claim 2.

7. associating each test case with one or more of its predicted attributes (132); generating a test plan for testing software, hardware, and / or a system based on the test cases having one or more predicted attributes (140); executing one or more test cases of the test plan to generate test results (148); identifying one or more defects in the software, the hardware, and / or the system based on the generated test results (156); further comprising executing the one or more test cases of the test plan preferably includes testing the software, the hardware, and / or the system on a vehicle and generating test results using signals received from one or more vehicle components. The computer-implemented method according to claim 1.

8. Executing the one or more test plans is preferably automatically performed using one or more automation scripts. The computer-implemented method according to claim 7.

9. A computing system comprising one or more processors and a memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions for performing the computer-implemented method according to any one of claims 1 to 8.

10. The computing system according to claim 9, which is connected to a vehicle and is configured to receive signals from one or more vehicle components. **Claim 11** Use of a computer-implemented method according to any one of claims 1 to 8. **Claim 12** A computer program, a machine-readable storage medium, or a data carrier signal that, when executed on a data processing device and / or a control unit, includes instructions to cause the data processing device and / or the control unit to execute the steps of a computer-implemented method according to any one of claims 1 to 8.

Images