Device and method of automated system test evaluation

Abstract

A method for test evaluation using machine learning models. It collects categorized test set data, generates training sets with artificial tests, and initializes models such as Decision Trees, Random Forests, Gradient-Boosted Decision Trees, and Gaussian Naive Bayes. The models learn from the training sets to evaluate real-world system tests and identify discrepancies when models disagree. The method focuses on electromagnetic compatibility (EMC) testing and increases accuracy by considering the decisions of multiple models to identify suspect tests for further investigation.

Description

[0001] The invention relates to a method for automated system test evaluation using machine learning systems, a computer program and a machine-readable storage medium, as well as a system. Advantages of the invention

[0002] The method offers the advantage of automatically analyzing test measurement data and warning of suspected cases using supervised machine learning technology. Disclosure of the invention

[0003] In a first aspect, the invention relates to a method for automated system test evaluation using machine learning systems.

[0004] The process begins with collecting and grouping test set data from a database, where the test set data includes system tests categorized by system test type and analyzed parameters.

[0005] This is followed by a step of creating training sets for these system tests, where the training sets consist of artificially generated correct and incorrect system tests derived from referential system tests.

[0006] This is followed by a step to initialize several machine learning models, including a decision tree model, a random forest model, a gradient boosted decision tree model, and a Gaussian Naive Bayes model.

[0007] This is followed by a step of training these machine learning models with these training sets to learn patterns and features of system tests.

[0008] This is followed by a step of evaluating real system tests using these trained machine learning models to determine the correctness or suspect nature of these real system tests.

[0009] This is followed by a step in which system tests are flagged as suspect if these machine learning models produce different decisions that indicate possible deviations.

[0010] Further advantageous embodiments are described in the dependent claims.

[0011] Embodiments of the invention are explained in more detail with reference to the following figures. The figures show: Fig. a schematic flow diagram of an embodiment of the invention; Description of the embodiments

[0012] Fig. shows an example of a method for automated system test evaluation using machine learning. The method is used to assess the accuracy of system tests by analyzing test set data retrieved from a database or directly from a measuring device.

[0013] Essentially, the method includes several steps of data collection, training set creation, initialization of the machine learning model, decision making, and identification of suspicious system tests.

[0014] The procedure preferably begins with data acquisition and test set grouping (S21).

[0015] The data can come from a specific database. The database contains various system tests, categorized by the type of system test and the parameter to be analyzed. These tests form the basis for training machine learning models to evaluate the accuracy of future system tests.

[0016] This means that the database contains different test sets depending on the type of system test and the parameters analyzed.

[0017] After step S21, a training set generation is performed (S22).

[0018] To train accurate machine learning models, appropriate training sets must be created. The amount of existing system test data may be insufficient to effectively train a model. Therefore, this step during the training set creation process artificially generates correct and incorrect system tests. These artificially generated tests are based on referential system tests.

[0019] After step S22, training of machine learning models is carried out (S23)

[0020] To find the best solution, several machine learning models are initialized. The following models are initialized to find the best solution: - Decision tree: A hierarchical model that makes decisions through a series of questions based on input features. - Random Forest: An ensemble method that combines multiple decision trees to improve prediction accuracy. - Gradient Boosted Decision Tree: A boosting technique that successively generates an ensemble of decision trees, with each new tree correcting the errors of the previous ones. - Gaussian Naive Bayes: A probabilistic model based on Bayes' theorem, often used for classification tasks.

[0021] Once the corresponding machine learning models are initialized, they are trained with the artificially generated correct and incorrect system tests. The models learn the patterns and characteristics of these tests to make informed decisions.

[0022] The performance of these models is determined based on the test set to be evaluated.

[0023] After step S23, a system test evaluation step is performed (S24).

[0024] After the training process, the trained models can evaluate real-world system tests to determine their accuracy or suspect nature. Because multiple machine learning models are used, the software considers the decisions of all models. In cases where different models lead to contradictory decisions, the system test is flagged as suspect.

[0025] After step S24, a suspicious system test investigation can be performed (S25).

[0026] Identified suspicious system tests are flagged for further investigation. The software does not rely on the decision of a single model to reduce false positives. Instead, it leverages the collective input of multiple models to increase the accuracy and reliability of flagging suspicious tests.

[0027] The presented automated system test evaluation method provides a comprehensive approach for evaluating the correctness of system tests by applying various machine learning models. By collecting and grouping test set data, creating appropriate training sets, initializing multiple machine learning models, and leveraging their joint decisions, the method helps identify suspect system tests that warrant further investigation.

Claims

[1] A computer-implemented method for automated system test evaluation using machine learning systems, in particular for predictive maintenance to verify calibration, consisting of the following steps: Collecting and grouping test set data from a database, where the test set data consists of system tests classified by system test type and analyzed parameters; Creation of training sets for these system tests, where the training sets consist of artificially generated correct and incorrect system tests derived from referential system tests; Initialization of several machine learning models, including a decision tree model, a random forest model, a gradient boosted decision tree model, and a Gaussian Naive Bayes model; Training these machine learning models using these training sets to learn patterns and features of system tests; Evaluating real system tests using these trained machine learning models to determine the accuracy or suspect nature of these real system tests; Flag system tests as suspicious if these machine learning models produce different decisions that indicate possible deviations. [2] The method of claim 1, wherein the system tests are retrieved from the database containing a plurality of test sets with different system test types and analyzed parameters. [3] The method of claim 1 or 2, wherein said training sets are generated by creating artificial true and false system tests, said artificial tests being extensions of referential system tests. [4] The method according to any one of the preceding claims, wherein the decision tree model uses a hierarchical structure to make decisions based on input features, wherein the random forest model uses an ensemble approach by combining multiple decision trees to improve prediction accuracy, wherein the gradient boosted decision tree model sequentially generates an ensemble of decision trees, with subsequent trees correcting errors of previous trees, wherein the Gaussian naive Bayes model applies probabilistic analysis based on Bayes' theorem to classify system tests. [5] The method according to any one of the preceding claims, wherein said system tests comprise electromagnetic compatibility (EMC) tests and said training sets are tailored to simulate correct and incorrect EMC system tests using referential EMC system tests as a basis. [6] The method according to any one of the preceding claims, wherein the evaluation of real system tests uses the collective decisions of said machine learning models to determine the correctness or suspect nature of said real system tests and, where appropriate, further investigate suspect system tests to verify their correctness or to identify possible deviations. [7] The method of any preceding claim, further comprising flagging identified suspect system tests for later investigation. [8] A computer-readable medium having stored thereon computer-executable instructions which, when executed by a computer system, carry out the method according to claims 1-7. [9] A computer system programmed to carry out the method according to any one of claims 1-7. [10] System configured to perform one of the methods of claims 1 to 7.

Images