Test evaluation system

Abstract

The test evaluation system includes: a first extraction AI model that extracts a description related to a pre-defined evaluation viewpoint from a description of a test condition input to a first input unit; a second extraction AI model that extracts a description related to the evaluation viewpoint from a description of a test procedure or a test command input to a second input unit; an evaluation item synthesizing unit that synthesizes the description of the test condition extracted by the first extraction AI model and the description of the test procedure or the like extracted by the second extraction AI model as an evaluation item of the evaluation viewpoint; and an evaluation AI model that evaluates consistency between the description of the test condition and the description of the test procedure or the like for the evaluation item of the evaluation viewpoint synthesized by the evaluation item synthesizing unit.

Description

Technical Field

[0001] This invention relates to a testing and evaluation system. Background Technology

[0002] Technologies for evaluating software are under development. For example, Patent Document 1 discloses a method that automatically organizes information that will serve as all benchmarks into a hierarchical model to evaluate the adaptability of an architecture analysis and design model. In the technology of Patent Document 1, users can determine whether adaptability and reliability are met by examining the hierarchical model structured based on the required input information generated from the model. Furthermore, Patent Document 2 discloses a method that evaluates the reliability of software based on predefined items such as defect rate, uptime, and crash rate.

[0003] Prior technology documents

[0004] Patent documents

[0005] Patent Document 1: US Patent No. 7,480,893

[0006] Patent Document 2: Description of Chinese Patent Application Publication No. 114489760 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] However, in the functional testing of software and hardware, the satisfaction of the testing criteria corresponding to the software and hardware functions is confirmed by executing testing procedures or testing commands corresponding to the testing conditions. These testing procedures or commands are generated by humans or AI (Artificial Intelligence). To properly perform the functional testing of software and hardware, it is necessary to evaluate the consistency between the testing conditions and the generated testing procedures or commands. However, since the testing conditions, testing procedures, and testing commands are described in natural language, the techniques in Patent Documents 1 and 2 are insufficient to address this, and improvements are required.

[0009] The present invention was made in view of the above-mentioned points, and its object is to provide a test evaluation system capable of assessing the consistency of any of the test conditions described in natural language, test steps described in natural language, and test commands described in natural language.

[0010] Methods for solving problems

[0011] The test evaluation system of the present invention for solving the above-mentioned problems is characterized by comprising: a first input unit that inputs a record of test conditions; a second input unit that inputs a record of either a test step or a test command generated based on the record of test conditions; a first extraction AI model that extracts a record related to a predefined evaluation viewpoint from the record of test conditions input to the first input unit; a second extraction AI model that extracts a record related to the evaluation viewpoint from the record of either a test step or a test command input to the second input unit; an evaluation item synthesis unit that synthesizes the record of test conditions extracted by the first extraction AI model with the record of either a test step or a test command extracted by the second extraction AI model, and uses it as an evaluation item for the evaluation viewpoint; and an evaluation AI model that evaluates the consistency between the record of test conditions and the record of either a test step or a test command for the evaluation item of the evaluation viewpoint synthesized by the evaluation item synthesis unit.

[0012] Invention Effects

[0013] According to the present invention, it is possible to evaluate the consistency of any of the test steps and test commands described in natural language for test conditions described in natural language.

[0014] Further features related to this invention will become clear from the description and drawings herein. Furthermore, issues, configurations, and effects other than those described above will become clear from the following description of embodiments. Attached Figure Description

[0015] Figure 1 This is a block diagram illustrating the overview of the test and evaluation system of the first embodiment.

[0016] Figure 2 It means Figure 1 A block diagram detailing the testing and evaluation system.

[0017] Figure 3 It means Figure 2 The first step is to extract the AI ​​model, and the second step is to extract the details of the AI ​​model.

[0018] Figure 4 It indicates composition Figure 3 A block diagram showing the structure of the first AI extraction model and the encoder of the second AI extraction model.

[0019] Figure 5 It indicates composition Figure 4 A graph of scaled dot-product attention in a multi-head self-attention layer.

[0020] Figure 6 It means by Figure 5 The graph is composed of multi-head self-attention layers formed by scaling dot product attention.

[0021] Figure 7 This is a diagram illustrating an example of the test conditions recorded when the first AI model is extracted as input.

[0022] Figure 8 It means decomposing by each word element. Figure 7 The diagram shows the recorded state of the test conditions.

[0023] Figure 9 It means targeting Figure 8 A graph showing the state of probability related to the "pre-event environment" for each word element in the calculation and evaluation viewpoint.

[0024] Figure 10 It means targeting Figure 8 A graph showing the state of probability related to the "observation variable" for each word element in the calculation and evaluation of the viewpoint.

[0025] Figure 11 It means according to Figure 9 as well as Figure 10 The probabilities calculated in the graph are combined with the state of terms related to the same evaluation viewpoint, "pre-event environment" and "observed variables".

[0026] Figure 12 It is a record of the test steps for extracting input to the second AI model, combined with a graph of the state of terms related to the same evaluation viewpoints "pre-conference environment" and "observed variables".

[0027] Figure 13 This diagram illustrates the preparation of training data for the first and second AI extraction models in the first embodiment.

[0028] Figure 14 It means Figure 2 A flowchart detailing the project's overall assessment.

[0029] Figure 15 It means Figure 2 A diagram detailing the evaluation of the AI ​​model.

[0030] Figure 16 It means Figure 15 A diagram of a multilayer perceptron.

[0031] Figure 17 This is a diagram representing the training data of the evaluation AI model of the first embodiment, which includes affirmative data with a reliability score of "okay".

[0032] Figure 18This is a diagram representing the training data of the evaluation AI model of the first embodiment, which includes negative data with a reliability score of "unacceptable".

[0033] Figure 19 This is a block diagram illustrating the details of the test and evaluation system of the second embodiment.

[0034] Figure 20 It means Figure 19 The first step is to extract the AI ​​model, and the second step is to extract the details of the AI ​​model.

[0035] Figure 21 It is a record of the test conditions for extracting input to the first AI model, and a graph that extracts the states of terms related to the same evaluation viewpoints "voltage" and "current".

[0036] Figure 22 This is a record of the test steps for extracting input to the second AI model, and a diagram showing the state of terms related to the same evaluation viewpoints "voltage" and "current".

[0037] Figure 23 This diagram illustrates the preparation of training data for the first and second AI extraction models in the second embodiment.

[0038] Figure 24 This is a diagram representing the training data of the evaluation AI model of the second embodiment, which includes affirmative data with a reliability score of "okay".

[0039] Figure 25 This is a graph representing the training data of the evaluation AI model of the second embodiment, which includes negative data with a reliability score of "unacceptable".

[0040] Figure 26 This is a block diagram illustrating the overview of the test and evaluation system of the third embodiment.

[0041] Figure 27 This is a block diagram illustrating the overview of the test and evaluation system of the fourth embodiment.

[0042] Figure 28 This is a diagram illustrating an example of pattern matching using the regular expressions of the fourth embodiment.

[0043] Figure 29 This is a diagram illustrating an example of pattern matching using the regular expressions of the fourth embodiment.

[0044] Figure 30 This is a diagram illustrating an example of pattern matching using the regular expressions of the fourth embodiment. Detailed Implementation

[0045] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Furthermore, the same reference numerals are used for the same elements and processing contents, and their descriptions are omitted.

[0046] [First Embodiment]

[0047] The test and evaluation system of the first embodiment will be described below. Figure 1 The test evaluation system 1A shown is configured as a computer, for example, for evaluating the functionality of the software and hardware in a vehicle's inverter ECU (Electronic Control Unit). The test evaluation system 1A performs the following processing by executing computer programs stored in the ROM (Read Only Memory) and RAM (Random Access Memory) of the test evaluation system 1A via its CPU (Central Processing Unit). The test evaluation system 1A evaluates whether any of the test criteria corresponding to the functionality of the software and hardware, the test procedures corresponding to the test criteria, and the test commands corresponding to the test criteria are consistent.

[0048] "Test conditions" refer, for example, to the conditions used to pass a test associated with a predefined evaluation viewpoint. "Test steps" refer, for example, to the steps used to confirm whether the test conditions associated with a predefined evaluation viewpoint are met. "Test commands" refer, for example, to the program source commands and source code used to confirm whether the test conditions associated with a predefined evaluation viewpoint are met. Among the evaluation viewpoints associated with test conditions, test steps, and test commands, as described later, are, for example, "pre-existing environment," "observed variables," "voltage," and "current," etc.

[0049] When the evaluation perspective is "pre-test environment," "test conditions" refer, for example, to the environment that must be met before the test is executed. When the evaluation perspective is "pre-test environment," "test steps" refer, for example, to the steps used to meet the environment that must be met before the test is executed. When the evaluation perspective is "pre-test environment," "test commands" refer, for example, to the commands and source code that execute the steps used to meet the environment that must be met before the test is executed.

[0050] More specifically, when the evaluation perspective is "pre-event environment," the "test conditions" are, for example, described as "initializing the current value." When the evaluation perspective is "pre-event environment," the "test steps" are, for example, described as "setting the current variable to 10." When the evaluation perspective is "pre-event environment," the "test command" is, for example, the command and source code of the program source used to cause the computer to perform the process of "setting the current variable to 10."

[0051] When the evaluation viewpoint is an "observed variable," "test conditions" refer, for example, to the values ​​of the observed object, variable, and threshold, etc., in order to pass the test. When the evaluation viewpoint is an "observed variable," "test steps" refer, for example, to the steps of setting the values ​​of the observed object, variable, and threshold, etc., in order to pass the test. When the evaluation viewpoint is an "observed variable," "test commands" refer, for example, to the commands and source code of the program source used to cause the computer to execute processes that set the values ​​of the observed object, variable, and threshold, etc., in order to pass the test.

[0052] More specifically, when the evaluation viewpoint is an "observation variable," the "test conditions" are, for example, records specifying the observation object such as "observe the condition of test system A." When the evaluation viewpoint is an "observation variable," the "test steps" are, for example, records such as "observation variable settings: 1. Current 2. Transmission 3. Temperature." When the evaluation viewpoint is an "observation variable," the "test command" is, for example, the program source command and source code used to cause the computer to execute the processing of settings such as "observation variable settings: 1. Current 2. Transmission 3. Temperature."

[0053] When the evaluation focus is "voltage," the "test conditions" are, for example, described as "low voltage source within normal range." When the evaluation focus is "voltage," the "test procedure" is, for example, described as "activating the high-voltage direct current transmission switch." When the evaluation focus is "voltage," the "test command" is, for example, the command and source code of the program source used to cause the computer to execute the process of "activating the high-voltage direct current transmission switch."

[0054] When the evaluation focus is "current," the "test conditions" are described, for example, "DC bias." When the evaluation focus is "current," the "test procedures" are described, for example, "DC bias calculation." When the evaluation focus is "current," the "test commands" are, for example, commands and source code used to cause the computer to perform processes such as "DC bias calculation."

[0055] like Figure 1As shown, the test evaluation system 1A includes a first input unit 119, a second input unit 129, a first extraction AI model 130A, a second extraction AI model 140A, an evaluation item integration unit 150, and an evaluation AI model 170A. The first input unit 119 inputs a record of test conditions 110a. The second input unit 129 inputs a record of test steps or test commands 120a generated based on the test conditions 110a. The test steps or test commands 120a are generated by a human or AI (Artificial Intelligence) based on the test conditions 110a. Furthermore, the records of test conditions 110a and test steps or test commands 120a refer to the characters, words, phrases, paragraphs, and sentences of natural language constituting the test conditions 110a and test steps or test commands 120a.

[0056] The first extraction AI model 130A extracts records related to a predefined evaluation viewpoint from the records of test conditions 110a input to the first input unit 119. The second extraction AI model 140A extracts records related to the evaluation viewpoint from the records of test steps or test commands 120a input to the second input unit 129.

[0057] The evaluation project integration department 150 integrates the records of test conditions 110a extracted by the first extraction AI model 130A and the records of test steps or test commands 120a extracted by the second extraction AI model 140A, and uses them as the evaluation project 160.

[0058] For the evaluation project 160, which is based on the evaluation viewpoints synthesized by the evaluation project integration department 150, the consistency between the recorded test conditions 110a and the recorded test procedures or test orders 120a is evaluated as a reliability score 180a.

[0059] The following describes an example of a test evaluation system 1A where the evaluation items are "pre-event environment" and "observed variables". However, the evaluation items "pre-event environment" and "observed variables" in this embodiment are merely examples and can be other evaluation items. The number of evaluation items can also be arbitrarily set. For example... Figure 2 As shown, the first extraction AI model 130A extracts records related to each of multiple evaluation views from the records of test conditions 110a input to the first input unit 119. Figure 2 In the example, the first extraction AI model 130A extracts test conditions (pre-environment) 111a as records related to the evaluation viewpoint "pre-environment". Additionally, the first extraction AI model 130A extracts test conditions (observation variables) 112a as records related to the evaluation viewpoint "observation variables".

[0060] The second extraction AI model 140A extracts records related to each of multiple evaluation viewpoints from the records of test steps or test commands 120a input to the second input unit 129. Figure 2 In the example, the second extraction AI model 140A extracts test steps or test commands (pre-examination environment) 121a that are recorded as related to the evaluation viewpoint "pre-examination environment". Additionally, the second extraction AI model 140A extracts test steps or test commands (observation variables) 122a that are recorded as related to the evaluation viewpoint "observation variables".

[0061] The evaluation project integration department 150 integrates, for each of the multiple evaluation viewpoints, the records of test conditions related to the evaluation viewpoint extracted by the first extraction AI model 130A for each evaluation viewpoint, and the records of test steps and test commands related to the evaluation viewpoint extracted by the second extraction AI model 140A for each evaluation viewpoint. The evaluation project integration department 150 uses the integrated records of test conditions and test steps and test commands for each of the multiple evaluation viewpoints as the evaluation project 160a for each evaluation viewpoint.

[0062] exist Figure 2 In the example, the evaluation project integration department 150 integrates the test conditions (pre-environment) 111a and test procedures or test orders (pre-environment) 121a related to the evaluation viewpoint "pre-environment" as evaluation item 161a for the evaluation viewpoint "pre-environment". The evaluation project integration department 150 integrates the test conditions (observation variables) 112a and test procedures or test orders (observation variables) 122a related to the evaluation viewpoint "observation variables" as evaluation item 162a for the evaluation viewpoint "observation variables".

[0063] The evaluation AI model 170A consists of multiple evaluation AI models (pre-environment) 171A and evaluation AI models (observation variables) 172A corresponding to the evaluation viewpoints "pre-environment" and "observation variables" integrated by the evaluation project integration department 150.

[0064] For the evaluation items (pre-event environment) 161a of the evaluation viewpoint "pre-event environment", the consistency between the description of the test conditions (pre-event environment) 111a and the description of the test steps or test commands (pre-event environment) 121a is evaluated as a reliability score (pre-event environment) 181a, which is indicated by either (allow / 1) or (not allowed / 0).

[0065] For the evaluation items (observation variables) 162a of the evaluation viewpoint "observation variables", the consistency between the recorded test conditions (observation variables) 112a and the recorded test steps or test commands (observation variables) 122a is evaluated as a reliability score (observation variable) 182a, which is indicated by either (yes / 1) or (no / 0).

[0066] The following describes the structure of each part of the test and evaluation system 1A. For example... Figure 3 As shown, the first extraction AI model 130A extracts test conditions (pre-event environment) 111a, which are records related to the evaluation viewpoint "pre-event environment", from the test conditions 110a input to the first input unit 119. Additionally, the first extraction AI model 130A extracts test conditions (observation variables) 112a, which are records related to the evaluation viewpoint "observation variables", from the test conditions 110a input to the first input unit 119.

[0067] The second extraction AI model 140A extracts test steps or test commands (pre-event environment) 121a from the records of test steps or test commands 120a input to the second input unit 129, which are records related to the evaluation viewpoint "pre-event environment". Additionally, the second extraction AI model 140A extracts test steps or test commands (observation variables) 122a from the records of test steps or test commands 120a input to the second input unit 129, which are records related to the evaluation viewpoint "observation variables".

[0068] A classification model is used in the first extraction AI model 130A and the second extraction AI model 140A. More specifically, the first extraction AI model 130A and the second extraction AI model 140A, for example, constitute a classification model with... Figure 4 The encoder 300 is a multi-head self-attention layer transformer. In this transformer, instead of processing each record word-by-word for the test conditions 110a and test steps or test commands 120a, all records are processed at once. Therefore, it is possible to quickly understand which parts of the record are important and the overall context of the record. Furthermore, by processing all records at once, the transformer enables parallel computation, thus improving computational speed. Additionally, as described later, the transformer analyzes how each part of the record relates to other parts through attention. Therefore, the meaning of the record can be accurately grasped.

[0069] The encoder 300 includes an input embedding layer 301, a position encoding layer 302, a stack 303, a multi-head self-attention layer 304, an additive normalization layer 305, a feedforward layer 306, and an additive normalization layer 307.

[0070] The Input Embedding Layer 301 transforms each word of the input test condition 110a and test step or test command 120a into a form that the first extraction AI model 130A and the second extraction AI model 140A can compute. The Input Embedding Layer 301 uses a learned transformation model to transform the input lexical units into vectors. Here, a lexical unit refers to the smallest unit of the string containing the test condition 110a and test step or test command 120a processed by the first extraction AI model 130A and the second extraction AI model 140A.

[0071] The Positional Encoding Layer 302 assigns information to the vector transformed by the Input Embedding Layer 301 to represent the position of each lexical term in the record of the test condition 110a and the test step or test command 120a.

[0072] Stack 303 has a construction of Nx identical layers stacked together. Figure 4 The diagram only shows one layer of stack 303; descriptions of the other layers are omitted. Each stack 303 has a multi-head self-attention layer 304, an additive normalization layer 305, a feedforward layer, and an additive normalization layer 307.

[0073] The multi-head self-attention layer 304 assigns multiple attention points to the vector transformed by the input embedding layer 301. Attention is used to direct attention to information of important parts.

[0074] Attention is comprised of a query Q, a key K, values ​​V, and output values. Query Q is the target of the computation. Key K is a set of word vectors used to calculate similarity with other lexical units. That is, key K represents relationships with other lexical units. Value V is a set of vectors used for weighted summation of lexical units. That is, value V represents the lexical unit itself. Output values ​​are all vectors. The output values ​​are calculated as a weighted sum of values ​​V. The weights assigned to each value V are calculated using a compatibility function with the query Q corresponding to the key K.

[0075] Multi-head self-attention layer 304 as Figure 5 The scaled dot-product attention structure 310 is shown below. The inputs to the scaled dot-product attention structure 310 are the query Q, the key K, and the value V. The dimension of the key K is d. k The dimension of value V is d.v In scaling dot product attention 310, in matrix multiplication 311, the dot product of the vectors of query Q and key K is calculated. In scaling 312, the dot product of the vectors of query Q and key K is divided by |(d k ) 1 / 2 |

[0076] In Mask 313, a mask is applied to the padding areas when the number of input words is shorter than the sequence length. The results of the matrix multiplication 311, scaling 312, and mask 313 are applied to the softmax function 314. The total weights are normalized to 1 by the softmax function 314. In Matrix Multiplication 315, the softmax function 314 is calculated as the dot product of the vector of values ​​V. The above calculations in the scaled dot product attention 310 are represented by the following equation (1).

[0077] [Number 1]

[0078]

[0079] like Figure 6 As shown, a multi-head self-attention layer 304 is constructed using multiple scaled dot product attention layers 310. By reducing the respective dimensions of the value V, key K, and query Q using linear layers 321, 322, and 323, which are the same number as the scaled dot product attention layers 310, computational speed is achieved.

[0080] The computational results in the scaled dot product attention 310 are concatenated through a concatenate layer 324. The computational results concatenated by the concatenate layer are then projected onto another dimension through a linear layer 325 to obtain the final output value of the multi-head self-attention layer 304.

[0081] like Figure 4 As shown, the add and normalization layer 305 of the encoder 300 adds the input to the multi-head self-attention layer 304 to the output of the multi-head self-attention layer 304, and normalizes the output of the multi-head self-attention layer 304.

[0082] The feed-forward layer has 306 pairs of input vectors x at position t.t A feedforward fully connected network is applied. The feedforward layer 306 consists of, for example, two layers. The dimension of the first layer is set to, for example, four times the dimension of the input. The second layer returns the output value to the original input dimension. The weights W1 and W2 are the same regardless of the position t. This ensures that the same words input to the feedforward layer 306 produce the same output. The ReLU (Rectified Linear Unit) activation function is applied in the feedforward layer 306. The feedforward layer 306 is relative to the input vector x. t The output FFN(x) t ) is represented by the following formula (2).

[0083] [Number 2]

[0084]

[0085] The add and normalization layer 307 adds the input to the feedforward layer 306 to the output of the feedforward layer 306 and normalizes the output of the feedforward layer 306. Furthermore, in addition to the encoder 300 constituting the converter, an LSTM (Long Short Term Memory) type encoder model can also be considered as the first AI extraction model 130A and the second AI extraction model 140A.

[0086] The specific processing in the first AI extraction model 130A and the second AI extraction model 140A will be explained below. As a representative example, the operation of the first AI extraction model 130A will be explained below. Assume that the following input is given to the first input unit 119: Figure 7 The test conditions described in 110a are as shown. (As indicated...) Figure 8 As shown, the first extraction AI model 130A will input the test condition 110a recorded to the first input unit 119, according to... Figure 8 Each word element shown by the dashed line is decomposed.

[0087] like Figure 9 As shown, the first extraction AI model 130A calculates the probability of each word being associated with each of the multiple evaluation viewpoints. For example, in Figure 9 In the example, regarding the evaluation viewpoint "pre-event environment," the probability of the term "pre-event environment" with the same title is as high as 0.95, and the probabilities of "(d)," "AAA…," "BBB…," and "CCC…" before and after it are also calculated to be high. Figure 10In the example, regarding the evaluation viewpoint "observed variable," the term "success benchmark" in the title is not the same as the title "observed variable." However, the "success benchmark" section is highly relevant because it usually contains the observed variable that needs to be observed in the test. Therefore, the probability for the term "success benchmark" is calculated as high as 0.85. The probabilities of "(f)," "DDD…," "EEE…," "FFF…," "GGG…," and "HHH…" before and after "success benchmark" are also calculated high.

[0088] like Figure 11 As shown, the first extraction AI model 130A calculates probabilities to determine which of the multiple evaluation viewpoints each lexical unit is associated with, and combines lexical units associated with the same evaluation viewpoint. Figure 11 In the example, the evaluation viewpoint "pre-event environment" is combined with "(d) pre-event environment...CCC", and the evaluation viewpoint "observed variable" is combined with "(f) success benchmark...HHH". Through the above processing, the first extraction AI model 130A extracts the records related to the evaluation viewpoint from the records of the test conditions 110a input to the first input unit 119, according to each of the multiple evaluation viewpoints.

[0089] exist Figure 11 In the example, regarding the evaluation viewpoint "pre-event environment", "(d) pre-event environment...CCC" was extracted as the test condition (pre-event environment) 111a. Regarding the evaluation viewpoint "observation variable", "(f) success benchmark...HHH" was extracted as the test condition (observation variable) 112a.

[0090] Similarly, the second extraction AI model 140A decomposes the record of the test steps or test commands 120a input to the second input unit 129 for each word. The second extraction AI model 140A determines which of the multiple evaluation views each word is associated with by calculating the probability of its association with each of the multiple evaluation views for each word. For example... Figure 12 As shown, the second extraction AI model 140A extracts records related to each of multiple evaluation views from the records of test steps or test commands 120a input to the second input unit 129 by combining lexical units related to the same evaluation view.

[0091] exist Figure 12 In the example, regarding the evaluation viewpoint "pre-event environment," the records following the title "pre-event environment" are extracted as test steps or test commands (pre-event environment) 121a. Regarding the evaluation viewpoint "observed variable," the records following the title "observed variable" are extracted as test steps or test commands (observed variable) 122a.

[0092] like Figure 13As shown, the first extraction AI model 130A will be trained using records related to each of the multiple evaluation viewpoints extracted from the records of test condition 110a as training data. Figure 13 In the example, the first training data preparation unit (test condition) 131A, which is constructed in the same way as the first extraction AI model 130A described above and has been trained, is used.

[0093] The first training data preparation unit (test conditions) 131A extracts test conditions (pre-existing environment) 111a and test conditions (observation variables) 112a from the record of test conditions 110a, in the same manner as described above. Test conditions 110a, test conditions (pre-existing environment) 111a, and test conditions (observation variables) 112a are provided as training data for the first extracted AI model 130A.

[0094] The second extraction AI model 140A will be trained using records related to each of the multiple evaluation views extracted from the records of the test steps or test commands 120a as training data. Figure 13 In the example, a second training data preparation unit (test step or test command) 141A, which is constructed in the same way as the second extraction AI model 140A described above and has been trained, is used.

[0095] The second training data preparation unit (test step or test command) 141A extracts the test step or test command (pre-test environment) 121a and the test step or test command (observation variable) 122a from the record of the test step or test command 120a, in the same manner as described above. The test step or test command 120a, the test step or test command (pre-test environment) 121a, and the test step or test command (observation variable) 122a are provided as training data for the second AI model 140A.

[0096] like Figure 14 As shown, in the comprehensive evaluation project department 150, the input is a record of the test conditions related to each evaluation viewpoint extracted by the first extraction AI model 130A. Figure 14 In the example, for the evaluation viewpoint "pre-event environment", input test condition (pre-event environment) 111a, and for the evaluation viewpoint "observation variable", input test condition (observation variable) 112a.

[0097] In the project evaluation integration section 150, the records of test steps or test commands related to each evaluation viewpoint extracted by the second extraction AI model 140A are entered. Figure 14In the example, for the evaluation viewpoint "Pre-event environment", enter the test step or test command (Pre-event environment) 121a; for the evaluation viewpoint "Observed variable", enter the test step or test command (Observed variable) 122a.

[0098] The Evaluation Project Integration Department 150 integrates the test conditions and test procedures or test orders extracted for each evaluation viewpoint, and outputs them as evaluation items for each evaluation viewpoint. In other words, the Evaluation Project Integration Department 150 reassembles the test conditions and test procedures or test orders extracted for each evaluation viewpoint into a group for each evaluation viewpoint.

[0099] exist Figure 14 In the example, as evaluation item (pre-environment) 161a, a set of test conditions (pre-environment) 111a and test steps or test commands (pre-environment) 121a is output. Additionally, as evaluation item (observation variable) 162a, a set of test conditions (observation variable) 112a and test steps or test commands (observation variable) 122a is output. Evaluation item (pre-environment) 161a output by evaluation item integration unit 150 is supplied to evaluation AI model (pre-environment) 171A. Evaluation item (observation variable) 162a output by evaluation item integration unit 150 is supplied to evaluation AI model (observation variable) 172A.

[0100] The following explanation uses AI model evaluation (pre-conference environment) 171A as a representative example of AI model evaluation 170A. In evaluating AI model evaluation (pre-conference environment) 171A and other AI models 170A, a classification model is used. More specifically, such as... Figure 15 As shown, the evaluation AI model (pre-evaluation environment) 171A for evaluating AI model 170A includes, for example, encoder 173, encoder 174, and multilayer perceptron 175. The test conditions (pre-evaluation environment) 111a for the evaluation item (pre-evaluation environment) 161a are input into encoder 173, and the test steps or test commands (pre-evaluation environment) 121a for the evaluation item (pre-evaluation environment) 161a are input into encoder 174. Since the configuration of encoders 173 and 174 is the same as that of encoder 300 constituting the aforementioned converter, description is omitted.

[0101] like Figure 16 As shown, the multilayer perceptron 175 includes an input layer 401, a hidden layer 402, and an output layer 403, each having one or more nodes 404. Each node 404 of the input layer 401 is connected to each node 404 of the hidden layer 402 via an edge 405.

[0102] Each node 404 of the hidden layer 402 is connected to a node 404 of the output layer 403 via an edge 405. Signals (information) propagating through the edge 405 propagate unidirectionally from the node 404 of the input layer 401 to the node 404 of the hidden layer 402. Signals propagating through the edge 405 also propagate unidirectionally from the node 404 of the hidden layer 402 to the node 404 of the output layer 403.

[0103] The signals output from node 404 of input layer 401 and propagating in edge 405 are each multiplied by weights and input to node 404 of hidden layer 402. Similarly, the signals output from node 404 of hidden layer 402 and propagating in edge 405 are each multiplied by weights and input to node 404 of output layer 403. Each node 404 transforms the input signal through an activation function and outputs it as a new signal.

[0104] Through the processing described above, the evaluation AI model 171A outputs a reliability score for the evaluation item (pre-evaluation environment) 161a, based on the consistency between the recorded test conditions (pre-evaluation environment) 111a and the recorded test steps or test commands (pre-evaluation environment) 121a, as indicated by either (allowable / 1) or (unallowable / 0). The same applies to the evaluation AI model 172A for the evaluation AI model 170A. Furthermore, in addition to the encoders 173 and 174 constituting the converter, an LSTM-type encoder model can also be considered as the classification model used in the evaluation AI model 170A.

[0105] The training data for evaluating AI model 170A is described below. AI model 170A is trained using a set of training data, which includes positive data where the description of test condition 110b is consistent with the description of test step or test command 120b and has a reliability score of (okay / 1); and negative data where the description of test step or test command 120b is inconsistent with the description of test step or test command 120b and has a reliability score of (unacceptable / 0).

[0106] like Figure 17 As shown, the first input AI model (test condition) 130A, which contains the record of test condition 110b, extracts the records of test condition (pre-conference environment) 111b and test condition (observation variable) 112b according to the evaluation viewpoints "pre-conference environment" and "observation variable". The second input AI model (test step or test command) 140A, which contains the record of test step or test command 120b, extracts the records of test step or test command (pre-conference environment) 121b and test step or test command (observation variable) 122b according to the evaluation viewpoints "pre-conference environment" and "observation variable". Furthermore, for ease of explanation, illustrations of the first input unit 119 and the second input unit 129 are omitted.

[0107] The descriptions of test conditions (pre-test environment) 111b and test procedures or test commands (pre-test environment) 121b are set to be related to each other. The descriptions of test conditions (observed variables) 112b and test procedures or test commands (observed variables) 122b are set to be related to each other.

[0108] In this case, the evaluation project 161b, which is integrated by the evaluation project integration department 150, includes a record of test conditions (pre-event environment) 111b that are related to each other, and a record of test procedures or test orders (pre-event environment) 121b. The evaluation AI model (pre-event environment) 171A outputs a reliability score (pre-event environment) 181b, which is represented by (possibly 1), as positive data.

[0109] The evaluation project 162b, which is integrated by the evaluation project integration department 150, includes records of interrelated test conditions (observation variables) 112b and test procedures or test commands (observation variables) 122b. The reliability score (observation variable) 182b, represented by the output of the evaluation AI model (observation variable) 172A, is used as positive data.

[0110] On the other hand, such as Figure 18 As shown, the first extraction AI model (test condition) 130A, which has the record of test condition 110c as input, extracts the records of test condition (pre-conference environment) 111b and test condition (observation variable) 112b, as well as test condition (pre-conference environment) 111c and test condition (observation variable) 112c, according to the evaluation viewpoints "pre-conference environment" and "observation variable". The second extraction AI model (test step or test command) 140A, which has the record of test step or test command 120c as input, extracts the records of test step or test command (pre-conference environment) 121c and test step or test command (observation variable) 122c, according to the evaluation viewpoints "pre-conference environment" and "observation variable".

[0111] The descriptions of test conditions (pre-test environment) 111b and test steps or test commands (pre-test environment) 121c are set to be unrelated to each other. The descriptions of test conditions (observed variables) 112b and test steps or test commands (observed variables) 122c are set to be unrelated to each other.

[0112] The records of test conditions (pre-existing environment) 111b, test conditions (observed variables) 112b, test conditions (pre-existing environment) 111c, and test conditions (observed variables) 112c extracted by the first extraction AI model (test conditions) 130A are input to the similarity confirmation unit 190. In the similarity confirmation unit 190, similar and duplicate records are deleted. The similarity confirmation unit 190 deletes similar and duplicate records through keyword comparison and similarity calculation, etc. Figure 18 In the example, the descriptions of test condition (pre-existing environment) 111c and test condition (observed variable) 112c have been deleted. Furthermore, for ease of explanation, only... Figure 18 as well as Figure 25 The similarity confirmation section 190 in the diagram is shown in the figure. Figure 1 , Figure 2 , Figure 14 , Figure 19 , Figure 24 as well as Figure 27 Illustrations omitted.

[0113] In this case, the evaluation project 161c, which is integrated by the evaluation project integration department 150, includes records of test conditions (pre-event environment) 111b that are not related to each other, and records of test procedures or test orders (pre-event environment) 121c. The reliability score (pre-event environment) 181c, which is represented by the evaluation AI model (pre-event environment) 171A (not possible / 0), is used as negative data.

[0114] The evaluation project 162c, synthesized by the evaluation project integration department 150, includes records of test conditions (observation variables) 112b that are not related to each other, and records of test procedures or test commands (observation variables) 122c. The evaluation AI model (observation variable) 172A outputs a reliability score (observation variable) 182c representing (not possible / 0) as negative data. As described above, a set of training data containing positive data with a reliability score of (possible / 1) and negative data with a reliability score of (not possible / 0) can be obtained.

[0115] As described above, in previous techniques, it was difficult to evaluate the consistency of test steps or test commands 120a written in natural language for test conditions 110a. This is because in large-scale language models (LLMs) of the GPT (Chat Generative Pre-trained Transformer) type, only the next word of the input record is predicted, making the processing of records written in natural language more difficult. Furthermore, in recent years, test conditions 110a and test steps or test commands 120a have tended to become more numerous and complex, making the evaluation of the consistency of test steps or test commands 120a for test conditions 110a even more difficult.

[0116] On the other hand, in the test evaluation system 1A of this embodiment, the first extraction AI model 130A extracts the records related to the predefined evaluation viewpoint from the records of test conditions 110a recorded in natural language, and the second extraction AI model 140A extracts the records related to the predefined evaluation viewpoint from the records of test steps or test commands 120a recorded in natural language. Therefore, it is able to handle test conditions 110a and test steps or test commands 120a recorded in natural language.

[0117] The evaluation project integration unit 150 integrates the records of test conditions 110a extracted by the first extraction AI model 130A and the records of test steps or test commands 120a extracted by the second extraction AI model 140A, and uses this as the evaluation project 160a from the evaluation perspective. The evaluation AI model 170A evaluates the consistency between the records of test conditions 110a and test steps or test commands 120a for the evaluation project 160a from the evaluation perspective integrated by the evaluation project integration unit 150. Therefore, it is possible to evaluate the consistency of test steps or test commands 120a recorded in natural language for test conditions 110a recorded in natural language.

[0118] In addition, in this embodiment, the first extraction AI model 130A extracts records related to the evaluation viewpoints "pre-event environment" and "observation variable" from the records of test conditions 110a, according to each of the multiple evaluation viewpoints "pre-event environment" and "observation variable". The second extraction AI model 140A extracts records related to the evaluation viewpoints "pre-event environment" and "observation variable" from the records of test steps or test commands 120a, according to each of the multiple evaluation viewpoints "pre-event environment" and "observation variable".

[0119] The project integration department 150 integrates the records of test conditions 110a related to the evaluation viewpoints "pre-evaluation environment" and "observation variables" extracted by the first extraction AI model 130A according to the evaluation viewpoints "pre-evaluation environment" and "observation variables", and the records of test steps or test commands 120a related to the evaluation viewpoints "pre-evaluation environment" and "observation variables" extracted by the second extraction AI model 140A according to the evaluation viewpoints "pre-evaluation environment" and "observation variables".

[0120] The evaluation AI model 170A consists of multiple evaluation AI models (pre-environment) 171A and evaluation AI model 172A corresponding to the evaluation viewpoints "pre-environment" and "observation variables" synthesized by the evaluation project integration department 150, respectively.

[0121] Multiple evaluation AI models (pre-conference environment) 171A and 172A, for the evaluation viewpoint "pre-conference environment" and "observation variable", respectively evaluate the consistency between the record of test condition 110a and the record of test procedure or test command 120a as a reliability score indicated by either accessibility or unaccessibility.

[0122] That is, by segmenting the evaluation by multiple evaluation viewpoints, the evaluation AI model 170A can fully understand the evaluation rules for each evaluation viewpoint from the training data, thus enabling the calculation of a more accurate reliability score. Furthermore, when processing common features by each evaluation viewpoint, the relationships become complex and difficult to identify; therefore, by segmenting the evaluation by multiple evaluation viewpoints, the accuracy of identifying the evaluation rules is improved. Additionally, by evaluating the consistency between the recorded test condition 110a and the recorded test steps or test commands 120a at each of the multiple evaluation viewpoints, problematic areas in the test can be identified. Moreover, in this embodiment, multiple evaluation viewpoints can be processed in parallel with limited resources. Therefore, even for a large number of complex test conditions 110a and test steps or test commands 120a, accurate evaluation can be performed in a short time.

[0123] The first extraction AI model 130A and the second extraction AI model 140A are encoders 300 constituting the converter. Therefore, thanks to the attention mechanism, even if the recorded test conditions 110a and test steps or test commands 120a are numerous and complex, the recorded test conditions 110a and test steps or test commands 120a can be processed accurately and in a short time. Since the evaluation AI model 170A also includes encoders 173 and 174, even if the recorded evaluation items 160a are numerous and complex, the evaluation items 160a can be processed accurately and in a short time.

[0124] [Second Embodiment]

[0125] The test and evaluation system of the second embodiment will be described below. Figure 19 As shown, the test evaluation system 1B of this embodiment processes the evaluation points as "voltage" and "current". Figure 19 , Figure 20 as well as Figure 21 As shown, the first extraction AI model 130B extracts the test condition (voltage) 111d recorded as related to the evaluation viewpoint "voltage". Additionally, the first extraction AI model 130B extracts the test condition (current) 112d recorded as related to the evaluation viewpoint "current".

[0126] like Figure 19 , Figure 20 as well as Figure 22 As shown, the second extraction AI model 140B extracts the test steps or test commands (voltage) 121d recorded as related to the evaluation viewpoint "voltage". Additionally, the second extraction AI model 140B extracts the test steps or test commands (current) 122d recorded as related to the evaluation viewpoint "current".

[0127] like Figure 19 As shown, the evaluation project integration unit 150 integrates the input test conditions (voltage) 111d, test conditions (current) 112d, test steps or test commands (voltage) 121d, and test steps or test commands (current) 122d, and outputs evaluation project 160d. The evaluation project integration unit 150 integrates the test conditions (voltage) 111d and test steps or test commands (voltage) 121d related to the evaluation viewpoint "voltage" as evaluation project 161d for the evaluation viewpoint "voltage". The evaluation project integration unit 150 integrates the test conditions (current) 112d and test steps or test commands (current) 122d related to the evaluation viewpoint "current" as evaluation project 162d for the evaluation viewpoint "current".

[0128] The evaluation AI model 170B consists of multiple evaluation AI models (voltage) 171B and evaluation AI model (current) 172B corresponding to the evaluation items (voltage) 161d and (current) 162d of the evaluation viewpoints "voltage" and "current" respectively, which are synthesized by the evaluation item synthesis department 150.

[0129] For the evaluation item (voltage) 161d of the evaluation viewpoint "voltage" of the AI ​​model (voltage) 171B, the consistency between the description of the test conditions (voltage) 111d and the description of the test procedure or test command (voltage) 121d is evaluated as a reliability score (voltage) 181d shown by either (allow / 1) or (not allowed / 0).

[0130] For the evaluation item (current) 162d of the evaluation viewpoint "current", the consistency between the test conditions (current) 112d and the test procedure or test command (current) 122d is evaluated as a reliability score (current) 182d, which is indicated by either (allow / 1) or (not allowed / 0).

[0131] The following explains the preparation of training data for the first extraction of AI model 130B and the second extraction of AI model 140B. For example... Figure 23 As shown, in order to prepare training data for the first AI model 130B, a first training data preparation unit (test condition) 131B, which is configured in the same way as the first AI model 130B described above and has already been trained, is used. The first training data preparation unit (test condition) 131B extracts test condition (voltage) 111d and test condition (current) 112d from the record of test condition 110a, similar to the first embodiment described above. Test condition 110a, test condition (voltage) 111d, and test condition (current) 112d are supplied as training data for the first AI model 130B.

[0132] To prepare training data for the second AI model 140B, a second training data preparation unit (test step or test command) 141B, which is configured and trained in the same manner as the second AI model 140B described above, is used. The second training data preparation unit (test step or test command) 141B extracts test step or test command (voltage) 121d and test step or test command (current) 122d from the recorded test step or test command 120a, similar to the first embodiment described above. Test step or test command 120a, test step or test command (voltage) 121d, and test step or test command (current) 122d are supplied as training data for the second AI model 140B.

[0133] The following explains the training data used to evaluate AI model 170B. For example... Figure 24 As shown, the first extraction AI model (test condition) 130B, which is input with the record of test condition 110e, extracts the record of test condition (voltage) 111e and test condition (current) 112e according to each evaluation viewpoint "voltage" and "current". The second extraction AI model (test step or test command) 140B, which is input with the record of test step or test command 120e, extracts the record of test step or test command (voltage) 121e and test step or test command (current) 122e according to each evaluation viewpoint "voltage" and "current". Furthermore, for ease of explanation, the first input unit 119 and the second input unit 129 are not shown in the figure.

[0134] The description of test condition (voltage) 111e is set to be related to the description of test procedure or test command (voltage) 121e. The description of test condition (current) 112e is set to be related to the description of test procedure or test command (current) 122e.

[0135] In this case, the evaluation project 161e, which is integrated by the evaluation project integration department 150, includes records of test conditions (voltage) 111e and test procedures or test commands (voltage) 121e that are related to each other. The evaluation AI model (voltage) 171B outputs a reliability score (voltage) 181e as positive data, which is represented by (can / 1).

[0136] The evaluation project 162e, which is integrated by the evaluation project integration department 150, includes records of interrelated test conditions (current) 112e and test procedures or test commands (current) 122e. The reliability score (current) 182e, which is represented by the evaluation AI model (current) 172B, is used as positive data.

[0137] On the other hand, such as Figure 25 As shown, the first extraction AI model (test condition) 130B, which is input with the record of test condition 110f, extracts the records of test condition (voltage) 111e and test condition (current) 112f for each evaluation viewpoint "voltage" and "current". The second extraction AI model (test step or test command) 140B, which is input with the record of test step or test command 120f, extracts the records of test step or test command (voltage) 121f and test step or test command (current) 122f for each evaluation viewpoint "voltage" and "current".

[0138] The description of test condition (voltage) 111e and the description of test procedure or test command (voltage) 121f are set to be unrelated to each other. The description of test condition (current) 112e and the description of test procedure or test command (current) 122f are set to be unrelated to each other. The similarity confirmation unit 190 deletes the descriptions of test condition (voltage) 111f and test condition (current) 112f.

[0139] In this case, the evaluation project 161f, which is integrated by the evaluation project integration department 150, includes descriptions of test conditions (voltages) 111e and test procedures or test commands (voltages) 121f that are not related to each other. The reliability score (voltage) 181f, which is represented by the evaluation AI model (voltage) 171B (not valid / 0), is used as negative data.

[0140] The evaluation project 162f, which is integrated by the evaluation project integration department 150, includes records of test conditions (currents) 112e and test procedures or test commands (currents) 122f that are not related to each other. The evaluation AI model (current) 172B outputs a reliability score (current) 182f representing (not possible / 0) as negative data. As described above, a set of training data containing positive data with a reliability score of (possible / 1) and negative data with a reliability score of (not possible / 0) can be obtained.

[0141] In this embodiment, for example, the evaluation points "voltage" and "current," which are important in testing the functionality of the software and hardware in the vehicle's inverter ECU, can be evaluated, similarly to the first embodiment described above, by a reliability score indicating either accessibility or unavailability, based on the consistency between the description of test condition 110a and the description of test steps or test command 120a. Evaluation points can also be set to "system speed," "motor," "system temperature," etc.

[0142] [Embodiment 3]

[0143] The test and evaluation system of the third embodiment will be described below. Figure 26As shown, the test evaluation system 1C in this embodiment also includes a test generation AI model 200, which generates test steps or test commands 120a based on the test conditions 110a input to the first input unit 119. The first extraction AI model 130A, the second extraction AI model 140A, and the evaluation AI model 170A use classification models, while the test generation AI model 200 uses a large-scale language model (LLM) that has undergone fine-tuning for domain knowledge. The test steps or test commands 120a generated by the test generation AI model 200 based on the test conditions 110a input to the first input unit 119 are input to the second input unit 129.

[0144] In this embodiment, the consistency of the test steps or test commands 120a generated by the test-generated AI model 200 according to the test conditions 110a can be evaluated.

[0145] [Example 4]

[0146] The test and evaluation system of the fourth embodiment will be described below. Figure 27 As shown, in the test evaluation system 1D of this embodiment, the first extraction AI model 130C extracts records related to the evaluation viewpoint from the records of test conditions 110g input to the first input unit 119 using regular expression pattern matching. The second extraction AI model 140C extracts records related to the evaluation viewpoint from the records of test steps or test commands 120g input to the second input unit 129 using regular expression pattern matching.

[0147] The following section uses the first extraction AI model 130C as a representative example for illustration. Figure 28 The example shown is an instance where the first extraction AI model 130C uses regular expression pattern matching to check sentences containing letters within parentheses in ascending order from the beginning. From the record of test condition 110g, test condition 113a is extracted for the record “(a) Validation Environment: XXX…YYY” after (a) and test condition 113b for the record “(b) Pre-context Environment AAA…CCC” after (b). The same applies to the second extraction AI model 140C. Furthermore, letters within parentheses can also be replaced with Arabic numerals and Roman numerals.

[0148] exist Figure 29The text describes an example where the first extraction AI model 130C uses a regular expression to match the names of predefined target title lists 114 from the test condition 110h input to the first input unit 119. The first extraction AI model 130C is input with test condition 110h and target title list 114. The target title list 114 includes, for example, titles "Verification Environment" and "Pre-test Environment". From the test condition 110h, test condition 113c is extracted, containing the record "(c) Verification Environment: XXX…YYY" after "(c) Verification Environment" and test condition 113d, containing the record "(d) Pre-test Environment: AAA…CCC" after "(d) Pre-test Environment". The same applies to the second extraction AI model 140C.

[0149] exist Figure 30 The text describes an example of pattern matching using a regular expression to check the symbols in a predefined target symbol list 115 from the test condition 110i input to the first input unit 119 for the first extraction AI model 130C. The first extraction AI model 130C is input with test condition 110i and target symbol list 115. For example, the symbol "###" is defined in the target symbol list 115. From the test condition 110i, test condition 113e, which contains the statement "###Verification Environment: XXX…YYY" after "###", and test condition 113f, which contains the statement "###Pre-conference Environment AAA…CCC", are extracted. The same applies to the second extraction AI model 140C.

[0150] According to this embodiment, since the first extraction AI model 130C and the second extraction AI model 140C utilize regular expression pattern matching, accurate pattern matching can be performed, reducing the risk of extracting erroneous information. Furthermore, by using regular expression pattern matching, resource consumption can be reduced.

[0151] Furthermore, the present invention is not limited to the embodiments described above, but includes various modifications. For example, the embodiments described above have been detailed for ease of understanding and explanation of the present invention, and are not necessarily limited to having all the described configurations. Additionally, a portion of the structure of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Furthermore, for a portion of the configuration of each embodiment, other configurations can be added, deleted, or replaced. Furthermore, the aforementioned configurations, functions, processing units, processing units, etc., can be partially or entirely implemented in hardware, for example, using integrated circuits, or all of them can be implemented in software.

[0152] Symbol Explanation

[0153] 1A, 1B, 1C, 1D Test Evaluation System

[0154] Test conditions for 110a, 110b, 110c, 110e, 110f, 110g, 110h, and 110i

[0155] Test conditions (pre-test environment) for 111a, 111b, and 111c

[0156] Test conditions (voltage) for 111d, 111e, and 111f

[0157] Test conditions (observed variables) for 112a, 112b, and 112c

[0158] Test conditions (current) for 112d, 112e, and 112f

[0159] Test conditions for 113a, 113b, 113c, 113d, 113e, and 113f

[0160] 114 Target Title List

[0161] 115 List of Target Symbols

[0162] 119 First Input Section

[0163] Test procedures or test commands for 120a, 120b, 120c, 120e, and 120f.

[0164] Test steps or commands (pre-test environment) for 121a, 121b, and 121c

[0165] Test procedures or test commands (voltage) for 121d, 121e, and 121f.

[0166] Test steps or test commands (observed variables) for 122a, 122b, and 122c.

[0167] Test procedures or test commands (current) for 122d, 122e, and 122f.

[0168] 129 Second Input Section

[0169] 130A, 130B, 130C - First Extraction AI Model

[0170] 131A, 131B First Training Data Preparation Department (Test Conditions)

[0171] 140A, 140B, 140C - Second extraction AI model

[0172] 141A, 141B Second Training Data Preparation Department (Test Procedures or Test Commands)

[0173] 150 Evaluation Project General Department

[0174] 160 Assessment Items

[0175] 160a, 160d evaluation items

[0176] 161a, 161b, 161c Assessment Items (Pre-Establishment Environment)

[0177] Evaluation items (voltage) for 161d, 161e, and 161f

[0178] Evaluation items (observed variables) for 162a, 162b, and 162c

[0179] Evaluation items (current) for 162d, 162e, and 162f

[0180] 170A, 170B Evaluating AI Models

[0181] 171A Evaluating AI Models (Pre-Evaluation Environment)

[0182] 171B Evaluate AI Model (Voltage)

[0183] 172A Evaluating AI Models (Observed Variables)

[0184] 172B Evaluate AI Model (Current)

[0185] Encoders 173 and 174

[0186] 175. Multilayer perceptron

[0187] 180a, 180d reliability scores

[0188] 181a, 181b, 181c Reliability Scores (Pre-event Environment)

[0189] 181d, 181e, 181f Reliability Score (Voltage)

[0190] Reliability scores for 182a, 182b, and 182c (observed variables)

[0191] Reliability scores (current) for 182d, 182e, and 182f

[0192] 190 Similarity Confirmation Department

[0193] 200 tests generate AI models

[0194] 300 Encoder

[0195] 301 Input Embedding Layer

[0196] 302 Positional Encoding Layer

[0197] 303 Stack

[0198] 304 Multi-Head Self-Attention Layer

[0199] 305 Add and Normalization Layer

[0200] 306 Feed-Forward Layer

[0201] 307 Add and Normalization Layer

[0202] 310 Scaled Dot-product Attention

[0203] 311 Matrix Multiplication

[0204] 312 Scaling

[0205] 313 Mask

[0206] 314 Normalization exponent (softmax) function

[0207] 315 Matrix Multiplication

[0208] 321, 322, 323 Linear Layers

[0209] 324 Concatenate Layer

[0210] 325 Linear Layer

[0211] 401 Input Layer

[0212] 402 Hidden Layer

[0213] 403 Output Layer

[0214] 404 node

[0215] 405 Edge.

Claims

1. A testing and evaluation system, characterized in that, have: The first input section contains a record of its input test conditions; The second input unit receives a record of either the test steps or the test command generated based on the record of the test conditions. The first extraction AI model extracts records related to a predefined evaluation viewpoint from the records of the test conditions input to the first input unit; The second extraction AI model extracts records related to the evaluation viewpoint from the records of either the test steps or the test commands input to the second input unit. The evaluation project integration department integrates the records of the test conditions extracted by the first extraction AI model and the records of any one of the test steps and test commands extracted by the second extraction AI model as the evaluation items for the evaluation viewpoint; and The AI ​​model is evaluated for the evaluation project based on the evaluation viewpoint synthesized by the evaluation project integration department, and the consistency between the recorded test conditions and the recorded test steps and test commands.

2. The testing and evaluation system according to claim 1, characterized in that, The first extraction AI model extracts records related to each of the multiple evaluation viewpoints from the records of the test conditions input to the first input unit. The second extraction AI model extracts records related to each of the multiple evaluation viewpoints from the records of either the test steps or the test commands input to the second input unit. The evaluation project integration department, based on multiple evaluation viewpoints, integrates the records of test conditions related to each evaluation viewpoint extracted by the first extraction AI model for each evaluation viewpoint, and the records of any one of the test steps and test commands related to each evaluation viewpoint extracted by the second extraction AI model for each evaluation viewpoint, as the evaluation project for each evaluation viewpoint. The evaluation AI model refers to multiple evaluation AI models corresponding to each evaluation project, based on the evaluation viewpoints synthesized by the evaluation project integration department. Each of the multiple evaluation AI models, for each evaluation item of the evaluation viewpoint, evaluates the consistency between the recorded test conditions and the recorded test steps and test commands, as a reliability score indicated by either accessibility or unaccessibility.

3. The testing and evaluation system according to claim 2, characterized in that, The evaluation AI model is trained using a set of training data. The training data includes: affirmative data where the records of the test conditions are consistent with the records of any of the test steps and the test commands, and the reliability score is acceptable; and The recorded test conditions are inconsistent with the recorded test steps and test commands, and the reliability score is negative data that cannot be obtained.

4. The testing and evaluation system according to claim 2, characterized in that, The first extraction AI model decomposes the record of the test conditions input to the first input unit by each word, calculates the probability of each word being related to each of the multiple evaluation views, and determines which of the multiple evaluation views each word is related to. By combining word words related to the same evaluation view, it extracts the record related to each of the multiple evaluation views from the record of the test conditions input to the first input unit. The second extraction AI model decomposes the record of any of the test steps and test commands input to the second input unit for each word. By calculating the probability of each word being related to each of the multiple evaluation views, it determines which of the multiple evaluation views each word is related to. By combining words related to the same evaluation view, it extracts the record related to each of the multiple evaluation views from the record of any of the test steps and test commands input to the second input unit.

5. The testing and evaluation system according to claim 2, characterized in that, The first extraction AI model will be trained by using records related to each of the multiple evaluation viewpoints extracted from the records of the test conditions as training data. The second extraction AI model will be trained using records related to each of the multiple evaluation viewpoints extracted from the records of either the test step or the test command as training data.

6. The testing and evaluation system according to claim 1, characterized in that, It also includes a test generation AI model, which generates a record of either the test steps or the test command based on the recorded test conditions input to the first input unit. The second input unit contains a record of either the test steps generated by the test generation AI model based on the test conditions input to the first input unit or the test command.

7. The testing and evaluation system according to claim 1, characterized in that, The first extraction AI model uses regular expression pattern matching to extract records related to the evaluation viewpoint from the records of the test conditions input to the first input unit. The second extraction AI model uses regular expression pattern matching to extract records related to the evaluation viewpoint from the records of either the test steps or the test commands input to the second input unit.

8. The testing and evaluation system according to claim 2, characterized in that, The first extracted AI model is the encoder that constitutes the converter. The second extracted AI model is the encoder that constitutes the converter.

9. The testing and evaluation system according to claim 2, characterized in that, The evaluation AI model includes an encoder that constitutes the converter.

Images