Testing evaluation system

Abstract

To evaluate consistency between testing criteria described in natural language and a testing procedure or testing command described in natural language.SOLUTION: A testing evaluation system comprises: a first extraction AI model that extracts descriptions related to predefined evaluation criteria from descriptions of testing criteria input to a first input part; a second extraction AI model that extracts descriptions related to the evaluation criteria from descriptions of a testing procedure or a testing command input to a second input unit; an evaluation item integration part that integrates the descriptions of the testing criteria extracted by the first extraction AI model and the descriptions of the testing procedure and the like extracted by the second extraction AI model to obtain evaluation items in the evaluation criteria; and an evaluation AI model that evaluates consistency between the descriptions of the testing criteria and the descriptions of the testing procedure and the like for the evaluation items in the evaluation criteria integrated by the evaluation item integration part.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a test evaluation system. [Background technology]

[0002] Technologies for evaluating software and the like have been developed. For example, Patent Document 1 discloses a method for evaluating the conformance of architecture analysis and design models by automatically organizing all reference information into a hierarchical model. With the technology disclosed in Patent Document 1, a user can determine whether conformance and reliability are met by examining a hierarchical model structured based on input information required for model creation. Furthermore, Patent Document 2 discloses a method for evaluating software reliability based on predetermined items such as defect rate, operation rate, and crash rate. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 7,480,893 [Patent Document 2] Chinese Patent Application Publication No. 114489760 Summary of the Invention [Problem to be solved by the invention]

[0004] In software and hardware function testing, whether or not test criteria corresponding to the software and hardware functions are satisfied is confirmed by executing a test procedure or test command corresponding to the test criteria. The test procedure or test command corresponding to the test criteria is generated by a human or artificial intelligence (AI). In order to properly test the software and hardware functions, it is necessary to evaluate the consistency between the test conditions and the generated test procedure or test command. However, because the test conditions, test procedure, and test command are written in natural language, the technologies disclosed in Patent Documents 1 and 2 described above are difficult to address, and improvements are needed.

[0005] The present invention has been made in consideration of the above points, and aims to provide a test evaluation system that can evaluate the consistency of either a test procedure or test commands written in natural language with test conditions written in natural language. [Means for solving the problem]

[0006] The test evaluation system of the present invention for solving the above-mentioned problems is characterized by comprising: a first input unit into which a description of test conditions is input; a second input unit into which a description of either a test procedure or a test command generated based on the description of test conditions is input; a first extraction AI model that extracts descriptions related to a predefined evaluation perspective from the description of test conditions input to the first input unit; a second extraction AI model that extracts descriptions related to the evaluation perspective from the description of either the test procedure or the test command input to the second input unit; an evaluation item integration unit that integrates the description of test conditions extracted by the first extraction AI model with the description of either the test procedure or the test command extracted by the second extraction AI model to form evaluation items in the evaluation perspective; and an evaluation AI model that evaluates the consistency between the description of test conditions and the description of either the test procedure or the test command for the evaluation items in the evaluation perspective integrated by the evaluation item integration unit. [Effects of the Invention]

[0007] According to the present invention, it is possible to evaluate the consistency of either a test procedure or a test command written in natural language with a test condition written in natural language. Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. In addition, the problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an outline of a test evaluation system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing details of the test evaluation system of FIG. 1. [Figure 3] FIG. 3 is a block diagram showing details of the first extracted AI model and the second extracted AI model of FIG. 2. [Figure 4] FIG. 4 is a block diagram showing the configuration of an encoder that constitutes the first extracted AI model and the second extracted AI model of FIG. 3. [Figure 5]A diagram showing the scaled dot-product attention that constitutes the multi-head self-attention layer of Figure 4. [Figure 6] A diagram showing a multi-head self-attention layer constructed with the scaled dot-product attention of Figure 5. [Figure 7] A diagram showing an example of test condition descriptions input into the first extraction AI model. [Figure 8] FIG. 8 is a diagram showing the test conditions in FIG. 7 broken down into tokens. [Figure 9] FIG. 9 is a diagram showing a state in which the probability related to the evaluation viewpoint "pre-environment" is calculated for each token in FIG. 8. [Figure 10] FIG. 9 is a diagram showing the state in which the probability associated with the evaluation perspective "observation variable" is calculated for each token in FIG. 8. [Figure 11] A diagram showing the state in which tokens related to the same evaluation perspectives, "prior environment" and "observation variables," are combined based on the probabilities calculated in Figures 9 and 10. [Figure 12] A diagram showing the state in which tokens related to the same evaluation perspectives, "pre-experiment environment" and "observation variables," are combined in the description of the test procedure input to the second extraction AI model. [Figure 13] FIG. 10 is a diagram showing the preparation of training data for the first extraction AI model and the second extraction AI model according to the first embodiment. [Figure 14] FIG. 3 is a block diagram showing details of the evaluation item integration unit of FIG. 2. [Figure 15] Block diagram showing details of the evaluation AI model in Figure 2. [Figure 16] 16 is a diagram showing the multilayer perceptron of FIG. 15. [Figure 17] FIG. 1 is a diagram showing training data for an evaluation AI model including positive data with a fair reliability score according to the first embodiment. [Figure 18] FIG. 1 is a diagram showing training data for an evaluation AI model including negative data with a reliability score of unacceptable in the first embodiment. [Figure 19] FIG. 10 is a block diagram showing details of a test evaluation system according to a second embodiment. [Figure 20]FIG. 20 is a block diagram showing details of the first extracted AI model and the second extracted AI model of FIG. 19. [Figure 21] A diagram showing the state in which tokens related to the same evaluation perspectives, "voltage" and "current," are extracted from the test condition description input to the first extraction AI model. [Figure 22] A diagram showing the state in which tokens related to the same evaluation perspectives, "voltage" and "current," are extracted from the description of the test procedure input to the second extraction AI model. [Figure 23] FIG. 10 is a diagram showing the preparation of training data for the first extracted AI model and the second extracted AI model according to the second embodiment. [Figure 24] FIG. 10 is a diagram showing training data for an evaluation AI model including positive data with a fair reliability score according to the second embodiment. [Figure 25] FIG. 10 is a diagram showing training data for an evaluation AI model including negative data with a reliability score of unacceptable in the second embodiment. [Figure 26] FIG. 10 is a block diagram showing an outline of a test evaluation system according to a third embodiment. [Figure 27] FIG. 10 is a block diagram showing an outline of a test evaluation system according to a fourth embodiment. [Figure 28] FIG. 13 is a diagram showing an example of pattern matching using regular expressions according to the fourth embodiment. [Figure 29] FIG. 13 is a diagram showing an example of pattern matching using regular expressions according to the fourth embodiment. [Figure 30] FIG. 13 is a diagram showing an example of pattern matching using regular expressions according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the same elements and processing contents are denoted by the same reference numerals and their description will be omitted.

[0010] [First Example] The test evaluation system of the first embodiment will be described below. The test evaluation system 1A shown in Fig. 1 is configured as a computer and is used, for example, to test and evaluate the software and hardware functions of a vehicle inverter ECU (Electronic Control Unit). The test evaluation system 1A performs the following processes by having a CPU (Central Processing Unit) of the test evaluation system 1A execute computer programs stored in a ROM (Read Only Memory) and RAM (Random Access Memory) of the test evaluation system 1A. The test evaluation system 1A evaluates whether or not there is consistency between test conditions (Testing Criteria) corresponding to the software and hardware functions and either a test procedure (Testing Procedure) or a test command (Testing Command) corresponding to the test conditions.

[0011] "Test conditions" refer to, for example, conditions for passing a test related to a predefined evaluation perspective. "Test procedures" refer to, for example, procedures for checking whether test conditions related to a predefined evaluation perspective are met. "Test commands" refer to, for example, program source commands and source code for checking whether test conditions related to a predefined evaluation perspective are met. Evaluation perspectives related to test conditions, test procedures, and test commands include, for example, "pre-environment," "observation variables," "voltage," and "current," as will be described later.

[0012] When the evaluation perspective is "pre-environment", "test conditions" refer to, for example, the environment that must be met before a test is executed. When the evaluation perspective is "pre-environment", "test procedures" refer to, for example, the procedures for meeting the environment that must be met before a test is executed. When the evaluation perspective is "pre-environment", "test commands" refer to, for example, the commands and source code of the program source that execute the procedures for meeting the environment that must be met before a test is executed.

[0013] More specifically, when the evaluation viewpoint is the "pre-environment," the "test condition" is, for example, a description such as "initialize the current value." When the evaluation viewpoint is the "pre-environment," the "test procedure" is, for example, a description such as "set the current variable to 10." When the evaluation viewpoint is the "pre-environment," the "test command" is, for example, a program source command and source code that causes a computer to execute a process such as "set the current variable to 10."

[0014] When the evaluation perspective is "observation variables," "test conditions" refer to, for example, the numerical values ​​of the objects, variables, and thresholds that are observed in order to pass the test. When the evaluation perspective is "observation variables," "test procedures" refer to, for example, the procedures for setting the numerical values ​​of the objects, variables, and thresholds that are observed in order to pass the test. When the evaluation perspective is "observation variables," "test commands" refer to, for example, the program source commands and source code that cause a computer to execute the process of setting the numerical values ​​of the objects, variables, and thresholds that are observed in order to pass the test.

[0015] More specifically, when the evaluation perspective is "observation variables," the "test conditions" are descriptions that specify the object of observation, such as "observe the status of test system A." When the evaluation perspective is "observation variables," the "test procedure" is descriptions such as "observation variable settings: 1. current 2. transmission 3. temperature." When the evaluation perspective is "observation variables," the "test commands" are program source commands and source code that cause a computer to execute setting processes such as "observation variable settings: 1. current 2. transmission 3. temperature."

[0016] When the evaluation viewpoint is "voltage," the "test condition" is, for example, a description such as "the low voltage power supply is within the normal range." When the evaluation viewpoint is "voltage," the "test procedure" is, for example, a description such as "enable the high voltage DC transmission switch." When the evaluation viewpoint is "voltage," the "test command" is, for example, a program source command and source code for causing a computer to execute a process such as "enable the high voltage DC transmission switch."

[0017] When the evaluation viewpoint is "current", the "test condition" is, for example, a description such as "DC offset". When the evaluation viewpoint is "current", the "test procedure" is, for example, a description such as "calculation of DC offset". When the evaluation viewpoint is "current", the "test command" is, for example, a program source command and source code for causing a computer to execute a process such as "calculation of DC offset".

[0018] As shown in FIG. 1, the test evaluation system 1A includes a first input unit 119, a second input unit 129, a first extraction AI model 130A, a second extraction AI model 140A, an evaluation item integration unit 150, and an evaluation AI model 170A. The first input unit 119 receives input of a description of test conditions 110a. The second input unit 129 receives input of a description of test procedures or test commands 120a generated based on the description of test conditions 110a. The test procedures or test commands 120a are generated by a human or AI (artificial intelligence) based on the test conditions 110a. Note that the description of the test conditions 110a and the test procedures or test commands 120a refers to the characters, words, phrases, clauses, and sentences in natural language that make up the test conditions 110a and the test procedures or test commands 120a.

[0019] The first extraction AI model 130A extracts descriptions related to predefined evaluation viewpoints from the descriptions of the test conditions 110a input to the first input unit 119. The second extraction AI model 140A extracts descriptions related to evaluation viewpoints from the descriptions of the test procedures or test commands 120a input to the second input unit 129.

[0020] The evaluation item integration unit 150 integrates the description of the test conditions 110a extracted by the first extraction AI model 130A and the description of the test procedures or test commands 120a extracted by the second extraction AI model 140A to create evaluation items 160 from an evaluation perspective.

[0021] The evaluation AI model 170A evaluates the consistency between the description of the test condition 110a and the description of the test procedure or test command 120a for the evaluation item 160 in the evaluation perspective integrated by the evaluation item integration unit 150 as a reliability score 180a.

[0022] An example of the test evaluation system 1A in which the evaluation items are a "pre-environment" and an "observation variable" will be described below. Note that the evaluation items "pre-environment" and "observation variable" in this embodiment are merely examples, and other evaluation items may be used. The number of evaluation items can also be set arbitrarily. As shown in FIG. 2, the first extraction AI model 130A extracts descriptions related to each of a plurality of evaluation perspectives from the descriptions of the test conditions 110a input to the first input unit 119. In the example of FIG. 2, the first extraction AI model 130A extracts test conditions (pre-environment) 111a, which are descriptions related to the evaluation perspective "pre-environment." Furthermore, the first extraction AI model 130A extracts test conditions (observation variable) 112a, which are descriptions related to the evaluation perspective "observation variable."

[0023] The second extraction AI model 140A extracts descriptions related to each of a plurality of evaluation perspectives from the descriptions of the test procedures or test commands 120a input to the second input unit 129. In the example of FIG. 2, the second extraction AI model 140A extracts test procedures or test commands (pre-environment) 121a, which are descriptions related to the evaluation perspective "pre-environment." In addition, the second extraction AI model 140A extracts test procedures or test commands (observation variables) 122a, which are descriptions related to the evaluation perspective "observation variables."

[0024] The evaluation item integration unit 150 integrates, for each of multiple evaluation perspectives, descriptions of test conditions related to the evaluation perspectives extracted by the first extraction AI model 130A and descriptions of either test procedures or test commands related to the evaluation perspectives extracted by the second extraction AI model 140A. The evaluation item integration unit 150 sets the descriptions of test conditions and descriptions of either test procedures or test commands integrated for each of the multiple evaluation perspectives as evaluation items 160a for each of the evaluation perspectives.

[0025] 2, the evaluation item integrating unit 150 integrates the test conditions (pre-environment) 111a and the test procedure or test command (pre-environment) 121a related to the evaluation perspective "pre-environment" to obtain an evaluation item 161a in the evaluation perspective "pre-environment." The evaluation item integrating unit 150 integrates the test conditions (observation variable) 112a and the test procedure or test command (observation variable) 122a related to the evaluation perspective "observation variable" to obtain an evaluation item 162a in the evaluation perspective "observation variable."

[0026] The evaluation AI model 170A is a plurality of evaluation AI models (pre-environment) 171A and evaluation AI models (observation variables) 172A corresponding to the evaluation items (pre-environment) 161a and evaluation items (observation variables) 162a in the evaluation perspectives "pre-environment" and "observation variables" integrated by the evaluation item integration unit 150.

[0027] The evaluation AI model (pre-environment) 171A evaluates the consistency between the description of the test conditions (pre-environment) 111a and the description of the test procedure or test command (pre-environment) 121a for the evaluation item (pre-environment) 161a in the evaluation perspective ``pre-environment'' as a reliability score (pre-environment) 181a indicated as either (pass / 1) or (fail / 0).

[0028] The evaluation AI model (observation variable) 172A evaluates the consistency between the description of the test condition (observation variable) 112a and the description of the test procedure or test command (observation variable) 122a for the evaluation item (observation variable) 162a in the evaluation perspective "observation variable" as a reliability score (observation variable) 182a indicated as either (pass / 1) or (fail / 0).

[0029] The configuration of each part of the test evaluation system 1A will be described below. As shown in Fig. 3, the first extraction AI model 130A extracts test conditions (pre-environment) 111a, which are descriptions related to the evaluation perspective "pre-environment," from the descriptions of the test conditions 110a input to the first input unit 119. The first extraction AI model 130A also extracts test conditions (observation variables) 112a, which are descriptions related to the evaluation perspective "observation variables," from the test conditions 110a input to the first input unit 119.

[0030] The second extraction AI model 140A extracts test procedures or test commands (pre-environment) 121a, which are descriptions related to the evaluation perspective "pre-environment", from the descriptions of the test procedures or test commands 120a input to the second input unit 129. In addition, the second extraction AI model 140A extracts test procedures or test commands (observation variables) 122a, which are descriptions related to the evaluation perspective "observation variables", from the descriptions of the test procedures or test commands 120a input to the second input unit 129.

[0031] A classification model is used for the first extraction AI model 130A and the second extraction AI model 140A. More specifically, the first extraction AI model 130A and the second extraction AI model 140A are, for example, an encoder 300 constituting a transformer with a multi-head self-attention layer as shown in FIG. 4. The transformer does not process the test conditions 110a and the test procedures or test commands 120a word by word, but processes the entire description at once. This allows the transformer to quickly understand which parts of the description are important and the overall context of the description. Furthermore, by processing the entire description at once, the transformer can perform calculations in parallel, thereby improving calculation speed. Furthermore, as described below, the transformer uses attention to analyze which parts of the description are related to other parts and how. This allows the transformer to accurately grasp the meaning of the description.

[0032] The encoder 300 includes an input embedding layer 301, a positional encoding layer 302, a stack 303, a multi-head self-attention layer 304, an additive normalization layer 305, a feedforward layer 306, and an additive normalization layer 307.

[0033] The Input Embedding Layer 301 converts each word in the input test conditions 110a and test procedures or test commands 120a so that the first extraction AI model 130A and second extraction AI model 140A can calculate them. The Input Embedding Layer 301 converts the input tokens into vectors using a previously trained conversion model. Here, a token refers to the smallest unit of a character string that the first extraction AI model 130A and second extraction AI model 140A use to process the test conditions 110a and test procedures or test commands 120a.

[0034] The Positional Encoding Layer 302 adds information to the vectors converted by the Input Embedding Layer 301 to indicate the positions of the tokens in the test conditions 110a and test procedures or test commands 120a.

[0035] The stack 303 has a structure in which Nx identical layers are stacked. In Fig. 4, only one layer of the stack 303 is shown, and the other layers are omitted. Each stack 303 has a multi-head self-attention layer 304, a summation normalization layer 305, a feedforward layer, and a summation normalization layer 307.

[0036] The Multi-Head Self-Attention Layer 304 assigns multiple attentions to the vectors converted by the Input Embedding Layer 301. Attention is information used to direct attention to important parts of the description.

[0037] Attention consists of a query Q, a key K, a value V, and an output value. The query Q is the goal of the calculation. The key K is a collection of word vectors used to calculate the similarity with other tokens. In other words, the key K indicates the relationship with other tokens. The value V is a collection of vectors used to calculate the weighted sum of tokens. In other words, the value V indicates the token itself. All output values ​​are vectors. The output value is calculated as the sum of the weighted sum of the values ​​V. The weight assigned to each value V is calculated by the compatibility function of the query Q with the corresponding key K.

[0038] The multi-head self-attention layer 304 consists of a scaled dot-product attention 310 as shown in Figure 5. The inputs of the scaled dot-product attention 310 are a query Q, a key K, and a value V. The dimension of the key K is d k The dimension of value V is d v In the scaled dot-product attention 310, the inner product of the vectors of the query Q and the key K is calculated in the matrix multiplication 311. In the scale 312, the inner product of the vectors of the query Q and the key K is calculated as |(d k ) 1 / 2 Divided by |

[0039] In Mask 313, a mask is applied to the padded portion when the number of input words is shorter than the sequence length. The calculation results of Matrix Inner Product 311, Scale 312, and Mask 313 are applied to Softmax function 314. The Softmax function 314 normalizes the sum of the weights to 1. In Matrix Multiplication 315, the inner product of the vector between the calculation result of Softmax function 314 and the value V is calculated. The above calculation in Scaled Dot-Product Attention 310 is expressed by the following equation (1).

[0040]

number

[0041] As shown in Figure 6, the multi-head self-attention layer 304 is configured using multiple scaled dot-product attention layers 310. The dimensions of the value V, key K, and query Q are reduced by the same number of linear layers 321, 322, and 323 as the scaled dot-product attention layers 310, respectively, thereby speeding up calculations.

[0042] The calculation results of each of the scaled dot-product attention layers 310 are concatenated by a concatenate layer 324. The calculation results concatenated by the concatenate layer are again projected to another dimension by a linear layer 325, thereby obtaining the final output value of the multi-head self-attention layer 304.

[0043] As shown in FIG. 4, the add and normalization layer 305 of the encoder 300 adds the input to the previous multi-head self-attention layer 304 to the output of the multi-head self-attention layer 304, and normalizes the output of the multi-head self-attention layer 304.

[0044] The Feed-Forward Layer 306 calculates the input vector x at position t. t A feedforward fully connected network is applied to the input vector x. The feedforward layer 306 is composed of, for example, two layers. The dimension of the first layer is set to, for example, four times the dimension of the input. The second layer restores the output value to the original input dimension. The weights W1 and W2 are the same regardless of the position t. This ensures that the same token input to the feedforward layer 306 generates the same output. In the feedforward layer 306, a ReLU (Rectified Linear Unit) activation function is applied. The input vector x tThe output FFN(x t ) is expressed by the following equation (2).

[0045]

number

[0046] The add and normalization layer 307 adds the input to the preceding feedforward layer 306 to the output of the feedforward layer 306, and normalizes the output of the feedforward layer 306. Note that as the first extraction AI model 130A and the second extraction AI model 140A, in addition to the encoder 300 constituting a transformer, an LSTM (Long Short Term Memory) type encoder model may be used.

[0047] Specific processing in the first extraction AI model 130A and the second extraction AI model 140A will be described below. The operation of the first extraction AI model 130A will be described below as a representative example. Assume that the description of test condition 110a shown in FIG. 7 is input to the first input unit 119. As shown in FIG. 8, the first extraction AI model 130A breaks down the description of test condition 110a input to the first input unit 119 into tokens indicated by dashed lines in FIG. 8.

[0048] As shown in FIG. 9, the first extraction AI model 130A calculates the probability associated with each of multiple evaluation perspectives for each token. For example, in the example of FIG. 9, for the evaluation perspective "Preliminary Environment," the token with the same title, "Preliminary Environment," has a high probability of 0.95, and the probabilities of the tokens before and after it, "(d)," "AAA...," "BBB...," and "CCC...," are also calculated to be high. In the example of FIG. 10, for the evaluation perspective "Observed Variables," the token "Success Criteria," which indicates the title, is not the same as "Observed Variables." However, the "Success Criteria" section typically contains observed variables that need to be observed in the test, so the relevance is high. Therefore, the probability of the token "Success Criteria" is calculated to be high at 0.85. The probabilities of the tokens before and after "Success Criteria," "(f)," "DDD...," "EEE...," "FFF...," "GGG...," and "HHH...," are also calculated to be high.

[0049] As shown in Figure 11, the first extraction AI model 130A calculates probabilities to determine which of multiple evaluation perspectives each token is associated with, and combines tokens associated with the same evaluation perspective. In the example of Figure 11, for the evaluation perspective "pre-environment," "(d) pre-environment...CCC" is combined, and for the evaluation perspective "observation variables," "(f) success criteria...HHH" is combined. Through the above processing, the first extraction AI model 130A extracts descriptions related to each of the multiple evaluation perspectives from the descriptions of the test conditions 110a input to the first input unit 119.

[0050] 11, for the evaluation perspective "pre-environment," "(d) pre-environment...CCC" is extracted as the test condition (pre-environment) 111a. For the evaluation perspective "observation variable," "(f) success criteria...HHH" is extracted as the test condition (observation variable) 112a.

[0051] The second extraction AI model 140A similarly decomposes the description of the test procedure or test command 120a input to the second input unit 129 into tokens. The second extraction AI model 140A determines which of multiple evaluation perspectives each token is associated with by calculating the probability that each token is associated with each of multiple evaluation perspectives. As shown in FIG. 12 , the second extraction AI model 140A combines tokens associated with the same evaluation perspective to extract descriptions related to each of multiple evaluation perspectives from the description of the test procedure or test command 120a input to the second input unit 129.

[0052] 12, for the evaluation perspective "pre-environment", the description after the title "pre-environment" is extracted as the test procedure or test command (pre-environment) 121a. For the evaluation perspective "observation variable", the description after the title "observation variable" is extracted as the test procedure or test command (observation variable) 122a.

[0053] As shown in Fig. 13, the first extracted AI model 130A is trained using descriptions related to evaluation viewpoints extracted for each of a plurality of evaluation viewpoints from the descriptions of the test conditions 110a as training data. In the example of Fig. 13, a trained first training data preparation unit (test conditions) 131A is used, which has the same configuration as the above-mentioned first extracted AI model 130A.

[0054] The first training data preparation unit (test conditions) 131A extracts test conditions (preliminary environment) 111a and test conditions (observation variables) 112a from the description of the test conditions 110a in the same manner as described above. The test conditions 110a, the test conditions (preliminary environment) 111a, and the test conditions (observation variables) 112a are supplied as training data for the first extracted AI model 130A.

[0055] The second extracted AI model 140A is trained using descriptions related to evaluation viewpoints extracted for each of a plurality of evaluation viewpoints from the descriptions of the test procedure or test command 120a as training data. In the example of Fig. 13, a trained second training data preparation unit (test procedure or test command) 141A is used, which is configured similarly to the second extracted AI model 140A.

[0056] The second training data preparation unit (test procedure or test command) 141A extracts the test procedure or test command (preliminary environment) 121a and the test procedure or test command (observation variable) 122a from the description of the test procedure or test command 120a in the same manner as described above. The test procedure or test command 120a, the test procedure or test command (preliminary environment) 121a, and the test procedure or test command (observation variable) 122a are supplied as training data for the second extraction AI model 140A.

[0057] 14, descriptions of test conditions related to the evaluation perspectives extracted for each evaluation perspective by the first extraction AI model 130A are input to the evaluation item integration unit 150. In the example of Fig. 14, test condition (pre-environment) 111a is input for the evaluation perspective "pre-environment", and test condition (observation variable) 112a is input for the evaluation perspective "observation variable".

[0058] Descriptions of test procedures or test commands related to the evaluation perspectives extracted for each evaluation perspective by the second extraction AI model 140A are input to the evaluation item integration unit 150. In the example of Fig. 14, for the evaluation perspective "pre-environment", a test procedure or test command (pre-environment) 121a is input, and for the evaluation perspective "observation variable", a test procedure or test command (observation variable) 122a is input.

[0059] The evaluation item integrating unit 150 integrates the descriptions of the test conditions and test procedures or test commands extracted for each evaluation perspective, and outputs them as evaluation items for each evaluation perspective. In other words, the evaluation item integrating unit 150 reassembles the descriptions of the test conditions and test procedures or test commands extracted for each evaluation perspective into groups for each evaluation perspective.

[0060] 14, a pair of a test condition (pre-environment) 111a and a test procedure or test command (pre-environment) 121a is output as an evaluation item (pre-environment) 161a. Also, a pair of a test condition (observation variable) 112a and a test procedure or test command (observation variable) 122a is output as an evaluation item (observation variable) 162a. The evaluation item (pre-environment) 161a output by the evaluation item integrating unit 150 is supplied to an evaluation AI model (pre-environment) 171A. The evaluation item (observation variable) 162a output by the evaluation item integrating unit 150 is supplied to an evaluation AI model (observation variable) 172A.

[0061] Below, an evaluation AI model (pre-environment) 171A will be described as a representative example of the evaluation AI model 170A. A classification model is used for the evaluation AI model 170A, such as the evaluation AI model (pre-environment) 171A. More specifically, as shown in FIG. 15 , the evaluation AI model (pre-environment) 171A of the evaluation AI model 170A includes, for example, an encoder 173, an encoder 174, and a multilayer perceptron 175. The test condition (pre-environment) 111a for the evaluation item (pre-environment) 161a is input to the encoder 173, and the test procedure or test command (pre-environment) 121a for the evaluation item (pre-environment) 161a is input to the encoder 174. The configurations of the encoders 173 and 174 are similar to the encoder 300 constituting the transformer described above, and therefore description thereof will be omitted.

[0062] 16, the multilayer perceptron 175 includes an input layer 401, a hidden layer 402, and an output layer 403, each of which has one or more nodes 404. Each of the nodes 404 in the input layer 401 is connected to each and every node 404 in the hidden layer 402 by an edge 405.

[0063] Each node 404 in the hidden layer 402 is connected to a node 404 in the output layer 403 by an edge 405. A signal (information) propagating along an edge 405 propagates in one direction from a node 404 in the input layer 401 to a node 404 in the hidden layer 402. A signal propagating along an edge 405 propagates in one direction from a node 404 in the hidden layer 402 to a node 404 in the output layer 403.

[0064] Each signal output from node 404 in input layer 401 and propagating along edge 405 is multiplied by a weight and input to node 404 in hidden layer 402. Similarly, each signal output from node 404 in hidden layer 402 and propagating along edge 405 is multiplied by a weight and input to node 404 in output layer 403. Each node 404 transforms the input signal using an activation function and outputs it as a new signal.

[0065] Through the above processing, the evaluation AI model 171A outputs the consistency between the description of the test condition (pre-environment) 111a and the description of the test procedure or test command (pre-environment) 121a for the evaluation item (pre-environment) 161a as a reliability score indicated by either (pass / 1) or (fail / 0). The same applies to the evaluation AI model 172A of the evaluation AI model 170A. Note that, in addition to the encoders 173 and 174 constituting the transformer, an LSTM-type encoder model may also be used as a classification model for the evaluation AI model 170A.

[0066] The following describes the training data for the evaluation AI model 170A. The evaluation AI model 170A is trained using a set of training data including positive data in which the description of the test condition 110b and the description of the test procedure or test command 120b are consistent with each other and have a reliability score of (pass / 1), and negative data in which the description of the test procedure or test command 120b are inconsistent with each other and have a reliability score of (fail / 0).

[0067] As shown in FIG. 17, the first extracted AI model (test condition) 130A, to which the description of test condition 110b has been input, extracts the description of test condition (pre-environment) 111b and test condition (observation variable) 112b for each of the evaluation perspectives "pre-environment" and "observation variable." The second extracted AI model (test procedure or test command) 140A, to which the description of test procedure or test command 120b has been input, extracts the description of test procedure or test command (pre-environment) 121b and test procedure or test command (observation variable) 122b for each of the evaluation perspectives "pre-environment" and "observation variable." For ease of explanation, the first input unit 119 and the second input unit 129 are not shown in the figure.

[0068] The description of the test conditions (pre-environment) 111b and the description of the test procedure or test command (pre-environment) 121b are assumed to be related to each other. The description of the test conditions (observation variables) 112b and the description of the test procedure or test command (observation variables) 122b are assumed to be related to each other.

[0069] In this case, the evaluation items 161b integrated by the evaluation item integration unit 150 include descriptions of test conditions (pre-environment) 111b and test procedures or test commands (pre-environment) 121b, which are related to each other. The evaluation AI model (pre-environment) 171A outputs a reliability score (pre-environment) 181b indicating (pass / 1) as positive data.

[0070] The evaluation items 162b integrated by the evaluation item integration unit 150 include descriptions of test conditions (observation variables) 112b and descriptions of test procedures or test commands (observation variables) 122b, which are related to each other. The evaluation AI model (observation variables) 172A outputs a reliability score (observation variable) 182b indicating (pass / 1) as positive data.

[0071] 18, the first extraction AI model (test condition) 130A to which the description of test condition 110c has been input extracts descriptions of test condition (pre-environment) 111c and test condition (observation variable) 112c in addition to descriptions of test condition (pre-environment) 111b and test condition (observation variable) 112b for each of the evaluation perspectives "pre-environment" and "observation variable." The second extraction AI model (test procedure or test command) 140A to which the description of test procedure or test command 120c has been input extracts descriptions of test procedure or test command (pre-environment) 121c and test procedure or test command (observation variable) 122c for each of the evaluation perspectives "pre-environment" and "observation variable."

[0072] The description of the test condition (pre-environment) 111b and the description of the test procedure or test command (pre-environment) 121c are not related to each other. The description of the test condition (observation variable) 112b and the description of the test procedure or test command (observation variable) 122c are not related to each other.

[0073] The descriptions of the test condition (preliminary environment) 111b, the test condition (observation variable) 112b, the test condition (preliminary environment) 111c, and the test condition (observation variable) 112c extracted by the first extraction AI model (test condition) 130A are input to the similarity checking unit 190. The similarity checking unit 190 deletes similar and overlapping descriptions. The similarity checking unit 190 deletes similar and overlapping descriptions through keyword comparison, similarity calculation, etc. In the example of FIG. 18, the descriptions of the test condition (preliminary environment) 111c and the test condition (observation variable) 112c are deleted. For ease of explanation, the similarity checking unit 190 is illustrated only in FIG. 18 and FIG. 25, and is omitted from FIGS. 1, 2, 14, 19, 24, and 27.

[0074] In this case, the evaluation items 161c integrated by the evaluation item integration unit 150 include a description of the test conditions (pre-environment) 111b and a description of the test procedure or test command (pre-environment) 121c, which are unrelated to each other. The evaluation AI model (pre-environment) 171A outputs a reliability score (pre-environment) 181c indicating (fail / 0) as negative data.

[0075] The evaluation items 162c integrated by the evaluation item integration unit 150 include descriptions of unrelated test conditions (observation variables) 112b and descriptions of test procedures or test commands (observation variables) 122c. The evaluation AI model (observation variables) 172A outputs a reliability score (observation variable) 182c indicating (fail / 0) as negative data. In this way, a set of training data can be obtained that includes positive data with a reliability score of (pass / 1) and negative data with a reliability score of (fail / 0).

[0076] As described above, with conventional techniques, it is difficult to evaluate the consistency of test procedures or test commands 120a written in natural language with test conditions 110a written in natural language. This is because large-scale language models (LLMs) of the GPT (Chat Generative Pre-trained Transformer) type only predict the next token of an input description, making it difficult to process descriptions written in natural language. Furthermore, in recent years, there has been a trend toward large numbers and complexities in test conditions 110a and test procedures or test commands 120a, making it more difficult to evaluate the consistency of test procedures or test commands 120a with test conditions 110a.

[0077] On the other hand, in the test evaluation system 1A of this embodiment, the first extraction AI model 130A extracts descriptions related to predefined evaluation perspectives from the descriptions of the test conditions 110a written in natural language, and the second extraction AI model 140A extracts descriptions related to predefined evaluation perspectives from the descriptions of the test procedures or test commands 120a written in natural language, so that it can respond to the test conditions 110a and the test procedures or test commands 120a written in natural language.

[0078] The evaluation item integrating unit 150 integrates the descriptions of the test conditions 110a extracted by the first extraction AI model 130A with the descriptions of the test procedures or test commands 120a extracted by the second extraction AI model 140A to generate evaluation items 160a in the evaluation perspective. The evaluation AI model 170A evaluates the consistency between the descriptions of the test conditions 110a and the descriptions of the test procedures or test commands 120a for the evaluation items 160a in the evaluation perspective integrated by the evaluation item integrating unit 150. This makes it possible to evaluate the consistency of the test procedures or test commands 120a written in natural language with the test conditions 110a written in natural language.

[0079] In this embodiment, the first extraction AI model 130A extracts descriptions related to the evaluation perspectives "pre-environment" and "observation variables" for each of the multiple evaluation perspectives "pre-environment" and "observation variables" from the descriptions of the test conditions 110a. The second extraction AI model 140A extracts descriptions related to the evaluation perspectives "pre-environment" and "observation variables" for each of the multiple evaluation perspectives "pre-environment" and "observation variables" from the descriptions of the test procedures or test commands 120a.

[0080] The evaluation item integration unit 150 integrates, for each of the multiple evaluation perspectives "pre-environment" and "observation variable", the descriptions of test conditions 110a related to the evaluation perspectives "pre-environment" and "observation variable" extracted by the first extraction AI model 130A for each of the evaluation perspectives "pre-environment" and "observation variable", and the descriptions of test procedures or test commands 120a related to the evaluation perspectives "pre-environment" and "observation variable" extracted by the second extraction AI model 140A for each of the evaluation perspectives "pre-environment" and "observation variable".

[0081] The evaluation AI model 170A is a plurality of evaluation AI models (pre-environment) 171A and evaluation AI models 172A corresponding to the evaluation items (pre-environment) 161a and evaluation items (observation variables) 162a in the evaluation perspectives "pre-environment" and "observation variables" integrated by the evaluation item integration unit 150.

[0082] Each of the multiple evaluation AI models (pre-environment) 171A and evaluation AI models 172A evaluates the consistency between the description of the test conditions 110a and the description of the test procedure or test command 120a for the evaluation items (pre-environment) 161a and evaluation items (observation variables) 162a in the evaluation perspectives of ``pre-environment'' and ``observation variables'', respectively, as a reliability score indicated as either ``pass'' or ``fail''.

[0083] In other words, by dividing the evaluation into multiple evaluation perspectives, the evaluation AI model 170A can fully understand the evaluation rules for each evaluation perspective from the training data, thereby calculating a more accurate reliability score. Furthermore, when processing common features for each evaluation perspective, the relationships between them become complex and difficult to recognize. Therefore, dividing the evaluation into multiple evaluation perspectives improves the accuracy of recognizing the evaluation rules. Furthermore, by evaluating the consistency between the description of the test conditions 110a and the description of the test procedure or test command 120a for each of the multiple evaluation perspectives, it is possible to identify problematic areas in the test. Furthermore, in this embodiment, multiple evaluation perspectives can be processed in parallel using limited resources. Therefore, accurate evaluation is possible in a short time, even for a large number of complex test conditions 110a and test procedure or test command 120a.

[0084] The first extracted AI model 130A and the second extracted AI model 140A are encoders 300 that constitute a transformer. Therefore, the attention mechanism allows the test conditions 110a and the test procedures or test commands 120a to be accurately processed in a short time, even if the descriptions of the test conditions 110a and the test procedures or test commands 120a are voluminous and complex. The evaluation AI model 170A also includes encoders 173 and 174, allowing the evaluation items 160a to be accurately processed in a short time, even if the descriptions of the evaluation items 160a are voluminous and complex.

[0085] [Second Example] The test evaluation system of the second embodiment will be described below. As shown in FIG. 19, the test evaluation system 1B of this embodiment executes processing using "voltage" and "current" as evaluation perspectives. As shown in FIGS. 19, 20, and 21, the first extraction AI model 130B extracts test condition (voltage) 111d, which is a description related to the evaluation perspective "voltage." Furthermore, the first extraction AI model 130B extracts test condition (current) 112d, which is a description related to the evaluation perspective "current."

[0086] 19, 20, and 22, the second extraction AI model 140B extracts a test procedure or test command (voltage) 121d, which is a description related to the evaluation viewpoint "voltage." The second extraction AI model 140B also extracts a test procedure or test command (current) 122d, which is a description related to the evaluation viewpoint "current."

[0087] 19, the evaluation item integrating unit 150 integrates the input test condition (voltage) 111d, test condition (current) 112d, test procedure or test command (voltage) 121d, and test procedure or test command (current) 122d, and outputs evaluation item 160d. The evaluation item integrating unit 150 integrates the test condition (voltage) 111d and the test procedure or test command (voltage) 121d related to the evaluation perspective "voltage" to generate evaluation item 161d for the evaluation perspective "voltage". The evaluation item integrating unit 150 integrates the test condition (current) 112d and the test procedure or test command (current) 122d related to the evaluation perspective "current" to generate evaluation item 162d for the evaluation perspective "current".

[0088] The evaluation AI model 170B is a plurality of evaluation AI models (voltage) 171B and evaluation AI models (current) 172B corresponding to the evaluation item (voltage) 161d and the evaluation item (current) 162d in the evaluation perspectives "voltage" and "current" integrated by the evaluation item integration unit 150.

[0089] The evaluation AI model (voltage) 171B evaluates the consistency between the description of the test condition (voltage) 111d and the description of the test procedure or test command (voltage) 121d for the evaluation item (voltage) 161d in the evaluation perspective "voltage" as a reliability score (voltage) 181d indicated as either (pass / 1) or (fail / 0).

[0090] The evaluation AI model (current) 172B evaluates the consistency between the description of the test condition (current) 112d and the description of the test procedure or test command (current) 122d for the evaluation item (current) 162d in the evaluation perspective ``current'' as a reliability score (current) 182d indicated as either (pass / 1) or (fail / 0).

[0091] The preparation of training data for the first extracted AI model 130B and the second extracted AI model 140B will be described below. As shown in FIG. 23, to prepare training data for the first extracted AI model 130B, a first training data preparation unit (test conditions) 131B, which has been configured similarly to the first extracted AI model 130B and has already been trained, is used. The first training data preparation unit (test conditions) 131B extracts test conditions (voltage) 111d and test conditions (current) 112d from the description of the test conditions 110a in the same manner as in the first embodiment. The test conditions 110a, test conditions (voltage) 111d, and test conditions (current) 112d are supplied as training data for the first extracted AI model 130B.

[0092] To prepare training data for the second extracted AI model 140B, a second training data preparation unit (test procedure or test command) 141B, which has been configured similarly to the second extracted AI model 140B and has been trained, is used. The second training data preparation unit (test procedure or test command) 141B extracts test procedure or test command (voltage) 121d and test procedure or test command (current) 122d from the description of the test procedure or test command 120a in the same manner as in the first embodiment. The test procedure or test command 120a, the test procedure or test command (voltage) 121d, and the test procedure or test command (current) 122d are supplied as training data to the second extracted AI model 140B.

[0093] The training data for the evaluation AI model 170B will be described below. As shown in FIG. 24, the first extraction AI model (test condition) 130B, to which the description of the test condition 110e has been input, extracts the description of the test condition (voltage) 111e and the description of the test condition (current) 112e for each of the evaluation perspectives "voltage" and "current." The second extraction AI model (test procedure or test command) 140B, to which the description of the test procedure or test command 120e has been input, extracts the description of the test procedure or test command (voltage) 121e and the description of the test procedure or test command (current) 122e for each of the evaluation perspectives "voltage" and "current." For ease of explanation, the first input unit 119 and the second input unit 129 are not shown in the figure.

[0094] The description of the test condition (voltage) 111e and the description of the test procedure or test command (voltage) 121e are assumed to be related to each other. The description of the test condition (current) 112e and the description of the test procedure or test command (current) 122e are assumed to be related to each other.

[0095] In this case, the evaluation item 161e integrated by the evaluation item integration unit 150 includes a description of the test condition (voltage) 111e and a description of the test procedure or test command (voltage) 121e, which are related to each other. The evaluation AI model (voltage) 171B outputs a reliability score (voltage) 181e indicating (pass / 1) as positive data.

[0096] The evaluation item 162e integrated by the evaluation item integration unit 150 includes a description of a test condition (current) 112e and a description of a test procedure or test command (current) 122e, which are related to each other. The evaluation AI model (current) 172B outputs a reliability score (current) 182e indicating (pass / 1) as positive data.

[0097] 25, the first extraction AI model (test condition) 130B to which the description of test condition 110f has been input extracts the descriptions of test condition (voltage) 111f and test condition (current) 112f in addition to the descriptions of test condition (voltage) 111e and test condition (current) 112e for each of the evaluation perspectives "voltage" and "current." The second extraction AI model (test procedure or test command) 140B to which the description of test procedure or test command 120f has been input extracts the descriptions of test procedure or test command (voltage) 121f and test procedure or test command (current) 122f for each of the evaluation perspectives "voltage" and "current."

[0098] The description of the test condition (voltage) 111e and the description of the test procedure or test command (voltage) 121f are assumed to be unrelated to each other. The description of the test condition (current) 112e and the description of the test procedure or test command (current) 122f are assumed to be unrelated to each other. The similarity confirmation unit 190 deletes the descriptions of the test condition (voltage) 111f and the test condition (current) 112f.

[0099] In this case, the evaluation item 161f integrated by the evaluation item integration unit 150 includes a description of the test condition (voltage) 111e and a description of the test procedure or test command (voltage) 121f, which are unrelated to each other. The evaluation AI model (voltage) 171B outputs a reliability score (voltage) 181f indicating (fail / 0) as negative data.

[0100] The evaluation items 162f integrated by the evaluation item integration unit 150 include descriptions of test conditions (current) 112e and descriptions of test procedures or test commands (current) 122f, which are unrelated to each other. The evaluation AI model (current) 172B outputs a reliability score (current) 182f indicating (fail / 0) as negative data. In this way, a set of training data can be obtained that includes positive data with a reliability score of (pass / 1) and negative data with a reliability score of (fail / 0).

[0101] In this embodiment, for example, for the evaluation viewpoints "voltage" and "current" which are important in testing the software and hardware functions of a vehicle inverter ECU, the consistency between the description of the test conditions 110a and the description of the test procedure or test command 120a can be evaluated as a reliability score indicated as either "good" or "bad," as in the first embodiment. Other evaluation viewpoints can also be set, such as "system speed," "motor," and "system temperature."

[0102] [Third Example] A test evaluation system according to a third embodiment will now be described. As shown in FIG. 26, the test evaluation system 1C according to this embodiment further includes a test generation AI model 200 that generates a description of a test procedure or test command 120a based on the description of the test condition 110a input to the first input unit 119. While the first extraction AI model 130A, the second extraction AI model 140A, and the evaluation AI model 170A use classification models, the test generation AI model 200 uses a large language model (LLM) that is fine-tuned for domain knowledge. The second input unit 129 receives as input the description of the test procedure or test command 120a generated by the test generation AI model 200 based on the description of the test condition 110a input to the first input unit 119.

[0103] In this embodiment, the test generation AI model 200 can evaluate the consistency of the description of the test procedure or test command 120a generated based on the description of the test condition 110a.

[0104] [Fourth Example] A test evaluation system according to a fourth embodiment will now be described. As shown in Fig. 27, in the test evaluation system 1D according to this embodiment, a first extraction AI model 130C extracts descriptions related to evaluation viewpoints from descriptions of test conditions 110g input to a first input unit 119 by pattern matching using regular expressions. A second extraction AI model 140C extracts descriptions related to evaluation viewpoints from descriptions of test procedures or test commands 120g input to a second input unit 129 by pattern matching using regular expressions.

[0105] The first extracted AI model 130C will be described below as a representative example. Figure 28 shows an example of pattern matching in which the first extracted AI model 130C uses regular expressions to match sentences containing letters in parentheses in ascending order from the beginning. From the description of test condition 110g, test condition 113a, which states "(a) Verification environment: XXX...YYY" from (a) onwards, and test condition 113b, which states "(b) Pre-environment AAA...CCC" from (b) onwards, are extracted. The same applies to the second extracted AI model 140C. Note that the letters in parentheses may be replaced with Arabic numerals and Roman numerals.

[0106] FIG. 29 shows an example of pattern matching using regular expressions in which the first extracted AI model 130C compares the test condition 110h entered into the first input unit 119 with a title in the predefined target title list 114. The first extracted AI model 130C receives the test condition 110h and the target title list 114. The target title list 114 defines, for example, the titles "Verification Environment" and "Preliminary Environment." From the test condition 110h, the following test condition 113c, "(c) Verification Environment: XXX...YYY" after "(c) Verification Environment," and the following test condition 113d, "(d) Preliminary Environment AAA...CCC" after "(d) Preliminary Environment," are extracted. The same applies to the second extracted AI model 140C.

[0107] FIG. 30 shows an example of pattern matching in which the first extracted AI model 130C uses regular expressions to match the test conditions 110i entered into the first input unit 119 with symbols in a predefined target symbol list 115. The test conditions 110i and the target symbol list 115 are input to the first extracted AI model 130C. For example, the symbol "###" is defined in the target symbol list 115. From the test conditions 110i, test conditions 113e, which begin with "###" and state "### verification environment: XXX...YYY," and test conditions 113f, which state "### pre-environment AAA...CCC," are extracted. The same applies to the second extracted AI model 140C.

[0108] According to this embodiment, the first extraction AI model 130C and the second extraction AI model 140C use pattern matching using regular expressions, which enables accurate pattern matching and reduces the risk of incorporating incorrect information. Furthermore, pattern matching using regular expressions can reduce resource consumption.

[0109] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be implemented in part or in whole in hardware, for example, by designing an integrated circuit, or entirely in software. [Explanation of symbols]

[0110] 1A, 1B, 1C, 1D Test Evaluation System 110a, 110b, 110c, 110e, 110f, 110g, 110h, 110i Test conditions 111a, 111b, 111c Test conditions (pre-test environment) 111d, 111e, 111f Test conditions (voltage) 112a, 112b, 112c Test conditions (observed variables) 112d, 112e, 112f Test conditions (current) 113a, 113b, 113c, 113d, 113e, 113f Test conditions 114 Target Title List 115 Goal Symbol List 119 First input section 120a, 120b, 120c, 120e, 120f Test procedure or test command 121a, 121b, 121c Test procedure or test command (pre-environment) 121d, 121e, 121f Test procedure or test command (voltage) 122a, 122b, 122c Test procedure or test command (observation variable) 122d, 122e, 122f Test procedure or test command (current) 129 Second Input Section 130A, 130B, 130C First Extraction AI Model 131A, 131B First training data preparation unit (test conditions) 140A, 140B, 140C Second Extraction AI Model 141A, 141B Second training data preparation unit (test procedure or test command) 150 Evaluation Item Integration Department 160 evaluation items 160a, 160d Evaluation items 161a, 161b, 161c Evaluation items (pre-environment) 161d, 161e, 161f Evaluation items (voltage) 162a, 162b, 162c Evaluation items (observed variables) 162d, 162e, 162f Evaluation items (current) 170A, 170B Evaluation AI model 171A Evaluation AI Model (Preliminary Environment) 171B Evaluation AI Model (Voltage) 172A Evaluation AI Model (Observed Variables) 172B Evaluation AI Model (Current) 173,174 Encoder 175 Multilayer perceptron 180a,180d reliability score 181a, 181b, 181c Reliability score (pre-environment) 181d, 181e, 181f Reliability score (voltage) 182a, 182b, 182c Reliability scores (observed variables) 182d, 182e, 182f Reliability score (current) 190 Similarity confirmation part 200 test generation AI models 300 Encoder 301 Input Embedding Layer 302 Positional Encoding Layer 303 Stack 304 Multi-Head Self-Attention Layer 305 Add and Normalization Layer 306 Feed-Forward Layer 307 Add and Normalization Layer 310 Scaled Dot-Product Attention 311 Matrix Multiplication 312 Scale 313 Mask 314 Softmax function 315 Matrix Multiplication 321,322,323 Linear Layer 324 Concatenate Layer 325 Linear Layer 401 Input Layer 402 Hidden Layer 403 Output Layer 404 Node 405 Edge

Claims

1. a first input section into which test conditions are input; a second input unit into which a description of either a test procedure or a test command generated based on the description of the test conditions is input; a first extraction AI model that extracts descriptions related to predefined evaluation viewpoints from the descriptions of the test conditions input to the first input unit; a second extraction AI model that extracts a description related to the evaluation viewpoint from a description of either the test procedure or the test command input to the second input unit; an evaluation item integration unit that integrates the description of the test conditions extracted by the first extraction AI model and the description of either the test procedure or the test command extracted by the second extraction AI model to set the evaluation items in the evaluation perspective; an evaluation AI model that evaluates the consistency between the description of the test conditions and the description of either the test procedure or the test command for the evaluation items in the evaluation perspective integrated by the evaluation item integration unit; A test evaluation system comprising:

2. The first extraction AI model extracts descriptions related to the evaluation viewpoints for each of the plurality of evaluation viewpoints from the descriptions of the test conditions input to the first input unit, The second extraction AI model extracts a description related to the evaluation viewpoint for each of the plurality of evaluation viewpoints from the description of either the test procedure or the test command input to the second input unit, the evaluation item integration unit integrates, for each of a plurality of evaluation perspectives, the descriptions of the test conditions related to the evaluation perspectives extracted for each of the evaluation perspectives by the first extraction AI model and the descriptions of either the test procedures or the test commands related to the evaluation perspectives extracted for each of the evaluation perspectives by the second extraction AI model, to define the evaluation items for each of the evaluation perspectives; The evaluation AI model is a plurality of evaluation AI models respectively corresponding to the evaluation items in each of the evaluation viewpoints integrated by the evaluation item integration unit, Each of the plurality of evaluation AI models evaluates the consistency between the description of the test conditions and the description of either the test procedure or the test command for the evaluation items in each of the evaluation perspectives as a reliability score indicated as either "pass" or "fail." 2. The test evaluation system of claim 1.

3. The evaluation AI model is Positive data in which the description of the test condition and the description of either the test procedure or the test command are consistent with each other and the reliability score is acceptable; Negative data in which the description of the test conditions and the description of either the test procedure or the test command are inconsistent with each other and the reliability score is unacceptable; trained with a set of training data including 3. The test evaluation system according to claim 2.

4. The first extracted AI model is decomposing the description of the test condition input to the first input unit into tokens; determining which of the plurality of evaluation perspectives each token is associated with by calculating a probability that each token is associated with each of the plurality of evaluation perspectives; By combining tokens related to the same evaluation perspective, extracting descriptions related to the evaluation viewpoints for each of the plurality of evaluation viewpoints from the descriptions of the test conditions input to the first input unit; The second extracted AI model is decomposing the description of either the test procedure or the test command input to the second input unit into tokens; determining which of the plurality of evaluation perspectives each token is associated with by calculating a probability that each token is associated with each of the plurality of evaluation perspectives; By combining tokens related to the same evaluation perspective, extracting a description related to the evaluation viewpoint for each of the plurality of evaluation viewpoints from the description of either the test procedure or the test command input to the second input unit; 3. The test evaluation system according to claim 2.

5. The first extracted AI model is training is performed using descriptions related to the evaluation viewpoints extracted for each of the plurality of evaluation viewpoints from the descriptions of the test conditions as training data; The second extracted AI model is The training is performed using descriptions related to the evaluation viewpoints extracted for each of the plurality of evaluation viewpoints from descriptions of either the test procedure or the test command as training data.

3. The test evaluation system according to claim 2.

6. a test generation AI model that generates a description of either the test procedure or the test command based on the description of the test condition input to the first input unit; The second input unit receives a description of either the test procedure or the test command generated by the test generation AI model based on the description of the test condition input to the first input unit.

2. The test evaluation system of claim 1.

7. the first extraction AI model extracts descriptions related to the evaluation viewpoints from the descriptions of the test conditions input to the first input unit by pattern matching using regular expressions; The second extraction AI model extracts a description related to the evaluation viewpoint from a description of either the test procedure or the test command input to the second input unit by pattern matching using a regular expression.

2. The test evaluation system of claim 1.

8. the first extracted AI model is an encoder that constitutes a transformer; The second extracted AI model is an encoder that configures a transformer.

3. The test evaluation system according to claim 2.

9. 3. The test evaluation system according to claim 2, wherein the evaluation AI model includes an encoder that configures a transformer.

Images