System verification method

Abstract

To perform verification with higher consistency with respect to an entire workflow system that realizes coordination with externally provided components whose behavior is uncertain and difficult to be predicted.SOLUTION: A computer 70: observes inputs for a process task block that constitutes a workflow system and calls external software whose outputs fluctuate in response to inputs, and outputs in response to the inputs; creates a test pattern that associates expected outputs with inputs of the workflow system while the process task block is being executed based on the inputs and outputs of the process task block; determines whether or not behavior of the process task block is normal based on a similarity between actual outputs in response to inputs of the workflow system while the process task block is being executed and the expected outputs in response to the inputs of the workflow system registered in the test pattern; and verifies the workflow system from the results of determining the behavior of the process task block.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a system verification method for verifying a workflow system. [Background technology]

[0002] Recently, a development method that calls external software as a service (SaaS) and integrates it on a workflow system to realize a desired workflow system has been attracting attention. The quality of the entire workflow system is guaranteed by comprehensively testing all possible paths within the learning model (relationships from input to output).To assure the quality of externally provided components (software from external services), a commonly used method is to infer the behavior of the externally provided components from the published API (Application Programming Interface) specifications and perform black-box testing.

[0003] The above-mentioned testing techniques for workflow systems are effective when the logic of the workflow system is open. Furthermore, testing techniques for externally provided components are effective for testing externally provided components with uniform behavior. However, there are limitations to testing when the workflow system logic is a black box, or when the workflow system or externally provided components use software with statistical behavior, such as a large language model (LLM).

[0004] Metamorphic testing is one of the testing methods for models (machine learning models) obtained through machine learning (ML). In metamorphic testing, input data and models are changed to predict the behavior of the target system and then the test is performed. Additionally, one of the testing methods for models obtained through deep learning (deep learning models) is "neuron coverage testing." In neuron coverage testing, all paths of the model are identified and tests are run by providing test inputs that more broadly cover the model's behavior. Neuron coverage is defined as the percentage of neurons activated by the test inputs (the output value of the activation function is above a threshold).

[0005] Patent Document 1 discloses a technology in which partial code, which is part of the code of a program that constitutes software, is extracted, a feature vector is generated based on the partial code, and the partial code to be verified is input into a performance verification model, which is a machine learning model trained using training data including the feature vector of the partial code for training and performance information, which is information indicating the performance of software realized based on the partial code, to generate information based on the output obtained as a verification result of the partial code. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-138708 Summary of the Invention [Problem to be solved by the invention]

[0007] The technology described in Patent Document 1 generates feature vectors based on partial code of programs that make up the software, and is therefore effective when the logic of the software is open as described above.

[0008] In conventional models such as ML, although there is some fluctuation in behavior, it is easy to predict the behavior of the model because it is specialized for a specific purpose and has a simple model structure. However, in LLM models, the purpose is general-purpose, the model structure is complex, and it is a black box, making it difficult to predict the behavior of the model.

[0009] The present invention has been made in light of the above situation, and aims to achieve more stable verification of the entire workflow system that realizes collaboration with externally provided components (services provided by external software) whose behavior is uncertain and difficult to predict. [Means for solving the problem]

[0010] In order to solve the above problem, one aspect of the present invention is a system verification method for verifying a workflow system in which a computer having an arithmetic unit that executes a program and a storage device that stores the program calls and sequentially executes one or more external software programs that provide services. The computer observes the inputs and outputs of processing task blocks that constitute a workflow system and call external software whose output fluctuates with respect to the inputs. Next, based on the inputs and outputs of the observed processing task block, the computer creates a test pattern that associates expected outputs with inputs of the workflow system during execution of the processing task block. Next, when executing a test using the test pattern, the computer calculates the similarity between the actual output for the input of the workflow system while executing the processing task block and the expected output for the input of the workflow system registered in the test pattern. The computer then determines whether the behavior of the process task block is normal or not based on the similarity between the two outputs, and verifies the operation of the workflow system based on the result of determining the behavior of the process task block. [Effects of the Invention]

[0011] According to at least one aspect of the present invention, it is possible to achieve more stable verification of the entire workflow system that realizes cooperation with externally provided components (services provided by external software) whose behavior is uncertain and difficult to predict. Problems, configurations, and effects other than those described above will become apparent from the following description of the preferred embodiments of the invention. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates a conventional method for verifying a workflow system. [Figure 2] 10 is a flowchart illustrating an example of a process for creating a test pattern for verifying a conventional workflow system. [Figure 3] 10 is a flowchart illustrating an example of a process for executing an E2E test to verify a conventional workflow system. [Figure 4] FIG. 1 is a diagram showing an overview of a method for verifying a workflow system according to a first embodiment of the present invention. [Figure 5] 1 is a block diagram showing an example of the configuration of a test platform that executes tests on a workflow system according to a first embodiment of the present invention. [Figure 6] 1 is a diagram illustrating an example of the hardware configuration of a service collaboration platform (workflow system) and a test platform according to a first embodiment of the present invention. [Figure 7] 10 is a flowchart illustrating an example of a service profile creation process in a test board according to the first embodiment of the present invention. [Figure 8] FIG. 2 is a diagram showing an example of an editing screen for a service profile created by a test platform according to the first embodiment of the present invention. [Figure 9] 10 is a flowchart illustrating an example of a test pattern generation process in a test board according to the first embodiment of the present invention. [Figure 10] 10 is a flowchart illustrating an example of an E2E test execution process in a test board according to the first embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing an example of a test result screen showing the test results of an arbitrary service profile executed by a test platform according to the first embodiment of the present invention. [Figure 12]FIG. 10 is a diagram showing an example of a test result screen showing the test results of each service profile and the entire workflow executed by the test platform according to the first embodiment of the present invention. [Figure 13] FIG. 10 is a diagram illustrating an overview of a method for verifying a workflow system according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, examples of modes for carrying out the present invention (hereinafter referred to as "embodiments") will be described with reference to the accompanying drawings.

[0014] In this specification and the accompanying drawings, identical or similar components are given the same reference numerals, and redundant explanations may be omitted or only explanations focusing on the differences may be given. Furthermore, when there are multiple identical or similar components, they may be described using the same reference numerals with different subscripts. Note that when it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description. The number of each component may be singular or plural unless otherwise specified.

[0015] In the following embodiment, various types of information are described in table format, but the various types of information may be in a data format other than a table format. Furthermore, various names such as "XX information," "XX table," "XX list," and "XX list" are interchangeable. Furthermore, when describing identification information, expressions such as "identification information," "name," and "ID" are used, but these are interchangeable.

[0016] <Method for verifying conventional workflow systems> First, before describing the method for verifying a workflow system of the present invention, a conventional method for verifying a workflow system will be described with reference to FIGS. 1 to 3. FIG. FIG. 1 is a diagram illustrating a conventional method for verifying a workflow system.

[0017] In Figure 1, a workflow system 11 that calls one or more external software programs that provide services and executes them sequentially is built on a service collaboration platform 10. The service collaboration platform 10 is the platform for the workflow system 11. The workflow system 11 has back-end processing that executes application software when it receives data or instructions, and realizes a workflow by executing the back-end processing.

[0018] The workflow system 11 calls external software 21 and external software 22 to use the services provided by the external software 21 and external software 22. The external software 21 and 22 are, for example, software as a service (SaaS). In FIG. 1, the external software 21 is denoted as "external SaaS1" and the external software 22 is denoted as "external SaaS2." The external software 21 has an API definition 23 that defines the procedures and data formats for using the service. Similarly, the external software 22 has an API definition 24. The external software 21 and 22 are a group of SaaS that exhibit deterministic behavior.

[0019] The workflow system 11 is made up of, for example, a start block 12, a process task block 13, a process task block 14, and an end block 15.

[0020] When the initiation block 12 receives an input such as an execution instruction (1), it passes the contents of the input to the processing task block 13.

[0021] The processing task block 13 uses the services of external software 21 (external SaaS1) via a communication network (cloud) such as the Internet, for example, to collect necessary information (2). Then, the processing task block 13 passes the collected information to the processing task block 14.

[0022] The processing task block 14 uses the services of external software 22 (external SaaS2) via the cloud, for example, and performs aggregation and analysis based on the information received from the processing task block 13 (3). The processing task block 14 passes the results of the aggregation and analysis to the termination block 15.

[0023] The end block 15 outputs the results of the processing (for example, aggregation and analysis) received from the processing task block 14 to the test board 30 (4).

[0024] The test infrastructure 30 is a platform that verifies the operation of the workflow system 11 based on the input of the workflow system 11 on the service collaboration infrastructure 10 and the output corresponding to that input. The test infrastructure 30 determines whether the behavior of the workflow system 11 is normal by, for example, executing E2E testing (End to End testing). In E2E testing, a test of the target system is executed based on a test pattern in which inputs and expected outputs are registered. Conventional E2E test cases use deterministic and strict test patterns. Figure 1 shows an example of pattern 1 with input "XX" and output "ZZ," and pattern 2 with input "YY" and output "AA."

[0025] Next, a process for creating a test pattern for verifying the conventional workflow system 11 will be described with reference to FIG. 2 is a flowchart showing an example of a process for creating a test pattern for verifying the conventional workflow system 11. Numbers in parentheses in the flowchart correspond to the numbers in parentheses in FIG.

[0026] First, the test platform 30 analyzes the logic of the workflow system 11 on the service collaboration platform 10 and the external SaaS group (step S1). In the example of Figure 1, the external SaaS group includes external SaaS1 and external SaaS2. Here, although there is some variation in the model behavior (output for the same input), it is assumed that the model behavior is easy to predict because the application is specialized and the model structure is simple. A simple example is when the output is uniquely determined for the input.

[0027] Next, the test platform 30 lists possible input patterns based on the analysis results and calculates the expected output corresponding to the input (step S2). Then, by registering the input and expected output in association with each other, a test pattern such as that shown in Fig. 1 can be created. After step S2 is completed, the processing of this flowchart ends.

[0028] Next, a process for executing an E2E test to verify the conventional workflow system 11 will be described with reference to FIG. FIG. 3 is a flowchart showing an example of a process for executing an E2E test to verify the conventional workflow system 11.

[0029] First, the test platform 30 inputs the input contents of the test pattern to the start block 12 of the workflow system 11 on the service collaboration platform 10 (step S11).

[0030] Next, the test platform 30 compares the expected output registered in the test pattern with the actual output for the input of the workflow system 11 (step S12). If the expected output registered in the test pattern for the same input and the actual output are the same, the test platform 30 determines that the test pattern (combination of input and output) is correct. After step S12 is completed, the processing of this flowchart ends.

[0031] Conventionally, external software called from the workflow system 11 is a group of SaaS that exhibit deterministic behavior, so the output of the external software for the same input is the same. Even if the behavior of the external software (SaaS) fluctuates, it is easy to predict the behavior of the model because the application is specialized and the model structure is simple. However, models using large-scale language models (LLMs) and the like are general-purpose, have complex model structures, and are black boxes, making it difficult to predict the behavior of the model.

[0032] First Embodiment Next, a method for verifying a workflow system according to the first embodiment of the present invention will be described with reference to FIG.

[0033] [System verification method overview] FIG. 4 is a diagram showing an outline of a method for verifying a workflow system according to the first embodiment of the present invention. The service collaboration platform 40 shown in Figure 4 is an example of a platform for building a system. A workflow system 41 is built in the service collaboration platform 40, which calls one or more external software programs that provide services and executes them sequentially. The workflow system 41 has back-end processing that executes application software when it receives data or instructions, and realizes a workflow by executing the back-end processing.

[0034] The workflow system 41 calls external software 51 and external software 52 to use the services provided by the external software 51 and external software 52. The external software 51 and 52 are, for example, software as a service (SaaS). In FIG. 4, the external software 51 is denoted as "external SaaS1a" and the external software 52 is denoted as "external SaaS2a." The external software 51 has an API definition 53 that defines the procedures, data formats, etc. for using the service. Similarly, the external software 52 has an API definition 54.

[0035] The external software 51 and 52 are components (SaaS groups) that exhibit uncertain (probabilistic) behavior. The services of the external software 51 and 52 are provided using, for example, a large-scale language model (LLM). Therefore, the services of the external software 51 and 52 include fluctuations in their behavior. As a result, the outputs of the processing task block 43 and the processing task block 44 also fluctuate.

[0036] As an example, the workflow system 41 is composed of elements including a start block 42, a processing task block 43 (processing), a processing task block 44, and an end block 45. The functions of the start block 42 and the end block 45 are basically the same as the functions of the start block 12 and the end block 15 in FIG.

[0037] When the start block 42 receives an input such as an execution instruction (1), it passes the contents of the input to the process task block 43.

[0038] The processing task block 43 uses the services of external software 51 (external SaaS1a) via a communication network (cloud) such as the Internet, for example, and collects necessary information (2). Then, the processing task block 43 passes the collected information to the processing task block 44. Note that although the processing task blocks 43, 43 each call one external software (SaaS), one processing task block may be configured to call multiple external software.

[0039] The processing task block 44 uses the services of external software 52 (external SaaS 2a) via the cloud, for example, and performs processing based on the information received from the processing task block 43 (3). In this embodiment, the processing task block 44 summarizes the received information and passes the summarized results to the end block 45. As an example, the processing task block 44 is configured using generative AI (artificial intelligence). The processing task block 44 interacts with the external software 52 (external SaaS 2a) by communicating with the external software 52.

[0040] Generative AI is a type of AI system that applies machine learning technology and is capable of generating large amounts of new information, such as text, images, audio, video, program code, etc. One example of generative AI is the use of large-scale language models (LLMs).

[0041] The end block 45 outputs the results of the processing (eg, summary) received from the processing task block 44 to the test board 60 (4).

[0042] The test infrastructure 60 is a platform that verifies the operation of the workflow system 41 based on the input of the workflow system 41 on the service collaboration infrastructure 40 and the output corresponding to the input. The test infrastructure 60 determines whether the behavior of the workflow system 41 is normal by, for example, executing an E2E test. In the E2E test, a test of the target system is executed based on a test pattern in which inputs and expected outputs are registered.

[0043] In the E2E test case of this embodiment, a test pattern that allows for output fluctuations is used. In the test pattern 66 (an example of a test pattern list) shown in Fig. 4, examples are shown in which input "XX" and output "ZZ" are used as pattern 1, input "YY" and output "AA" are used as pattern 2, and input "YY" and output "BB" are used as pattern 3. In order to allow for output fluctuations, there are outputs "AA" and "BB" for the same input "YY".

[0044] The test pattern 66 is created based on the service profile 64. The service profile 64 is created by the test platform 60 by analyzing the output trends of the processing task block for the same input, and by associating the input and output of the processing task block when the external software is called, based on a model that has learned the normal behavior of the external software. The service profile 64 contains information that reflects the fluctuations in the behavior of the external software, that is, information on both normal and abnormal outputs for the same input. As the service profile thus represents the fluctuations in output for each external software (external service), the service profile can also be called a "fluctuation table." The service profile 64 and test pattern 66 will be described in detail later.

[0045] In the present invention, the testing method for the workflow system 41 is not limited to E2E testing, and any method that can verify the entire workflow system may be used.

[0046] [Test infrastructure configuration] Next, the configuration of the test platform 60 that executes tests on the workflow system 41 according to this embodiment will be described with reference to FIG. FIG. 5 is a block diagram showing an example of the configuration of a test platform 60 that executes tests on the workflow system 41. As shown in FIG.

[0047] The test infrastructure 60 is composed of, for example, an observation unit 61, a modeling unit 62, a service profile creation unit 63, a service profile 64, a test pattern creation unit 65, a test pattern 66, and a test execution unit 67. The test infrastructure 60 has a "workflow behavior learning mode" for observing and learning the behavior of the workflow system 41, and a "workflow test mode" for executing tests of the workflow system 41.

[0048] In the workflow behavior learning mode, the observation unit 61 observes the inputs and outputs of processing task blocks that constitute the workflow system 41 and call external software whose output fluctuates in response to the inputs. In this embodiment, the observation unit 61 observes the inputs and outputs of processing task blocks 43 and 44, and passes the respective observation results to the modeling unit 62. Furthermore, when statically analyzing the logic of the workflow system 41, the observation unit 61 passes the analysis results to the modeling unit 62.

[0049] The modeling unit 62 analyzes the tendency of the output of the processing task block for the same input from the observation results of the input and output of the processing task block by the observation unit 61, and learns the normal behavior of the external software to create a model. The created model is stored in the RAM 73 or non-volatile storage 76 (see FIG. 6). Then, the modeling unit 62 passes the created model to the service profile creation unit 63. For example, the modeling unit 62 creates a machine learning model by learning using a machine learning technique.

[0050] The service profile creation unit 63 creates a service profile 64 that associates inputs and outputs of processing task blocks when external software is called, based on the model created by the modeling unit 62. The created service profile 64 is stored in the RAM 73 or non-volatile storage 76. A service profile is created for each call pattern of external software.

[0051] (Service Profile) As shown in FIG. 4, the service profile 64 has the following items for each piece of pattern identification information: service, flow, input, output, and correct / incorrect. "Service" is information indicating a service provided by external software called by a processing task block, and stores information identifying the service. "Flow" stores information for identifying a workflow system. "Input" stores information about data input to a processing task block that calls external software. "Output" stores information about data output to the input of the processing task block that calls external software. "True / False" stores information indicating whether the relationship between the input and output of the processing task block that calls external software is true (correct) or false (incorrect). If the relationship is correct, "T" is stored, and if it is incorrect, "F" is stored.

[0052] As an example, in the case of a service profile for a processing task block 44 that calls external software 52 (SaaS2a), in pattern 1, the service is "SaaS2a", the flow is "F1", the input is "XX1", the output is "ZZ1", and the correct / incorrect information is "T". In the case of pattern 2, the service is "SaaS2a", the flow is "F1", the input is "XX1", the output is "KK1", and the correct / incorrect value is "F". In the case of pattern 3, the service is "SaaS2a", the flow is "F1", the input is "XX1", the output is "JJ1", and the correct / incorrect value is "F". In this way, for the same input “XX1”, only the output “ZZ1” is correct or incorrect “T”, and the relationship between the input and output is correct.

[0053] In the case of pattern 4, the service is "SaaS2a", the flow is "F1", the input is "YY1", the output is "AA1", and the correct / incorrect value is "T". In the case of pattern 5, the service is "SaaS2a", the flow is "F1", the input is "YY1", the output is "BB1", and the correct / incorrect value is "T". In patterns 4 and 5, the output for the same input “YY1” is different, “AA1” and “BB1”, but the external software 52 (SaaS2a) determines that the relationship between the input and output is correct, resulting in a correct / incorrect “T”.

[0054] Based on the observed input and output of the processing task block, the test pattern creation unit 65 creates a test pattern 66 that associates expected outputs with inputs of the workflow system 41 while the processing task block is being executed. More specifically, based on the inputs and outputs registered in the service profile 64, the test pattern creation unit 65 creates a test pattern 66 that associates expected outputs with inputs of the workflow system 41 while the processing task block is being executed by calling external software. The created test pattern 66 is stored in the RAM 73 or non-volatile storage 76. A test pattern is created for each call pattern of the external software, i.e., for each service profile.

[0055] (Test pattern) The test pattern creation unit 65 extracts patterns with a correct / incorrect result of "T" from the patterns in the service profile 64 and registers them in a list of test patterns. As shown in FIG. 4, the test pattern 66 has at least the following items for each piece of pattern identification information: input, output, and correct / incorrect. The information stored in these items is the same as that in the service profile 64. However, since the test pattern 66 is based on the assumption that the correct / incorrect result is "T," the correct / incorrect item may be deleted.

[0056] As a premise, the test pattern creation unit 65 understands the relationship (conversion) between the input / output of the workflow system 41 and the input / output of each processing task block. In other words, the test pattern creation unit 65 can infer the input / output of each processing task block from the input / output of the workflow system 41. Conversely, the test pattern creation unit 65 can infer the input / output of the workflow system 41 from the input / output of each processing task block. Therefore, from the input "XX1" and output "ZZ1" of the processing task block 44, the test pattern creation unit 65 sets the input of the test pattern for the workflow system 41 to "XX" and the output to "ZZ".

[0057] As an example, in the case of a test pattern 66 for a processing task block 44 that calls the external software 52 (SaaS2a), in the case of pattern 1, the input is "XX" and the output is "ZZ". In pattern 2, the input is "YY" and the output is "AA". In pattern 3, the input is "YY" and the output is "BB". Patterns 1 to 3 of the test pattern 66 correspond to patterns 1 and patterns 4 and 5 of the service profile 64 .

[0058] With this configuration, in this embodiment, it is possible to create a service profile and a test pattern that reflect the behavior of external software called by a processing task block that is currently being executed.

[0059] In workflow test mode, the test execution unit 67 inputs input data to the workflow system 41 for each pattern based on the test pattern 66, and receives the output for that input. When executing a test using the test pattern 66, the test execution unit 67 calculates the similarity between the actual output for the input of the workflow system 41 while the processing task block is being executed and the expected output for the input of the workflow system 41 registered in the test pattern 66. The test execution unit 67 then determines whether the behavior of the processing task block is normal or not based on the similarity between the two outputs, and verifies the operation of the workflow system 41 based on the result of determining the behavior of the processing task block. The test execution unit 67 outputs the results of the verification of the workflow system 41 (test results) to a display device 74 (see FIG. 6 ) or the like.

[0060] The test infrastructure 60 may be incorporated into the service collaboration infrastructure 40, and the test infrastructure 60 may be configured as an integrated part of the service collaboration infrastructure 40. In other words, the service collaboration infrastructure 40 may be configured with a processing task block having the function of verifying the workflow system 41.

[0061] [Hardware configuration of the service integration platform and test platform] Here, the hardware configuration of the service collaboration platform 40 (workflow system 41) and the test platform 60 will be described with reference to FIG. 6 is a diagram showing an example of the hardware configuration of the service collaboration platform 40 (workflow system 41) and the test platform 60. Note that the service collaboration platform 40 and the test platform 60 may share the same hardware.

[0062] The calculator 70 shown in Fig. 6 is an example of hardware used as a computer. The service collaboration platform 40 and the test platform 60 according to this embodiment implement a system verification method in which the blocks shown in Figs. 4 and 5 cooperate with each other by executing a program on the respective calculators 70 (computers).

[0063] The computer 70 includes a CPU (Central Processing Unit) 71, a ROM (Read Only Memory) 72, and a RAM (Random Access Memory) 73, all connected to a system bus. The system bus also includes a display device 74, an input device 75, a non-volatile storage 76, and a communication interface 77.

[0064] The CPU 71 reads the program code of the software that realizes each function of the service collaboration platform 40 and the test platform 60 from the ROM 72, loads it into the RAM 73, and executes it. Variables, parameters, etc. generated during the calculation processing of the CPU 71 are temporarily written to the RAM 73, and these variables, parameters, etc. are read by the CPU 71 as appropriate. The CPU 71 executes the program code read from the ROM 72 to realize the functions of the blocks of the service collaboration platform 40 (FIG. 4) and the test platform 60 (FIG. 5). However, other processors such as an MPU (Micro Processing Unit) may be used instead of the CPU 71. The CPU 71 is an example of a calculation device, and the RAM 73 and non-volatile storage 76 are examples of storage devices.

[0065] The display device 74 is a monitor such as an LCD display, and displays a GUI (Graphical User Interface) screen, the results of arithmetic processing by the CPU 71, etc. The input device 75 generates an input signal in response to a user's operation and outputs it to the CPU 71. The input device 75 may be, for example, a mouse, a keyboard, or a touch sensor, and the user can operate the input device 75 to input information and instructions. The display device 74 and the input device 75 may be integrated into a touch panel. It is sufficient that there is at least one display device 74 and one input device 75 for the service collaboration platform 40 and the test platform 60.

[0066] The nonvolatile storage 76 is an example of a non-transitory recording medium, and is capable of storing data used by a program, data obtained by executing a program, etc. The service profile 64 and the test pattern 66 are stored in the nonvolatile storage 76. The nonvolatile storage 76 may also store an operating system (OS) or a program executed by the CPU 71. The nonvolatile storage 76 may be a hard disk drive (HDD), a solid state drive (SSD), an optical or magnetic disk medium, a semiconductor memory card, or the like.

[0067] The communication interface 77 is implemented by a communication device such as a network interface card (NIC). The communication interface 77 can transmit and receive various data to and from external devices via a communication network such as a LAN connected to a terminal of the NIC, a dedicated line, etc. The communication between the service collaboration platform 40 and the test platform 60 and other application platforms is realized by the communication interface 77.

[0068] [Service profile creation process (first example)] Next, the service profile creation process in the test board 60 according to this embodiment will be described with reference to FIG. FIG. 7 is a flowchart showing an example (first example) of a service profile creation process in the test platform 60. As shown in FIG.

[0069] First, the service profile creation unit 63 (FIG. 5) of the test platform 60 lists possible call patterns for the call of the external software (SaaS) of the workflow from the API specification and the workflow definition (step S21). In the example of the workflow system 41 (FIG. 4), the API specification is API definition 53 and API definition 54. When predicting the behavior of the external software (SaaS), the behavior of the external software (SaaS) can be limited by referring to the API specification.

[0070] Next, the service profile creation unit 63 lists possible input patterns for the workflow system 41 and analyzes the output tendency for the same input by simulation or trial observation (step S22). Next, the service profile creation unit 63 creates a service profile as shown in Fig. 4 based on the analysis results (step S23).

[0071] Next, the service profile creation unit 63 adds RLHF (Reinforcement Learning from Human Feedback) to the output of the workflow system 41 (step S24). That is, the service profile creation unit 63 applies manual feedback to the service profile to optimize it. For example, the user checks the content of each item for each service profile pattern and operates the input device 75 to edit the content of each item of the service profile as appropriate.

[0072] The service profile creation unit 63 creates a service profile for each call pattern of the external software (SaaS), i.e., for each processing task block that calls the external software (SaaS). After the process of step S24 is completed, once it is confirmed that the creation of service profiles for all call patterns has been completed, the process of this flowchart ends.

[0073] [Edit Service Profile screen] Here, an editing screen for a service profile created by the test platform 60 according to this embodiment will be described with reference to FIG. FIG. 8 is a diagram showing an example of an editing screen for a service profile created by the test platform 60. As shown in FIG.

[0074] As shown in Fig. 8, the service profile editing screen 80 has an area for displaying the service profile 64, a change button 81, and a confirm button 82. The service profile 64 displayed on the editing screen 80 is designed to indicate which service (external software) it corresponds to. The example in Fig. 8 is a service profile corresponding to the processing task block 44 when external software 52 (SaaS2a) is called.

[0075] When the user presses the change button 81, the content of the specified item can be edited. In Fig. 8, the correct / incorrect item of pattern 5 is specified. After editing, when the user presses the confirm button 82, the content of the service profile is confirmed and the service profile is saved in the RAM 73 or the non-volatile storage 76.

[0076] [Service profile creation process (second example)] Next, a service profile creation process (second example) in the test infrastructure 60 according to this embodiment will be described. The second example is a method for creating a service profile when the logic of the workflow system 41 is open.

[0077] The modeling unit 62 of the test platform 60 learns a model based on the inputs and outputs of the processing task blocks observed by the observation unit 61 and the results of static analysis of the logic of the workflow system 41. Then, the service profile creation unit 63 creates a service profile based on the learned model.

[0078] With this configuration, it is possible to create a service profile that more accurately reflects the behavior of the workflow system (processing task block).

[0079] [Service profile creation process (third example)] Next, a service profile creation process (third example) in the test platform 60 according to this embodiment will be described. The third example is a method for creating a service profile by externally observing the called external software (SaaS). This method is based on the premise that the called external software is a black box.

[0080] The observation unit 61 of the test platform 60 observes the input (e.g., "XX2") of the external software called by the processing task block and the output (e.g., "ZZ2") corresponding to the input (see FIG. 4). At this time, it is assumed that the input of the processing task block 44 is "XX1" and the output is "ZZ1". The modeling unit 62 learns a model using the input and output of the processing task block 44 and the input and output of the external software observed by the observation unit 61. Then, the service profile creation unit 63 creates a service profile that reflects the input and output of the external software based on the learned model.

[0081] With this configuration, even if the logic of the external software is not open, it is possible to create a service profile that reflects the behavior of the external software through external observation.

[0082] [Service profile creation process (fourth example)] Next, a service profile creation process (fourth example) in the test platform 60 according to this embodiment will be described. The fourth example is a method of statically analyzing the logic of the called external software to create a service profile. This method is based on the premise that the called external software is a white box.

[0083] An observation unit 61 of a test platform 60 statically analyzes the logic of external software (SaaS) called by a processing task block. A modeling unit 62 uses the results of the analysis by the observation unit 61 to learn a model. Then, a service profile creation unit 63 creates a service profile that reflects the results of the analysis based on the learned model.

[0084] With this configuration, if the logic of the external software is open, it is possible to create a service profile that more accurately reflects the behavior of the external software.

[0085] [Service profile creation process (5th example)] Next, a service profile creation process (fifth example) in the test platform 60 according to this embodiment will be described. The fifth example is an example of a method for creating a service profile when the logic of the called external software can be statically analyzed and includes a series of interactions between a processing task block and the external software.

[0086] Here, the processing task block that calls the external software is configured using the generation AI. If the behavior of the external software includes a series of interactions with the generation AI involving multiple responses, the service profile creation unit 63 creates a service profile based on the log information of the series of interactions.

[0087] This configuration makes it possible to create a service profile that more accurately reflects the behavior of external software from log information of a series of interactions between the processing task block consisting of the generation AI and the external software.

[0088] [Service profile creation process (6th example)] Next, a service profile creation process (sixth example) in the test platform 60 according to this embodiment will be described. The sixth example is another example of a method for creating a service profile when the logic of the called external software can be statically analyzed and includes a series of interactions between a processing task block and the external software.

[0089] Here, the processing task block that calls the external software is configured using the generated AI. When the behavior of the external software includes a series of dialogues with the generated AI involving multiple responses, the service profile creation unit 63 creates a service profile based on a prompt flow that defines the prompt logic for the external software to respond to the generated AI.

[0090] With this configuration, a service profile can be created that more accurately reflects the behavior of external software based on a prompt flow that defines prompt logic for a series of interactions between a processing task block consisting of a generation AI and external software.

[0091] [Test pattern creation process] Next, the test pattern generation process in the test board 60 according to this embodiment will be described with reference to FIG. FIG. 9 is a flowchart showing an example of a test pattern generation process in the test board 60. First, the test pattern generation unit 65 (FIG. 5) of the test board 60 refers to the service profile (variation table) and extracts true pattern information (step S31). The true pattern information is information on patterns whose correctness is "T" in the service profile. In the example of FIG. 8, patterns 1, 4 to 5 in the service profile 64 correspond to the true pattern information.

[0092] Next, the test pattern generating unit 65 obtains an expected output corresponding to the input from the extracted pattern information (step S32).

[0093] Then, the test pattern creation unit 65 associates the input with the expected output and registers them as test patterns, thereby creating a list of test patterns 66 as shown in Fig. 4 (step S33). After step S33 is completed, the process of this flowchart ends.

[0094] [Test execution process] Next, the E2E test execution process in the test board 60 according to this embodiment will be described with reference to FIG. 10 is a flowchart showing an example of an E2E test execution process in the test board 60. Numbers in parentheses in the flowchart correspond to the numbers in parentheses in FIG.

[0095] First, the test execution unit 67 (FIG. 5) inputs the input contents of the test pattern to the start block 12 of the workflow system 41 on the service collaboration platform 40 (step S41). Next, the test execution unit 67 compares the expected output registered in the test pattern with the actual output for the input of the workflow system 41 (step S42).

[0096] Next, the test execution unit 67 determines whether the actual output for the input of the workflow system 41 is included in the service profile (step S43). If the actual output is included in the service profile (YES in step S43), the test execution unit 67 determines whether the pattern corresponding to the actual output is correct (step S44).

[0097] If the actual output for the input of the workflow system 41 is correct (determination T in step S44), the test execution unit 67 determines that the "test is successful," that is, that the behavior of the workflow system 41 is normal.

[0098] If the actual output for the input of the workflow system 41 is incorrect (judgment F in step S44), the test execution unit 67 "retry," that is, sends the input contents of the test pattern to the workflow system 41 again and performs the test again.

[0099] In addition, if it is determined in the determination process of step S43 that the actual output is not within the service profile (NO determination in step S43), the test execution unit 67 determines that the test has failed and displays the determination result on the display device 74.

[0100] Here, a supplementary explanation will be given of the process of comparing the expected output registered in the test pattern with the actual output in response to the input of the workflow system 41 in step S42. In step S42, the test execution unit 67 converts an expected output for an input of a test pattern registered in the test pattern into a feature vector. The test execution unit 67 also converts an actual output for an input of the workflow system 41 while the processing task block is being executed into a feature vector. The feature vector expresses information on the output for the input in the form of a vector with variable values ​​as elements.

[0101] Next, the test execution unit 67 compares the feature vector of the actual output for the input of the workflow system 41 with the feature vector of the expected output for the input of the workflow system 41 registered in the test pattern, and calculates the similarity between the two feature vectors. Next, the test execution unit 67 determines whether the behavior of the target processing task block is normal based on the similarity between the two feature vectors. Then, the test execution unit 67 verifies the workflow system based on the result of determining the behavior of the processing task block.

[0102] In this way, by converting the output information for the input into vectors and comparing them, it is possible to calculate the similarity between the expected output registered in the test pattern and the actual output for the input of the workflow system 41. Therefore, it is possible to verify the entire workflow system based on the test pattern.

[0103] [Test result screen for any service profile] Here, a test result screen showing the test results of an arbitrary service profile executed by the test platform 60 according to this embodiment will be described with reference to FIG. FIG. 11 is a diagram showing an example of a test results screen showing the results of a test run by the test platform 60 for a given service profile.

[0104] 11, the test result screen 90 has an area for displaying the service profile 64, an area for displaying the test result 91, a retry button 92, and an end button 93. The service profile 64 displayed on the test result screen 90 is designed to indicate which service (external software) it corresponds to. The example in FIG. 11 is a service profile corresponding to the processing task block 44 when external software 52 (SaaS2a) is called.

[0105] The test result 91 has items of input, output, and pass / fail for each piece of pattern identification information. The "input" and "output" are the same as the input and output items of the test pattern 66 in FIG. "Pass / Fail" stores information indicating the test result. If the test result is good, that is, the actual output is correct, "OK" is stored, and if the test result is bad, that is, the actual output is incorrect (unacceptable variation), "NG" is stored.

[0106] For example, if "NG" is displayed in the pass / fail field of the test result 91, the user can select the corresponding pattern and press the retry button 92, causing the test platform 60 to retest the workflow system 41. When the user presses the end button 93, the contents of the test result 91 are finalized, and the test result 91 is saved in the RAM 73 or the non-volatile storage 76.

[0107] As described above, the test platform 60 judges the behavior of all processing task blocks (processing task block 44 in Figure 4) involved in a certain calling pattern of external software by the workflow system 41 (for example, a call to external software 52), and verifies the workflow system 41 for the certain calling pattern based on the results of the judgment.

[0108] With this configuration, it is possible to verify the entire workflow system by determining the behavior of all process task blocks involved in calling external software in one call pattern of the external software. By determining the behavior of all process task blocks involved in calling external software in one call pattern, it is possible to increase the accuracy of verification of one call pattern.

[0109] [Test results screen for the entire workflow system] Next, a test result screen showing the test results of each service profile and the entire workflow executed by the test platform 60 according to this embodiment will be described with reference to FIG. FIG. 12 is a diagram showing an example of a test result screen showing the test results of each service profile and the entire workflow executed by the test platform 60.

[0110] As shown in FIG. 12, the test result screen 100 has an area for displaying the test results 101 of each service profile, an area for displaying the test results 102 of the entire workflow system, and a back button 103.

[0111] The test result screen 100 shows the results of the judgment for each of the multiple external software (service) patterns called. The test result screen 100 has fields for the flow and pass / fail for each called service. "Flow" is the same as the flow item of the service profile 64 in FIG. "Pass or Fail" is the same as the pass or fail item in the test result 91 in FIG.

[0112] The test results 101 for each service profile shown in Fig. 12 display the test results for "SaaS2a" and "SaaS1a" when one service is called, and "SaaS2a, SaaS1a" when two services are called. That is, these test results 101 show the test results for three call patterns of external software. For all call patterns, the test result is "OK." Since the test result for all three call patterns is "OK," the test result for the entire workflow system 102 also displays the test result for the entire workflow system as "OK."

[0113] When the user presses the back button 103, the screen returns from the test result screen 100 shown in FIG. 12 to the test result screen 90 for each service profile shown in FIG.

[0114] As described above, the test platform 60 judges the behavior of all processing task blocks involved in calling external software for each of multiple call patterns (e.g., three call patterns in Figure 12) of external software by the workflow system 41, and verifies the workflow system 41 based on the results of the judgment for the multiple call patterns.

[0115] With this configuration, it is possible to verify the entire workflow system by determining the behavior of all process task blocks involved in calling external software for each call pattern for multiple call patterns of external software. By determining the behavior of process task blocks across multiple call patterns of external software, it is possible to increase the accuracy of verification of the entire workflow system.

[0116] As described above, the system verification method of this embodiment is a method for verifying a workflow system (e.g., workflow system 41) in which a computer (computer 70) having an arithmetic unit (e.g., CPU 71) that executes a program and a storage device (e.g., ROM 72, non-volatile storage 76) that stores the program calls and sequentially executes one or more external software programs (e.g., external software programs 51, 52) that provide services. The computer observes the inputs and outputs of processing task blocks (for example, processing task block 44) that constitute the workflow system and call external software that has fluctuations in output relative to the inputs. Next, based on the observed inputs and outputs of the processing task block, the computer creates a test pattern (for example, test pattern 66) that associates expected outputs with inputs of the workflow system during execution of the processing task block. Next, when executing a test using the test pattern, the computer calculates the similarity between the actual output for the input of the workflow system while executing the processing task block and the expected output for the input of the workflow system registered in the test pattern. The computer then determines whether the behavior of the process task block is normal or not based on the similarity between the two outputs, and verifies the operation of the workflow system based on the result of determining the behavior of the process task block.

[0117] The system verification method using the test platform 60 (computer 70) according to this embodiment configured as described above can achieve more stable verification of the entire workflow system that realizes collaboration with externally provided components (services provided by external software) whose behavior is uncertain and difficult to predict. For example, in the development of an application system incorporating LLM services, it is possible to realize stable and flexible system testing and achieve system quality assurance that takes into account the characteristics of LLM. This embodiment also has the following advantages: According to this embodiment, it is possible to profile fluctuations in output behavior for each service component (processing task block and called external software), as shown in FIG. Furthermore, the amount of profiling can be reduced by limiting the scope of service profiling creation to patterns of calling external software (SaaS) from the workflow system.

[0118] <Second embodiment> A method for verifying the workflow system 41 in the second embodiment of the present invention will be described with reference to FIG. FIG. 13 is a diagram showing an outline of a method for verifying a workflow system 41 according to the second embodiment of the present invention.

[0119] [Service profile creation process (first example)] When the behavior of the external software includes referencing an external knowledge database through Retrieval-Augmented Generation (RAG), the test infrastructure 60 identifies the meta information or reference data of the referenced external knowledge database to create a service profile. This method is based on the premise that the logic of the called external software is a white box.

[0120] With this configuration, it is possible to create a service profile that more accurately reflects the behavior of the external software by using the meta information or reference data of the referenced external knowledge database.

[0121] [Service profile creation process (second example)] If the behavior of the external software includes the invocation of a plug-in function or a further external service, the test infrastructure 60 identifies the invoked plug-in function or the new external service and creates a service profile.

[0122] With this configuration, it is possible to create a service profile that more accurately reflects the behavior of external software by using information on the called plug-in function and the new external service.

[0123] As described above, the present invention is not limited to the above-described embodiments, and various other modifications and applications are possible without departing from the spirit of the invention as defined in the claims. For example, the above-described embodiments have been described in detail and specifically to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the components described. Furthermore, it is possible to replace part of the configuration of one embodiment with a component of another embodiment. It is also possible to add a component of another embodiment to the configuration of one embodiment. It is also possible to add, replace, or delete other components from part of the configuration of each embodiment.

[0124] Furthermore, the above-described configurations, functions, processing units, etc. may be partially or entirely realized in hardware, for example, by designing them as integrated circuits, etc. As the hardware, a broad processor device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used. [Explanation of symbols]

[0125] 40...Service collaboration platform (workflow system), 41...Workflow system, 42...Start end, 43...Processing task block (processing (1)), 44...Processing task block (processing (2)), 45...End end, 51...External software (external SaaS1a), 52...External software (external SaaS2a), 53...API definition, 54...API definition, 60...Test platform, 61...Observation unit, 62...Modeling unit, 63...Service profile creation unit, 64...Service profile, 65...Test pattern creation unit, 66...Test pattern, 67...Test execution unit, 70...Calculator, 80...Service profile editing screen, 90...Test result screen, 91...Test result, 100...Test result screen, 101...Test results of each service profile, 102...Test results of the entire workflow system

Claims

1. 1. A system verification method for verifying a workflow system in which a computer having an arithmetic unit that executes a program and a storage device that stores the program calls and sequentially executes one or more external software programs that provide services, comprising: The computer The workflow system is configured to observe the input and the output of a processing task block that calls external software whose output varies with respect to the input; creating a test pattern that associates expected outputs with inputs of the workflow system during execution of the processing task block based on the observed inputs and outputs of the processing task block; When a test using the test pattern is executed, a similarity is calculated between an actual output of the workflow system in response to an input during execution of the processing task block and an expected output of the workflow system in response to the input registered in the test pattern; determining whether the behavior of the processing task block is normal or not based on the similarity of the outputs; The workflow system is verified based on the result of determining the behavior of the processing task block. System validation methods.

2. The computer analyzing the tendency of the output for the same input of the processing task block, and creating a service profile that associates the input and output of the processing task block when the external software is called based on a model that has learned the normal behavior of the external software; Based on the input and output of the service profile, the test pattern is created in which the expected output is associated with the input of the workflow system during execution of the processing task block by calling the external software. The system verification method of claim 1 .

3. The computer In one call pattern of the external software by the workflow system, the behavior of all processing task blocks involved in the call of the external software is determined, and the workflow system in the one call pattern is verified based on the result of the determination. The system verification method of claim 2 .

4. The computer In a plurality of calling patterns of the external software by the workflow system, the behavior of all processing task blocks involved in the calling of the external software is determined for each of the calling patterns, and the workflow system is verified based on the results of the determination for the plurality of calling patterns.

4. The system verification method of claim 3.

5. The computer The model is trained based on the observed inputs and outputs of the processing task blocks and the results of static analysis of the logic of the workflow system, and the service profile is created based on the model.

3. The system verification method of claim 2.

6. The computer The input and output of the external software called by the processing task block are observed, and the service profile that reflects the input and output of the external software is created.

3. The system verification method of claim 2.

7. The computer Statically analyze the logic of the external software called by the processing task block, and create the service profile that reflects the results of the analysis.

3. The system verification method of claim 2.

8. The processing task block is configured using generated AI; The computer If the behavior of the external software includes a series of interactions with the generated AI involving multiple responses, the service profile is created based on log information of the series of interactions.

8. The system verification method of claim 7.

9. The processing task block is configured using generated AI; The computer If the behavior of the external software includes a series of interactions with the generated AI involving multiple responses, the service profile is created based on a prompt flow that defines prompt logic for the external software to respond to the generated AI.

8. The system verification method of claim 7.

10. The computer If the behavior of the external software includes referencing an external knowledge database by generating a search extension, the service profile is created by identifying meta information or reference data of the referred external knowledge database.

8. The system verification method of claim 7.

11. The computer If the behavior of the external software includes a call to a plug-in function or an external service, the called plug-in function or external service is identified and the service profile is created.

8. The system verification method of claim 7.

12. The computer When the test using the test pattern is executed, a similarity is calculated between a feature vector of an actual output for an input to the workflow system during execution of the processing task block and a feature vector of an expected output for an input to the workflow system registered in the test pattern; determining whether the behavior of the processing task block is normal or not based on the similarity between the two feature vectors; The workflow system is verified based on the result of determining the behavior of the processing task block. The system verification method of claim 2 .

13. The external software services are provided using a large-scale language model. The system verification method of claim 1 .

Images