Scenario testing system, scenario testing method, and program

Abstract

To reduce the time required to find failures in scenario creation of a chatbot while reducing overlooking of such failures.SOLUTION: A scenario inspection device 230 for inspecting a scenario of a chatbot is provided. This scenario inspection device includes: a scenario input unit 241 that receives input of the scenario constituted of elements; a policy input unit 242 that receives input of a policy including conditions to be satisfied in the scenario by the elements constituting the scenario; a structural analysis unit 243 that performs structural analysis of the elements included in the received scenario; an inspection unit 244 that inspects whether or not the elements subjected to the structural analysis conform to the conditions of the received policy; and an output unit 245 that outputs the inspection results. By a scenario developer describing a desired state of the scenario as a policy, it is possible to inspect whether or not the scenario conforms to the policy, and reduce the time required to find failures in scenario creation or the like even for the scenario developer who does not have special knowledge.SELECTED DRAWING: Figure 3

Description

Technical Field

[0005]

[0001] The present invention relates to a scenario inspection system, a scenario inspection method, and a program, and particularly to a scenario inspection system, a scenario inspection method, and a program for inspecting a scenario of a chatbot.

Background Art

[0002] In recent years, chatbots that automatically conduct conversations with users through text or voice have been used in existing operations that conduct conversations according to predetermined procedures, such as call centers operated by companies, implementation of questionnaires, scheduling of luggage delivery, reservation of facilities, and issuance of resident cards. Such chatbots that conduct conversations according to predetermined procedures proceed with conversations along a predetermined dialogue template called a scenario. For example, this is described in Non-Patent Document 1.

[0003] In the development of such so-called scenario-based chatbots, scenarios are often created by chatbot developers using dedicated GUI (Graphical User Interface) tools. For example, this is described in Non-Patent Document 2.

[0004] In addition, for the purpose of managing information assets (automated operations), research has been conducted to reduce management man-hours by making it possible to easily define management guidelines (policies) for reducing the labor of information operation management. For example, this is described in Non-Patent Document 3.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

Non-Patent Document 2

[0006] When scenario creators create scenarios, errors can occur, such as incorrectly specifying the flow of the dialogue. For example, incorrectly describing transition conditions can lead to unintended transitions and circular dialogue, or a step that uses a slot may be executed before the step that records a value in the slot, resulting in a reference error. In the chatbot development process, to prevent scenario creation errors that affect the chatbot's performance, it is necessary to simulate dialogues as the scenario is created and check whether the scenario works as intended. However, sometimes business personnel who do not have the expertise of a chatbot developer create / modify scenarios, and in such cases, it takes time to find where the problem lies in the scenario, and the simulated dialogue may be insufficient, making it difficult to detect creation errors. Therefore, the present invention provides a scenario inspection system, a scenario inspection method, and a program that reduce the time required to detect and overlook flaws in scenario creation, even if performed by someone without specialized knowledge. [Means for solving the problem]

[0007] To solve the above problems, a scenario inspection system for inspecting chatbot scenarios comprises: a scenario input unit that accepts input of a scenario composed of elements; a policy input unit that accepts input of a policy that includes conditions that the elements constituting the scenario must satisfy in the scenario; a structure analysis unit that performs structure analysis of the elements included in the scenario accepted by the scenario input unit; an inspection unit that checks whether the elements structurally analyzed by the structure analysis unit conform to the conditions of the policy accepted by the policy input unit; and an output unit that outputs the results of the inspection unit. The policy accepted by the policy input section includes the elements that constitute the scenario, the scope to which those elements are subject to the policy, and the conditions that the elements within that scope must satisfy. A scenario testing system is provided. [Effects of the Invention]

[0008] According to the present invention, even if performed by someone without specialized knowledge, a scenario inspection system can be provided that reduces the time required to detect and overlook flaws in scenario creation. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows a computer system for carrying out an embodiment of the present invention. [Figure 2] Figure 2 shows an example of the configuration of a scenario testing system according to an embodiment of the present invention. [Figure 3] Figure 3 shows an example of a functional block of a scenario testing device according to an embodiment of the present invention. [Figure 4] Figure 4 shows an example of the operation of a chat session according to an embodiment of the present invention. [Figure 5] Figure 5 shows an example of a scenario flow defining the operation of a chat session according to an embodiment of the present invention. [Figure 6] Figure 6 shows an example of a scenario definition structure that defines the operation of a chat session according to an embodiment of the present invention. [Figure 7] Figure 7 shows an example of conditions included in the policy according to an embodiment of the present invention. [Figure 8] Figure 8 shows an example of the operation of a scenario testing device according to an embodiment of the present invention. [Figure 9] Figure 9 is a flowchart showing the processing flow for extracting target element structures in the structural analysis section of the scenario inspection apparatus according to an embodiment of the present invention. [Figure 10] Figure 10 is a flowchart showing the processing flow for creating a pseudo-route in the structural analysis section of a scenario inspection device according to an embodiment of the present invention. [Modes for carrying out the invention]

[0010] The embodiments of the present invention will be described below with reference to the drawings. First, with reference to Figure 1, a computer system 300 for carrying out embodiments of the present disclosure will be described. Mechanisms and apparatus of various embodiments disclosed herein may be applied to any suitable computing system. The main components of the computer system 300 include one or more processors 302, memory 304, terminal interface 312, storage interface 314, I / O (input / output) device interface 316, and network interface 318. These components may be interconnected via a memory bus 306, an I / O bus 308, a bus interface unit 309, and an I / O bus interface unit 310.

[0011] The computer system 300 may include one or more general-purpose programmable central processing units (CPUs) 302A and 302B collectively referred to as a processor 302. In certain embodiments, the computer system 300 may comprise multiple processors, and in other embodiments, the computer system 300 may be a single CPU system. Each processor 302 executes instructions stored in the memory 304 and may include an on-board cache.

[0012] In certain embodiments, the memory 304 may include a random-access semiconductor memory, a storage device, or a storage medium (either volatile or non-volatile) for storing data and programs. The memory 304 may store all or part of the programs, software modules, and data structures for implementing the functions described herein. For example, the memory 304 may store the chatbot scenario inspection application 350. In certain embodiments, the chatbot scenario inspection application 350 may include instructions or descriptions for executing the functions described below on the processor 302.

[0013] In certain embodiments, the chatbot scenario inspection application 350 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. In certain embodiments, the chatbot scenario inspection application 350 may include data other than instructions or descriptions. In certain embodiments, a camera, sensor, or other data input device (not shown) may be provided to communicate directly with the bus interface unit 309, the processor 302, or other hardware of the computer system 300.

[0014] Computer system 300 may include a bus interface unit 309 that facilitates communication between a processor 302, a memory 304, a display system 324, and an I / O bus interface unit 310. The I / O bus interface unit 310 may be coupled to an I / O bus 308 for transferring data to and from various I / O units. The I / O bus interface unit 310 may communicate via the I / O bus 308 with a plurality of I / O interface units 312, 314, 316, and 318, also known as I / O processors (IOPs) or I / O adapters (IOAs).

[0015] The display system 324 may include a display controller, a display memory, or both. The display controller may be capable of providing video, audio, or both data to a display device 326. Additionally, computer system 300 may include devices such as one or more sensors configured to collect data and provide the data to processor 302. The display system 324 may be connected to a display device 326 such as a single display screen, a television, a tablet, or a portable device.

[0016] The I / O interface unit is equipped with the function of communicating with various storage or I / O devices. For example, the terminal interface unit 312 can be attached with a user I / O device 320 such as a user output device like a video display device, a speaker television, or a user input device like a keyboard, a mouse, a keypad, a touch pad, a trackball, a button, a light pen, or other pointing devices. The user can use the user interface to input input data and instructions to the user I / O device 320 and the computer system 300 by operating the user input device, and may receive output data from the computer system 300. The user interface may be, for example, displayed on the display device, played by the speaker, or printed via the printer through the user I / O device 320.

[0017] The storage interface 314 can accommodate one or more disk drives or direct-access storage devices 322 (usually magnetic disk drive storage devices, but may be arrays of disk drives configured to appear as a single disk drive or other storage devices). In some embodiments, the storage device 322 may be implemented as any secondary storage device. The contents of memory 304 may be stored in the storage device 322 and read from the storage device 322 as needed. The I / O device interface 316 may provide an interface to other I / O devices such as printers and fax machines. The network interface 318 may provide a communication path so that the computer system 300 and other devices can communicate with each other. This communication path may be, for example, a network 330.

[0018] In one embodiment, the computer system 300 may be a device that receives requests from other computer systems (clients) that do not have a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. In another embodiment, the computer system 300 may be a desktop computer, a portable computer, a laptop computer, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

[0019] Next, with reference to Figure 2, the chatbot scenario testing system 200 according to the present invention will be described. The scenario testing system 200 according to the present invention mainly includes a mobile terminal 210, a communication network 215, an external system 220, and a scenario testing device 230. The mobile terminal 210, the external system 220, and the scenario testing device 230 are connected via the communication network 215.

[0020] The communication network 215 may include, for example, a local area network (LAN), a wide area network (WAN), a satellite network, a cable network, a Wi-Fi network, or any combination thereof. Furthermore, the connection between the external system 220 and the scenario inspection device 230 may be wired or wireless.

[0021] The scenario testing device 230 is a computing device used by business personnel without specialized knowledge to perform means for testing chatbot scenarios according to the present invention. The scenario testing device 230 may be, for example, a desktop computer, server computer, laptop computer, tablet computer, workstation, mobile terminal, or other type of computing device, and is not particularly limited in the present invention.

[0022] The scenario testing device 230 includes a processor 231 for executing instructions stored in memory 240, an I / O interface 232 for controlling communication between internal and external devices of the scenario testing device 230, a network interface 233 for controlling communication via a communication network 215, a user I / O interface 234 for receiving user input, a memory 240 for storing functional units for executing each function of the scenario testing means according to the embodiment of the present invention, and a bus 236 for controlling bidirectional communication between these components.

[0023] The memory 240 includes, for example, a scenario input unit 241 that accepts scenario input, a policy input unit 242 that accepts policy input, a structure analysis unit 243 that performs structural analysis of the scenario, an inspection unit 244 that checks whether the scenario conforms to the conditions of the policy, and an output unit 245 that outputs the results of the inspection.

[0024] Each functional unit included in the scenario testing device 230 may be a software module constituting the chatbot scenario testing application 350 in the computer system 300 shown in Figure 1, or it may be an independent, dedicated hardware device. Furthermore, these functional units may be implemented in the same computing environment or in a distributed computing environment.

[0025] The external system 220 may be a system for developing a chatbot system. This external system 220 may be a system consisting of, for example, a remote PC, server, storage device, database, cloud storage, or a combination thereof. In Figure 2, for the sake of explanation, one external system 220 is shown, but in reality, the scenario testing device 230 is connected to numerous external systems and accesses the necessary external systems depending on the content and purpose of the chat scenario.

[0026] The external system 220 is a system used by developers with expertise in developing chatbot systems, and may be a system that stores software modules for the chatbot system under development and corresponding test scenarios. The external system 220 may store different chatbot systems and data depending on the purpose of use. For example, in the case of an external system for developing a chatbot that supports restaurant reservations, it may store software modules that support the scenarios necessary for restaurant reservations, those scenarios, and the test data required for those scenarios (for example, a list of restaurants in a given area, information such as each restaurant's specialty dishes, price range, business hours, and current reservation status). In this case, the scenario testing device 230 can access the storage of the external system 220 via the network interface 233 and use it in the same way as its own storage unit 235.

[0027] Furthermore, the external system 220 may include a device with a configuration similar to that of the scenario inspection device 230. The scenario inspection device 230 used by business personnel may be configured to include only a scenario input unit 241, a policy input unit 242, and an output unit 245, and to use the structural analysis unit 243 and inspection unit 244 located in the device within the external system 220. In this way, the scenario inspection system 200 can employ a flexible hardware configuration.

[0028] The mobile terminal 210 is a terminal used by an individual user and may be, for example, a personal computer, a smartphone, or a tablet computer. The user may use a designated communication application (SNS, email, messaging function, etc.) or browser on the mobile terminal 210 to interact (exchange messages, etc.) with a chatbot system that executes a scenario tested by the scenario testing device 230 via the communication network 215, and perform a final confirmation of the scenario using the actual communication application.

[0029] Referring to Figure 3, the logical configuration of each functional unit included in the scenario inspection device 230 will be explained. As explained in Figure 2, the functional units included in the scenario inspection device 230 are the scenario input unit 241, the policy input unit 242, the structural analysis unit 243, the inspection unit 244, and the output unit 245. The scenario input unit 241 accepts the input of a scenario consisting of elements. An "element" consists of processes (steps) in the scenario, transitions between steps, and slots, and is subject to policy.

[0030] A chatbot scenario defines the content and flow of the conversation with the user, such as "what questions to ask the user" and "how to ask the next question based on the user's response." A scenario consists of steps, transitions between them, and slots. (1) Step The chatbot defines the messages (utterances) it speaks to the user, the choices it makes, and the steps it takes to process the user's responses. (2) Transitions between steps The flow of the dialogue is constructed by connecting steps using conditional transitions. (3) Slot Define a dedicated area for saving result data such as user utterances and selections in each step. The defined slot can then be used in other steps or transition conditions.

[0031] The scenario input unit 241 may accept, for example, a file containing a scenario definition structure 400 as shown in Figure 6 (described later), a file exported from a GUI tool for creating scenarios, or a file written in a simpler description language as input. There are no particular limitations on the format in which the scenario is accepted.

[0032] The policy input unit 242 accepts policy input that includes conditions that the elements constituting the scenario must satisfy within the scenario. A policy is a set of conditions that specify the desired state or behavior of a scenario. As shown in Definition 1, a policy is defined as a combination of a "target" that specifies which part of the scenario the policy applies to, and a "condition" that specifies the desired state of the target. The "target" consists of an "element" and a "scope". The "element" represents the constituent elements of the scenario that the policy applies to, and the "scope" indicates whether the policy applies to a specific element or all elements. The "condition" describes the state under which the "target" conforms to the policy, and describes one of the following conditions: the number of executions, the number of reads, the number of writes, or a sequential condition. [Table 1]

[0033] The policy input section 242 may accept input from a GUI tool for creating scenarios, or it may accept a file written in a simpler description language; there are no particular limitations on the format in which policies are accepted. Policies that are effective in general scenarios are provided as templates that make it easy to specify commonly used conditions and which targets those conditions apply to, as shown in Figure 7, and the scenario creator may modify them as needed. For example, a policy may specify, as a sequential condition, the element "slot" with the condition "there are no slots that have been written to but not read", the element "step" with the condition "a step is executed only once", and with the scope set to "all", the element "transitions between steps" with the condition "all transitions are executed".

[0034] The structural analysis unit 243 performs structural analysis of the elements included in the received scenario. The structural analysis performed by the structural analysis unit 243 may be syntactic analysis, a known method. For example, it may extract a sequence of tokens, i.e., elements, from the string of the scenario received by the scenario input unit 241, and perform structural analysis of the scenario by creating a data structure such as a syntax tree from that sequence of tokens. The scenario analyzed by the structural analysis unit 243 consists of elements that have been analyzed into slots, which are storage areas that store information obtained at each step and / or the results obtained by processing that information; steps that define the utterance / response processing in the scenario; and step transitions that connect these steps and define the flow of the dialogue. In particular, the structural analysis unit 243 creates all dialogue routes that cover the flow of the dialogue based on the step transitions.

[0035] The inspection unit 244 checks whether the elements of the scenario structurally analyzed by the structural analysis unit 243 conform to the conditions of the policy received by the policy input unit 242. The inspection unit 244 checks whether the analyzed slots, steps, and inter-step transitions refer to the conditions specified in the policy and satisfy each of them. For example, if the policy specifies the condition "there is always a write" for the number of writes to a slot, the inspection unit 244 checks whether the slot "always has a write". Also, for example, if the policy specifies the condition "the step is always executed once" for the number of executions of a step, the inspection unit 244 checks whether the step "is always executed once".

[0036] The output unit 245 outputs the results of the inspection performed by the inspection unit 244. The output from the output unit 245 is the output from the processor 231 to the outside, which is not particularly limited to the results of the inspection function of the inspection unit 244 performed by the processor 231. For example, the output unit 245 may store the results in the memory 240, display the results on a display via the user I / O interface 234, or transmit the results to another computer via the network interface 233.

[0037] The scenario inspection device 230 described above allows scenario developers to describe the desired state of a scenario as a policy, enabling them to check whether a scenario conforms to that policy. Even if the scenario developer lacks specialized knowledge, the device can mechanically check for flaws in scenario creation, reducing the time it takes to detect and potentially overlooking such flaws.

[0038] Referring to Figure 4, a simple example of the operation of a chatbot system and a chat session conducted by its user is illustrated. The chatbot provided by this system asks the user, "How old are you?", accepts the user's response, and responds with "We will start the application" if the user is 20 years of age or older, and with "You cannot apply" if the user is 19 years of age or younger. This chatbot is a so-called scenario-based chatbot that proceeds with dialogue according to a set procedure determined by the user's age. In the actual screen, a chatbot that operates according to such a scenario will ask "How old are you?", wait for the user's response, and if the user's response is "25 years old", it will respond with "We will start the application", and if it is "15 years old", it will respond with "You cannot apply". In this way, the chatbot interacts with the user according to the scenario.

[0039] Referring to Figure 5, an example of a scenario flow defining the operation of a chatbot system and the chat session conducted by its user is explained in more detail as a flowchart. In the figure, S represents a step. In step S100, the chatbot provided by this chatbot system has the step name [Name] and says "Please enter your name," waiting for a response from the user, which corresponds to a prompt described later. Once a response from the user is received, the process proceeds to S102. In step S102, the step name [Date of Birth] and says "Please enter your date of birth," waiting for a response from the user, which corresponds to a prompt described later. Once a response from the user is received, the process proceeds to S104.

[0040] Step S104, named [Age], calculates the user's age based on the date of birth received in step S102 [Date of Birth], and corresponds to an activity described later. Once the calculation is complete, the user proceeds to either step S106 [Application Complete] or step S108 [Application Not Accepted]. A transition condition is defined between step S104 and steps S106 / S108. If the calculation result in S104 is 20 years of age or older, the user proceeds to S106; if it is under 20 years of age, the user proceeds to S108. If the user proceeds to S106, step [Application Complete] is executed, the utterance "Application Complete" is made, and the process ends. If the user proceeds to S108, step [Application Not Accepted] is executed, the utterance "Minors cannot apply" is made, and the process ends. Thus, the scenario has various types of steps and functions to hold and process user input data. These functions are defined by the definition structure described later.

[0041] Referring to Figure 6, a definition structure that defines the operation of a chat session according to an embodiment of the present invention will be described. Figure 6 is a diagram showing an example of a scenario definition structure 400 that defines the operation of a chat session according to an embodiment of the present invention. The scenario definition structure 400 according to an embodiment of the present invention is a data structure that defines the transitions (i.e., the flow) of chat in a chat session. In the present invention, definition structures corresponding to multiple different use cases (facility reservations, restaurant reservations, airport check-in, personnel recruitment, etc.) are prepared in advance and stored in a storage unit (for example, the storage unit 235 shown in Figure 2).

[0042] The scenario definition structure 400 consists of a module section 420, a slot section 440, and a step section 460. The module section 420 functions as a function that specifies which program to use within the scenario. For example, the module section 420 may specify a program that defines the dialogue, utterances, prompts, activities, dialogue branching points, etc., to be used in the scenario.

[0043] Slot section 440 defines the storage location for saving information acquired during a chat session. The information acquired during a chat session is, for example, information entered by the user interacting with the chatbot, and differs depending on the use case of the chat session. This information may be configured as the dialogue state described above. For example, in a recruitment chat scenario, this information may include the user's name, desired job, and work history, while in the case of a restaurant reservation, this information may include the user's desired cuisine, time slot, area, etc. Generally, the storage location defined in slot section 440 is a memory area in network storage, but it is also possible to configure the system to save the information acquired during the chat session to a local storage device.

[0044] Step section 460 defines the processes (steps) in a chat scenario and defines each transition within the chat scenario. The steps defined in step section 460 include types such as prompts, activities, scripts, and sub-scenarios. A prompt is a process that issues an utterance to the user and waits for a response from the user (for example, "What is your name?"). An activity is a process that issues an utterance to the user and continues to the next process without waiting for a response (for example, "Understood. I will search for restaurants."). A script is a process that calls a predetermined function (such as accessing an external system or searching the internet). A sub-scenario is a new subordinate scenario called within a main scenario that is already running (for example, in a main scenario for restaurant reservations, the process of deciding the contents of a course meal).

[0045] Although not shown in Figure 6, the scenario definition structure 400 also defines rules that define the behavior of the chat scenario. These rules may, for example, specify which slot to save information obtained from the user in, or which step (prompt, script, etc.) to transition to next depending on the dialogue state saved in the slot. In this way, the chat session between the user and the chatbot is defined by the module section 420, slot section 440, step section 460, and chat rules included in the scenario definition structure. The chatbot system reads and interprets the scenario defined in the scenario definition structure 400 and controls and executes the flow of the dialogue.

[0046] Furthermore, when the scenario input unit 241 receives input in the format of the scenario definition structure 400, the structural analysis unit 243 recognizes the existence of a slot called "age" based on the "slots" token, and recognizes the existence of two steps, "sci02-01" and "sci02-02," based on the "steps" token. In addition, the structural analysis unit 243 recognizes that the "sci02-01" step has a scenario transition to either "sci02-02" or "sci02-03" based on the "transitions" token, and that the "sci02-02" step has a scenario transition to "sci02-04" based on the "transitions" token.

[0047] Referring to Figures 8 to 10, the operation of the scenario testing device 230 will be explained. As shown in Figure 8, the scenario in this example consists of three steps. Step [1] is the step of making the utterance "Please select" and giving the user two options, "name" and "skip," and waiting for a response, and writing the response to the slot "selection." Step [2] is the step of making the utterance "Enter your name" and waiting for a response, and writing the response to the slot "name." Step [3] is the step of reading from the slot "name" and making the utterance "Welcome, 'name'." In this scenario, there are two slots, "selection" and "name," and in step [1] data is written to "selection," in step [2] data is written to "name," and in step [3] data is read from "name."

[0048] Furthermore, this scenario defines three step transitions: a step transition from step[1] to step[2], a step transition from step[1] to step[3], and a step transition from step[2] to step[3]. When the user responds with "name" to the options in step[1], "name" is stored in the "selection" slot, and the system transitions to step[2] if the response is "name". Then, when a response is received in step[2], the system transitions to step[3]. Also, when the user responds with "skip" to the options in step[1], "skip" is stored in the "selection" slot, and the system transitions directly to step[3] if the response is "skip".

[0049] Furthermore, the policy in this example consists of two conditions. Policy number 1 stipulates that for the element "slot" in the range "entire," the number of writes is 1 or less, meaning that no slot in the scenario is written to more than once. Policy number 2 stipulates that for the element "step" in the range "specific (step[2])," that is, only step[2], the number of executions is 1, meaning it is always executed once.

[0050] Furthermore, if the scenario inspection device 230 provides a development environment with a GUI to the scenario developer, the structural analysis unit 243 and inspection unit 244 may perform structural analysis in real time and inspection according to the policy conditions, or they may perform structural analysis and inspection when saving the scenario. It is preferable that the real-time structural analysis / inspection is a relatively light process performed on each element of the scenario each time the scenario is edited, while it is preferable that the structural analysis / inspection at the time of saving is a relatively heavy process, such as comprehensively creating and inspecting routes. Therefore, in this example, the policy for policy number 1 is a relatively light process, such as extracting that steps [1] and [2] write to slots "Selection" and "Name" respectively, so it is acceptable to perform structural analysis / inspection in real time. Conversely, policy number 2 is a heavy process because it requires the creation of routes, so it is preferable to perform structural analysis / inspection at the time of saving.

[0051] In real-time checking, the structural analysis unit 243 extracts the target element structure, and the inspection unit 244 checks whether it matches the conditions of the corresponding policy. The process of extracting the target element structure is shown in Figure 9. In S200, the structural analysis unit 243 receives a scenario from the scenario input unit 241. In S202, the structural analysis unit 243 initializes the slot usage list to be output last. In S204, the structural analysis unit 243 reads one step from the step array. In this example, it reads the first step [1].

[0052] Next, in S206, the structural analysis unit 243 analyzes the slots used by the loaded step. In this example, the structural analysis unit 243 recognizes that there is a slot "selection" used by the loaded step [1]. In S208, the structural analysis unit 243 stores the slot usage information for that step, i.e., writing to slot "selection", in the slot usage list located in the storage unit 235, etc. In S210, the structural analysis unit 243 determines whether there are any steps in the step sequence that have not yet been loaded.

[0053] The structural analysis unit 243 returns to S204 because there are still steps remaining and reads the second step [2]. Similarly, in S206, the structural analysis unit 243 analyzes the slot used by the read step [2] and recognizes that the slot "name" exists. In S208, the structural analysis unit 243 stores in the slot usage list that step [2] will write to the slot "name". In S210, the structural analysis unit 243 determines whether there are any steps in the step sequence that have not yet been read.

[0054] The structural analysis unit 243 returns to S204 because there are still steps remaining and reads the third step [3]. Similarly, in S206, the structural analysis unit 243 analyzes the slot used by the read step [3] and recognizes that this step refers to slot "Name". In S208, the structural analysis unit 243 stores the fact that it will read from slot "Name" in the slot usage list. In S210, the structural analysis unit 243 determines whether there are any steps in the step sequence that have not yet been read, and determines that there are no more steps so it terminates.

[0055] In S212, the structural analysis unit 243 outputs a slot usage list that includes information on how slots are used. In this example, the slot usage list includes information that step [1] writes to slot "Selection", step [2] writes to slot "Name", and step [3] reads from slot "Name". In S214, the inspection unit 244 inspects this slot usage information by referring to policy number 1, which has the condition that all slots are written to at least once. In this case, it is determined that this slot usage information conforms to policy number 1, so the inspection unit 244 does not determine that there are any elements that violate the policy.

[0056] During the save check, the structural analysis unit 243 creates a pseudo-route, and the inspection unit 244 checks whether it matches the conditions of the corresponding policy. The process of creating the pseudo-route is shown in Figure 10. In S300, the structural analysis unit 243 receives a scenario from the scenario input unit 241. In S302, the structural analysis unit 243 initializes the search for step-inter-step transitions. In S304, the structural analysis unit 243 initializes the routes that the scenario can take.

[0057] In S306, the structural analysis unit 243 searches for step transitions, which are components of the scenario that construct the flow of dialogue within the scenario. For example, in this example scenario, there is a step transition from step [1] to step [2], so the structural analysis unit 243 recognizes the existence of this step transition in S306. In S308, the structural analysis unit 243 stores the information for step [1] and step [2]. In S310, the structural analysis unit 243 analyzes the slots used by these steps. In this example, the structural analysis unit 243 recognizes that there are slots "selection" and "name" to be written to in step [1] and step [2], respectively.

[0058] Next, in S312, the structural analysis unit 243 determines whether the route has ended or not. For example, since there is an inter-step transition from step [2] to step [3], it does not determine that the route has ended. In that case, the structural analysis unit 243 returns to S306 to search for inter-step transitions and stores the information for step [2] and step [3] in S308. In S310, the structural analysis unit 243 analyzes the slots used by these steps. In this example, the structural analysis unit 243 recognizes that there is a slot "name" to write to in step [2] and a slot "name" to read from in step [3].

[0059] Next, in S312, the structural analysis unit 243 determines whether the route has ended or not. For example, since there are no step transitions from step [3], it determines that the route has ended. Then, in S314, the structural analysis unit 243 stores the route explored above (step [1] ⇒ step [2] ⇒ step [3]) in the route list located in the storage unit 235 or the like.

[0060] Next, in S316, the structural analysis unit 243 determines whether it has finished searching for all inter-step transitions. For example, in this example, there is an inter-step transition from step [1] to step [3], so it does not determine that all inter-step transitions have been searched for. Therefore, the structural analysis unit 243 returns to S304 to initialize the route in order to search for a new route, and in S306 searches for further inter-step transitions. As a result, in S306, the structural analysis unit 243 recognizes that there is an inter-step transition from step [1] to step [3], and in S308 stores the information for step [1] and step [3]. In S310, the structural analysis unit 243 analyzes the slots used by these steps and recognizes that there is a slot "selection" to write to in step [1] and a slot "name" to read in step [3].

[0061] Next, in S312, the structural analysis unit 243 determines whether the route has ended and, since there are no inter-step transitions from step [3], determines that the route has ended. Then, in S314, the structural analysis unit 243 stores the route explored above (step [1] ⇒ step [3]) in the route list. In S316, the structural analysis unit 243 determines whether all inter-step transitions have been explored and, in this example, since there are no remaining inter-step transitions, determines that all inter-step transitions have been explored. In this way, during the save check, the structural analysis unit 243 comprehensively creates candidate steps (routes) for the dialogue when saving the scenario and creates routes for all dialogues.

[0062] In S318, the structural analysis unit 243 outputs a route list that includes the created route. In this example, the route list contains two routes, namely route (step[1]⇒step[2]⇒step[3]) and route (step[1]⇒step[3]). In S320, the inspection unit 244 checks whether the information in this route list matches policy number 2, which has the condition that step[2] is executed only once, i.e., that there are no routes where step[2] is not executed. In this case, since route (step[1]⇒step[3]) does not execute step[2], the inspection unit 244 determines that there is an element that violates policy number 2.

[0063] According to the scenario inspection system 200 described above, by inspecting whether a scenario conforms to the policy, even if the inspection is performed by someone without specialized knowledge, the time it takes to detect flaws in scenario creation and the chances of overlooking them can be reduced.

[0064] Furthermore, the flowcharts shown in Figures 9 and 10 represent a scenario testing method for testing chatbot scenarios, which is a computer-based scenario testing method that accepts a scenario input consisting of elements, accepts a policy input containing conditions that the elements constituting the scenario must satisfy in the scenario, performs a structural analysis of the elements included in the accepted scenario, tests whether the structurally analyzed elements conform to the conditions of the accepted policy, and outputs the test results. This scenario testing method may be performed, for example, by a scenario testing device 230.

[0065] Furthermore, the scenario testing method may be provided as a program for causing a computer to function as a scenario testing method for testing chatbot scenarios, comprising: a scenario receiving means for receiving input of a scenario composed of elements; a policy receiving means for receiving input of a policy that includes conditions that the elements constituting the scenario must satisfy in the scenario; a structural analysis means for performing structural analysis of the elements included in the received scenario; a testing means for checking whether the structurally analyzed elements conform to the conditions of the received policy; and an output means for outputting the results of the testing.

[0066] These scenario testing methods and the programs that enable them to function on a computer, like the scenario testing device 230, can reduce the time it takes to detect and overlook flaws in scenario creation, even if performed by someone without specialized knowledge, by checking whether a scenario conforms to the policy.

[0067] <Rating> The effectiveness of the policy-based checking method using the scenario inspection device 230 described above was evaluated by comparing the time it took to detect defects using conventional methods with the time it took to write the policy.

[0068] In conventional checking methods, identifying which step, transition, or slot has a problem due to an incorrect definition often requires adding verification steps to the likely source and re-checking, which is time-consuming, especially for creators unfamiliar with scenario creation. Experience shows that finding and correcting each defect can take anywhere from several tens of minutes to several hours. Since several to dozens of defects can occur depending on the scenario's size, checking a single scenario can take several to tens of hours, and this burden can lead to overlooking defects due to insufficient checking.

[0069] On the other hand, describing the policies for the scenario testing device 230 and the scenario testing method can be completed in about 10 minutes per scenario, as it only requires adjusting a standard template. The actual scenario check time is about 10 seconds.

[0070] Thus, even with a small-scale creation scenario, assuming only two defects and a response time of 10 minutes per defect, the proposed method can be expected to reduce work time by approximately 50% compared to conventional methods.

[0071] <Summary> We proposed a method for mechanically checking for errors in scenario definitions by having scenario developers describe guidelines for the desired state of a scenario as policies. As a result, the proposed method was found to be effective in reducing the time it takes to find the causes of scenario defects and the number of defects that are overlooked, as well as reducing the time required to write policies.

[0072] It should be noted that the present invention is not limited to the exemplary embodiments, and can be implemented in configurations that do not depart from the content described in each claim. In other words, although the present invention is illustrated and described in particular with respect to specific embodiments, those skilled in the art can make various modifications to the embodiments described above in terms of quantity and other detailed configurations without departing from the scope of the technical idea and objectives of the present invention. [Explanation of symbols]

[0073] 200 Scenario Testing System 210 Mobile devices 220 External Systems 230 Scenario Inspection Device 241 Scenario Input Section 242 Policy Input Section 243 Structural Analysis Department 244 Inspection Department 245 Output section

Claims

1. A scenario testing system for testing chatbot scenarios, A scenario input unit that accepts a scenario composed of elements, A policy input unit that accepts input of a policy that includes conditions that the elements constituting the scenario must satisfy in the scenario, The aforementioned scenario input unit performs structural analysis of the elements included in the scenario, An inspection unit that checks whether the elements whose structure has been analyzed by the structural analysis unit conform to the conditions of the policy received by the policy input unit, The aforementioned inspection unit outputs the results of the inspection, Equipped with, A scenario inspection system in which the policy received by the policy input unit includes elements constituting a scenario, the scope to which the elements are subject to the policy, and the conditions that the elements within that scope must satisfy.

2. The elements that make up the scenario are: Steps to define the processing in the scenario, Step transitions that connect the steps and define the flow of the dialogue, A slot for storing information obtained in a step and / or the results obtained by processing said information, The scenario testing system according to claim 1, characterized by including the following:

3. A scenario testing method for testing chatbot scenarios, It accepts a scenario input consisting of elements. The system accepts input policies that include conditions that the elements constituting the scenario must satisfy within the scenario. We perform a structural analysis of the elements included in the submitted scenario. The structurally analyzed elements are checked to see if they meet the requirements of the accepted policy. Output the test results, The accepted policy includes the elements that constitute the scenario, the scope to which those elements are subject to the policy, and the conditions that the elements within that scope must satisfy. A computer-based scenario testing method.

4. Computers, A scenario receiving means that accepts input of a scenario composed of elements, A policy receiving means that accepts input of a policy that includes conditions that the elements constituting the scenario must satisfy in the scenario, A structural analysis means for performing structural analysis of elements included in the accepted scenario, A testing method for checking whether the structurally analyzed elements conform to the conditions of the accepted policy, An output means for outputting the test results, Includes, The policy received by the policy receiving means is a program that functions as a scenario checking method for checking a chatbot scenario, comprising elements constituting a scenario, the scope to which the elements are subject to the policy, and the conditions that the elements within that scope must satisfy.

Images