Information processing systems, information processing methods, and programs

Abstract

This technology provides control over the processing related to user interaction. [Solution] The information processing system acquires first scenario information to be used for a first dialogue with a user, executes a first dialogue with a user based on the first scenario information and a machine learning model, acquires second scenario information different from the first scenario information to be used for a second dialogue with a user based on the first scenario information, and after the first dialogue is completed, executes a second dialogue with a user based on the second scenario information and a machine learning model, and the first scenario information and the second scenario information have at least some information in common regarding dialogue control.

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing method, and a program.

Background Art

[0002] Techniques for interacting with a person are known. For example, there is a technique that acquires the content of an answer to a predetermined question from a job seeker, numerically evaluates the interview result of the job seeker based on the answer content of the job seeker and evaluation criterion information, and displays scoring information as information of the interview result.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides a technique for controlling processing related to interaction with a user.

Means for Solving the Problems

[0005] An information processing system according to one aspect of the present disclosure includes at least one memory and at least one processor, wherein the at least one processor acquires first scenario information to be used for a first interaction with a user, executes the first interaction with the user based on the first scenario information and a machine learning model, acquires second scenario information different from the first scenario information to be used for a second interaction with the user based on the first scenario information, executes the second interaction with the user based on the second scenario information and the machine learning model after the first interaction ends, and the first scenario information and the second scenario information share at least part of information related to interaction control.

Brief Description of the Drawings

[0006] [Figure 1] This is a block diagram showing an example of the overall structure of a dialogue system. [Figure 2] This is a block diagram showing an example of the functional configuration of a control device. [Figure 3] This figure shows the first example of template information. [Figure 4] This figure shows a second example of template information. [Figure 5] This figure shows the first example of a prompt. [Figure 6] The second example of a prompt is shown in the figure. [Figure 7] This is a processing flow illustrating an example of an interaction method. [Figure 8] This is a processing flow illustrating an example of a scenario control method. [Figure 9] This is a processing flow illustrating an example of an evaluation method. [Figure 10] A block diagram showing an example of a computer hardware configuration. [Modes for carrying out the invention]

[0007] Hereinafter, embodiments of this disclosure will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0008] One embodiment of this disclosure is an example of an information processing system that interacts with a user. Hereinafter, the information processing system according to this embodiment will be referred to as the "interaction system." The interaction system may perform the interaction with the user by controlling an interaction device that acts as the user's conversation partner.

[0009] In this embodiment, the dialogue system has a function to evaluate the user based on the dialogue with the user. For example, the dialogue system may interact with the user using a dialogue device and output an evaluation result based on the dialogue data recorded from the conversation between the user and the dialogue device. The dialogue system may control the dialogue with the user according to a specific scenario. For example, the dialogue scenario may be one in which the user is a store clerk, the dialogue device is a customer, and the user performs tasks such as responding to the customer's requests. Using this type of scenario, for example, the customer service skills of the user can be evaluated. Hereinafter, the function of evaluating (examining) the user through dialogue between the user and the dialogue device will be referred to as the "dialogue examination function".

[0010] In scenario-based dialogue tests, all test-takers use the same scenario, ensuring they are tested under the same conditions. However, testing multiple test-takers using the exact same scenario can compromise fairness among them. For example, if a particular test-taker possesses extensive knowledge of the scenario, the dialogue test may be conducted in a way that favors that test-taker. For instance, if a test-taker connects to the dialogue system remotely, and the connection is lost or the system malfunctions, that test-taker will have to retake the same scenario, gaining more knowledge about the dialogue in that scenario compared to a test-taker taking it for the first time. Another example is when a test-taker intentionally takes the test multiple times to gain knowledge about the dialogue in a given scenario.

[0011] This embodiment provides technology for controlling a device that interacts with users. In one respect, this embodiment allows for comparative testing of interactions with multiple test takers. In another respect, this embodiment reduces the impact of the number of times a test taker has taken the test on the evaluation results, as the flow of the interaction with the test taker changes even when using the same type of scenario, thereby improving fairness among test takers.

[0012] Furthermore, this embodiment provides a technology for evaluating users based on dialogue with them. In one aspect, this embodiment allows for the objective quantification of the characteristics of the test taker's dialogue (e.g., conciseness or logicality). In this embodiment, this quantified numerical value will be referred to as a score.

[0013] <Overall structure of the dialogue system> The overall configuration of the dialogue system according to this embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing an example of the overall configuration of the dialogue system.

[0014] As shown in Figure 1, the dialogue system 1000 includes a dialogue device 10, a control device 20, a generation device 30, and a terminal device 40. The dialogue device 10, the control device 20, the generation device 30, and the terminal device 40 may be connected to each other via a communication network so that they can communicate data with one another. The communication network may be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), a VPN (Virtual Private Network), or the Internet.

[0015] The dialogue device 10 is an example of a device that interacts with a dialogue participant S, who is an example of a user of the dialogue system 1000. The dialogue device 10 may receive utterances from the dialogue participant S. The dialogue device 10 may present utterances made by the dialogue device 10 to the dialogue participant S. The dialogue device 10 may interact with the dialogue participant S by repeatedly receiving utterances from the dialogue participant S and presenting utterances made by the dialogue device 10. Hereinafter, utterances made by the dialogue participant S will also be referred to as "user utterances." Also, utterances made by the dialogue device 10 will be referred to as "system utterances."

[0016] The interaction device 10 may be an information processing terminal such as a personal computer, a smartphone, or a tablet terminal. As an example, the interaction device 10 may display a CG (Computer Graphics) character on a display device, and execute interaction with the interlocutor S by controlling the CG character. As an example, the CG character may be a virtual human also called an avatar or the like. The CG character is not limited to a human, and may be a real living organism, a fictional living organism, a robot, an anthropomorphized object, an animation of a geometric pattern, or the like. In this case, the interlocutor S interacts with the virtual CG character displayed on the display device.

[0017] The interaction device 10 may be a robot. The interaction device 10 may execute interaction with the interlocutor S by causing the robot to operate. As an example, the robot may be a humanoid imitating a human. The robot is not limited to a human, and may be a real living organism, a fictional living organism, an anthropomorphized object, or the like. In this case, the interlocutor S interacts with the robot having a physical entity.

[0018] The interaction device 10 may generate information related to a user utterance and transmit the information to the control device 20. The interaction device 10 may receive information related to a system utterance from the control device 20, and present the system utterance to the interlocutor S. Hereinafter, the information related to a user utterance is also referred to as "user utterance information". Further, the information related to a system utterance is also referred to as "system utterance information".

[0019] The user utterance information may include at least one of an audio signal obtained by collecting the user utterance, text data indicating the utterance content of the user utterance, or a video signal or image signal obtained by imaging the face or body of the interlocutor S. The interaction device 10 may acquire the audio signal by collecting the user utterance using a microphone. The interaction device 10 may generate the video signal or image signal by imaging the face or body of the interlocutor S using a camera. The interaction device 10 may generate the text data indicating the utterance content by performing speech recognition on the audio signal obtained by collecting the user utterance.

[0020] System utterance information may include at least one of the following: a synthesized voice signal of system utterance, text data indicating the content of the system utterance, a video signal or image signal representing a CG character, or a control signal for controlling the expression of the dialogue device 10. The dialogue device 10 may present the system utterance to the interlocutor S by outputting a voice signal indicating the system utterance using a speaker. The dialogue device 10 may generate a voice signal indicating the system utterance by synthesizing the text data indicating the content of the system utterance. The dialogue device 10 may present the system utterance information to the interlocutor S by displaying a video or image representing a CG character on a display device. If the dialogue device 10 is a robot, the system utterance may be presented to the interlocutor S by moving the robot's face or body.

[0021] The control device 20 is an example of an information processing device that controls the dialogue between the interlocutor S and the dialogue device 10. The control device 20 may be a computer such as a personal computer, workstation, or server. The control device 20 may acquire user utterance information from the dialogue device 10. The control device 20 may generate system utterance information based on the user utterance information. The control device 20 may control the dialogue device 10 to utter system utterances based on the system utterance information.

[0022] The control device 20 may store dialogue data D. Dialogue data D may be electronic data recording the dialogue between the dialogue participant S and the dialogue device 10. Dialogue data D may include at least one of user utterance information or system utterance information. Dialogue data D may include at least a portion of user utterance information or system utterance information. Dialogue data D may be stored for each dialogue between the dialogue participant S and the dialogue device 10. Dialogue data D may include identification information that can identify the dialogue. An example of identification information that can identify the dialogue is time information indicating the time the dialogue was performed. The time information may, for example, be the time the dialogue started or the time the dialogue ended.

[0023] The generation device 30 is an example of an information processing device that performs a predetermined task based on machine learning. The generation device 30 may be a computer such as a personal computer, workstation, or server. The generation device 30 may perform a task of generating system utterances in response to a request from the control device 20. The generation device 30 may transmit the results of the system utterance generation to the control device 20.

[0024] The generation device 30 may include a machine learning model M. The machine learning model M is an example of a machine learning model trained to perform a predetermined task. The machine learning model M may be a machine learning model capable of generating various types of data such as text, audio, images, and videos. The machine learning model M may be, for example, a neural network, a large-scale language model, a generative model, or a foundational model.

[0025] The machine learning model M may support multimodal operation. That is, the machine learning model M may take at least two inputs, such as text data, audio data, image data, or video data. The machine learning model M may also output at least two outputs, such as text data, audio data, image data, or video data.

[0026] The machine learning model M may be implemented using a single machine learning model. The machine learning model M may be implemented by the collaboration of multiple machine learning models. The machine learning model M may consist of multiple machine learning models depending on the task being performed.

[0027] Terminal device 40 is an example of an information processing terminal such as a personal computer, smartphone, or tablet terminal operated by evaluator E, who is an example of a user of the dialogue system 1000. Evaluator E is the entity that evaluates dialogue participant S. For example, evaluator E may be the person in charge of evaluating dialogue participant S, who is a job seeker, in order to decide whether or not to hire them. For example, evaluator E may be the person in charge of performance evaluation of dialogue participant S, who is an employee. For example, evaluator E may be a training instructor. For example, evaluator E may be a test grader.

[0028] The terminal device 40 may obtain evaluation results regarding the interlocutor S from the control device 20. The terminal device 40 may present the evaluation results regarding the interlocutor S to the evaluator E. For example, the terminal device 40 may display the evaluation results on its display device. For example, the terminal device 40 may output synthesized audio of the evaluation results from its speaker.

[0029] Presenting information to a user of the dialogue system 1000 may include the processor performing at least a portion of the processing necessary to display the information on a display device. The display device may be provided in the same device as the processor, or in a different device from the processor. There may be multiple display devices.

[0030] Note that the overall configuration of the dialogue system 1000 shown in Figure 1 is just one example, and various system configurations are possible depending on the application and purpose. The dialogue system 1000 may consist of one or more devices. Each device included in the dialogue system 1000 may be part of a system composed of multiple devices. Each function included in the dialogue system 1000 may be implemented by any device that makes up the system. Each component included in the dialogue system 1000 may be included in any device that makes up the system.

[0031] The dialogue system 1000 may include multiple units of one or more of the dialogue devices 10, control devices 20, generation devices 30, or terminal devices 40. The control device 20 or generation device 30 may be implemented by multiple computers or as a cloud computing service. The control device 20 and generation device 30 may be implemented by an integrated standalone computer. The classification of devices such as the dialogue device 10, control device 20, generation device 30, and terminal device 40 shown in Figure 1 is just one example.

[0032] The dialogue device 10 can be any device that interacts with the dialoguer S, and is not limited to the following examples: The dialogue device 10 may be one or more information processing devices. The dialogue device 10 may be an information processing system that includes multiple information processing devices. The dialogue device 10 may include a terminal device having a display device. The dialogue device 10 may be a CG character such as a virtual human controlled by one or more hardware devices. The dialogue device 10 may be implemented by other devices included in the dialogue system 1000. The dialogue device 10 may be implemented by an information processing device or information processing system outside of the dialogue system 1000. Note that "external" means not included in the dialogue system 1000.

[0033] As an example, the dialogue system 1000 may consist of one or more server devices and one or more client devices. The one or more server devices may have one or more of the functions of the control device 20 and the generation device 30. The one or more client devices may have one or more of the functions of the dialogue device 10 and the terminal device 40. The server device may be implemented as a system including multiple information processing devices. The server device may be implemented as a cloud computing service.

[0034] As another example, the dialogue system 1000 may consist of a single information processing device. This single information processing device may include the functions of a dialogue device 10, a control device 20, a generation device 30, and a terminal device 40.

[0035] <Functional Configuration of Control Device> The functional configuration of the control device 20 will be explained with reference to Figure 2. Figure 2 is a block diagram showing an example of the functional configuration of the control device.

[0036] As shown in Figure 2, the control device 20 comprises a scenario storage unit 101, a dialogue storage unit 102, an evaluation result storage unit 103, a scenario acquisition unit 110, an utterance acquisition unit 120, an utterance generation unit 130, an utterance presentation unit 140, a dialogue evaluation unit 150, and a result output unit 160. The control device 20 functions as the scenario storage unit 101, dialogue storage unit 102, evaluation result storage unit 103, scenario acquisition unit 110, utterance acquisition unit 120, utterance generation unit 130, utterance presentation unit 140, dialogue evaluation unit 150, and result output unit 160 when a pre-installed control program is executed.

[0037] (Scenario Memory Unit) The scenario memory unit 101 stores information for acquiring scenario information. Scenario information is information that indicates the scenario of a dialogue. Scenario information may also include information that indicates the state of the dialogue. For example, information that indicates the state of the dialogue may include at least one of the following: the relationship between the dialogue participant S and the dialogue device 10, the preconditions for the dialogue, or the constraints on the utterances. The relationship between the dialogue participant S and the dialogue device 10 may include the relationship between the dialogue participant S and a CG character controlled by the dialogue device 10 or a robot representing the dialogue device 10. The relationship between the dialogue participant S and the dialogue device 10 may include the role of the dialogue participant S and the role of the dialogue device 10. Scenario information may include fixed lines spoken by the dialogue device 10, or it may include information for the dialogue device 10 to dynamically generate utterances in response to the utterances of the dialogue participant S. The scenario memory unit 101 may store multiple scenario information.

[0038] The relationship between the interlocutor S and the dialogue device 10 may include at least one or at least two combinations of the following: a relationship in which the interlocutor S provides services or goods to the dialogue device 10; a relationship in which the interlocutor S instructs the dialogue device 10; a relationship in which the dialogue device 10 pays remuneration to the interlocutor S; or a relationship in which the interlocutor S provides information to the dialogue device 10. Specifically, the relationship between the interlocutor S and the dialogue device 10 may include at least one of the following: a relationship between a test-taker and an interviewer; a relationship between a retail store clerk and a customer; a relationship between a superior and a subordinate; a relationship between a teacher and a student; a relationship between a speaker and an audience; a relationship between a doctor and a patient; or a relationship between a client and a representative. The client may include, for example, a professional such as a lawyer or a consultant.

[0039] For example, the scenario information may be a scenario relating to a conversation between a store employee and a customer in a retail store or restaurant. In this case, the role of the interlocutor S may be any of the following: "customer," "guest," "customer," "customer," "visitor," etc. The role of the dialogue device 10 may be any of the following: "store employee," "staff," "store person," etc.

[0040] Scenario information may include one or more system prompts and one or more guidance information. System prompts are information used to cause a machine learning model M to generate system utterances in accordance with the scenario. Guidance information is information used to explain the scenario to the interlocutor S when starting a dialogue. Guidance information may include voice guidance information to be spoken by the dialogue device 10 when starting a dialogue. Guidance information may include display guidance information to be shown by the dialogue device 10 when starting a dialogue. Guidance information may include information to be spoken by the dialogue device 10 when ending a dialogue. Guidance information may be associated with a corresponding system prompt. Guidance information may include information indicating the relationship between the interlocutor S and the dialogue device 10. Guidance information may be fixed lines of dialogue or lines dynamically generated by the machine learning model M.

[0041] System prompts may include information indicating the context of the dialogue. System prompts may be pre-generated based on the dialogue scenario. System prompts may include information indicating the relationship between the dialoguer S and the dialogue device 10. System prompts may include preconditions for the dialogue. System prompts may include constraints on utterances.

[0042] If a system prompt includes information indicating the relationship between the interlocutor S and the dialogue device 10, the system prompt only needs to include information about the role of the dialogue device 10, and does not need to explicitly include information about the role of the interlocutor S. The system prompt may also include information that indirectly indicates the role of the interlocutor S. The role of the dialogue device 10 may include at least one of the following: interviewer, customer, boss, subordinate, colleague, teacher, student, friend, audience, patient, or client.

[0043] The system prompt can also be the following string, for example. "You are a male customer in a clothing or accessories store. You are purchasing clothes while being asked questions by a store clerk. You don't have anything specific in mind that you want to buy. Your preferences are: simple design, blue cotton material, everyday use, and a budget of 50,000 yen. Please reveal the information gradually. You may mention that you are looking for clothes. Output only the customer's dialogue in 50 characters or less."

[0044] Here, "customer" and "shop assistant" are examples of information that indicate the relationship between the dialogue device 10 and the dialogue participant S. Note that "customer" is the role of the dialogue device 10, and "shop assistant" is the role of the dialogue participant S. "Please output only the customer's utterances within 50 characters" is an example of a constraint on the utterances. The other parts are examples of preconditions for the dialogue.

[0045] The system prompt may also be, as an example, the following string. The following string is an unspecified example of a system prompt that indirectly indicates the role of the interlocutor S. "You are a male customer in a clothing or accessories store. You are looking to buy clothes. You don't have a specific item in mind. Your preferences are simple design, blue cotton material, everyday wear, and a budget of 50,000 yen. Please provide information gradually. You may mention that you are looking for clothes. Output only the customer's dialogue within 50 characters."

[0046] Here, "customer" is an example of information regarding the role of the dialogue device 10. Furthermore, "customer" is an example of information indirectly indicating the role of the interlocutor S. By indicating that the dialogue device 10's role is "customer," it is indirectly indicated that the interlocutor S's role is "shop assistant."

[0047] Scenario information may include multiple system prompts. These multiple system prompts may, for example, be system prompts corresponding to the progress of the dialogue. The progress of the dialogue may, for example, be the number of turns in the dialogue or the elapsed time since the dialogue began.

[0048] Guidance information may include information related to the system prompt. Guidance information may include information indicating the relationship between the interlocutor S and the dialogue device 10. The information indicating the relationship between the interlocutor S and the dialogue device 10 may be the same as the information in the system prompt. Guidance information may contain less information than the system prompt. Guidance information may omit at least some of the prerequisites regarding the dialogue device 10 from the system prompt. Guidance information may include information unrelated to the system prompt. Guidance information may include information instructing the start of the dialogue. Information instructing the start of the dialogue may be located at the end of the guidance information.

[0049] The guidance information may be, for example, the following string of characters. "You are a clothing store employee. I am standing here as a customer, so please speak to me as if you were an employee. Go ahead."

[0050] Here, "shop assistant" and "customer" are examples of information that indicate the relationship between the interlocutor S and the dialogue device 10. Note that "customer" is the role of the dialogue device 10, and "shop assistant" is the role of the interlocutor S. "Please proceed" is an example of information that indicates the start of the dialogue.

[0051] The information for generating scenario information may include template information and text information. The template information may include system prompt templates. The template information may also include guidance information templates. The guidance information templates may include at least one of voice guidance information templates or display guidance information templates.

[0052] The system prompt template may include information regarding dialogue control. This dialogue control information may include information indicating the relationship between the interlocutor S and the dialogue device 10. If the system prompt template includes multiple templates, each of the multiple templates may include common information regarding dialogue control. Scenario information generated based on a system prompt template that includes dialogue control information may share at least some of the dialogue control information.

[0053] System prompt templates and guidance information templates may include one or more placeholders that embed one or more texts. These one or more texts may include one or more words. These one or more texts may constitute any processing unit in natural language processing. Examples of natural language processing processing units include words, phrases, clauses, or sentences.

[0054] The text information may include one or more pieces of text embedded in the placeholders of each template included in the template information. The text information may also include text that indicates the subject of the conversation. The text information may include, for example, nouns, noun phrases, or numerals. Nouns may include common nouns or proper nouns (e.g., store names, product names, service names, etc.).

[0055] Multiple templates included in template information may have the same text embedded within them. That is, if template information includes a system prompt template and a guidance information template, the placeholders in the system prompt template and the placeholders in the guidance information template may have the same text embedded within them. Also, if template information includes a voice guidance information template and a display guidance information template, the placeholders in the voice guidance information template and the placeholders in the display guidance information template may have the same text information embedded within them.

[0056] Template information and text information may be associated. That is, the text contained in the text information associated with the template information may be embedded in the template information. As an example of how to associate template information and text information, the template information and text information may be stored as a set, information that allows one piece of information to be retrieved from the other may be stored, or identification information for one piece of information and the other may be stored as a set.

[0057] The information used to obtain scenario information may include one or more pre-generated scenario information. The one or more scenario information may be scenario information in which at least some of the information differs. The one or more scenario information may be scenario information in which at least some of the information relating to dialogue control is common. The information relating to dialogue control may include one of the following: a system prompt template, information indicating the relationship between the interlocutor S and the dialogue device 10, preconditions for the dialogue, or constraints on utterances.

[0058] One or more scenario information entries may be generated based on template information and text information. One or more scenario information entries may be generated based on the same template information. One or more scenario information entries may be generated based on different text. One or more scenario information entries may be generated by embedding different text into the same template information.

[0059] (Dialogue Memory Unit) The dialogue memory unit 102 stores dialogue data D. Dialogue data D may be stored for each dialogue between the dialoguer S and the dialogue device 10. Dialogue data D may include user utterance information and system utterance information acquired by the control device 20 during the dialogue between the dialoguer S and the dialogue device 10, from start to finish. User utterance information included in dialogue data D may be acquired by the utterance acquisition unit 120. System utterance information included in dialogue data D may be generated by the utterance generation unit 130.

[0060] The dialogue data D may include, as an example, at least one of the following: time information, attribute information of the dialogue participant S, voice information of the dialogue participant S, image information of the dialogue participant S, voice information of the dialogue device 10, image information of the dialogue device 10, control information of the dialogue device 10, recorded data of the entire dialogue, and log information of the dialogue system 1000.

[0061] The attribute information of the interlocutor S may include the profile of the interlocutor S. The profile may, for example, include information from a resume or work history document. The attribute information of the interlocutor S may be pre-set or entered by the interlocutor S at the start of the conversation.

[0062] The voice information of the interlocutor S may include audio signals obtained by recording the interlocutor S's speech. The voice information of the interlocutor S may also include text data obtained by recognizing the audio signals. The image information of the interlocutor S may include still images or videos taken of at least a part of the interlocutor S's face or body. The image information of the interlocutor S may also include information recognized from the image of the interlocutor S. For example, the image information of the interlocutor S may include facial expression labels or posture information. Facial expression labels are labels that indicate a class that classifies a person's facial expression. Posture information is information that indicates the position of a person's feature points. For example, posture may be estimated by obtaining the position of each joint using posture estimation technology.

[0063] The voice information of the dialogue device 10 may include a voice signal synthesized from system utterances. The voice information of the dialogue device 10 may also include text data used to synthesize the voice signal. The image information of the dialogue device 10 may include still images or videos that include at least a part of the face or body of a CG character. The image information of the dialogue device 10 may also include information used to generate the image of the CG character. For example, the image information of the dialogue device 10 may include facial expression labels or posture information. The control information of the dialogue device 10 may include body control information, such as the facial expressions and gestures of the robot, if the dialogue device 10 is a robot.

[0064] The recorded data of the entire dialogue may be video data obtained by combining the audio and video signals of the dialogue participant S with the audio and video signals of the dialogue device 10. The recorded data of the entire dialogue may be combined by the dialogue device 10 or by the control device 20.

[0065] The log information of the dialogue system 1000 may also be historical information recording the results of processing performed by the dialogue system 1000 during the dialogue between the dialoguer S and the dialogue device 10. The log information of the dialogue system 1000 may include at least one of the processing results of the dialogue device 10, the processing results of the control device 20, or the processing results of the generation device 30. As an example, the log information of the dialogue system 1000 may include the recognition results of user utterances, input information to the machine learning model M, output information from the machine learning model M, and the content of system utterances. Hereinafter, the input information to the machine learning model M will also be referred to as "model input information." Similarly, the output information from the machine learning model M will also be referred to as "model output information."

[0066] Each piece of information contained in the dialogue data D may be stored in association with time information. Methods for associating each piece of information with time information include, for example, storing each piece of information and time information as a set, storing information that allows one piece of information to be retrieved from the other, or storing one piece of information and its identification information as a set.

[0067] The dialogue memory unit 102 may store history information regarding the use of the dialogue system 1000. The history information may include information used in dialogue with the dialogue participant S. The information used in dialogue may include scenario information used to execute the dialogue. The information used in dialogue may also include information used to generate the scenario information. The information used to generate the scenario information may include, for example, template information and text information. The history information may be stored for each dialogue participant S.

[0068] (Evaluation result storage unit) The evaluation result storage unit 103 stores evaluation results for the interlocutor S. The evaluation results for the interlocutor S may be generated by the dialogue evaluation unit 150. The evaluation result storage unit 103 may store multiple evaluation results for the interlocutor S. Multiple evaluation results may be generated based on dialogue data D for multiple different dialogues. Multiple dialogues may differ in at least one of the following: time, scenario, evaluation criteria, evaluation value, etc. The evaluation result storage unit 103 may store the trends or statistical information of the evaluation values ​​shown in the multiple evaluation results. Statistical information may include, for example, maximum, minimum, average, etc.

[0069] External evaluation values ​​may be stored in the evaluation result storage unit 103. External evaluation values ​​are evaluation values ​​obtained by evaluating the dialogue participant S based on information other than the dialogue data D. External evaluation values ​​may be generated by an external information processing device or information processing system of the dialogue system 1000 and stored in the evaluation result storage unit 103. As an example, external evaluation values ​​may include the results of the dialogue participant S's aptitude test, information indicating language ability, information indicating work experience, information indicating that the participant possesses specialized knowledge in a particular field, etc.

[0070] (Scenario acquisition section) The scenario acquisition unit 110 acquires scenario information. The scenario acquisition unit 110 may acquire scenario information to be used for dialogue between the interlocutor S and the dialogue device 10. The scenario acquisition unit 110 may acquire first scenario information to be used for the first dialogue and second scenario information to be used for the second dialogue. The first dialogue and the second dialogue may be dialogues with the same interlocutor S or dialogues with different interlocutor S.

[0071] The scenario acquisition unit 110 may acquire scenario information before starting a dialogue with the interlocutor S. The scenario acquisition unit 110 may acquire first scenario information before starting a first dialogue. The scenario acquisition unit 110 may acquire second scenario information before starting a second dialogue. The scenario acquisition unit 110 may acquire second scenario information after the first dialogue has finished and before starting a second dialogue. The scenario acquisition unit 110 may acquire second scenario information before starting a first dialogue. The scenario acquisition unit 110 may acquire first and second scenario information simultaneously.

[0072] The first scenario information and the second scenario information may differ in at least some of their information. The first scenario information and the second scenario information may share at least some of their information regarding dialogue control. The information regarding dialogue control may include information indicating the status of the dialogue. The information indicating the status of the dialogue may include information indicating the relationship between the dialogue participant S and the dialogue device 10.

[0073] The scenario acquisition unit 110 may acquire scenario information by generating scenario information. The scenario acquisition unit 110 may read template information and text information for generating scenario information from the scenario storage unit 101. The scenario acquisition unit 110 may generate scenario information based on the template information and text information read from the scenario storage unit 101. The scenario acquisition unit 110 may generate scenario information by rewriting the template information using one or more texts included in the text information. The scenario acquisition unit 110 may generate scenario information by embedding one or more texts included in the text information into placeholders for each template included in the template information.

[0074] The scenario acquisition unit 110 may generate first scenario information based on template information and first text. The scenario acquisition unit 110 may generate second scenario information based on template information and second text. The first text and the second text may be different. The first text does not have to contain at least one of the one or more words contained in the second text. The second text does not have to contain at least one of the one or more words contained in the first text.

[0075] The scenario acquisition unit 110 may select one or more texts to embed in the template information from the text information. The scenario acquisition unit 110 may select one or more texts randomly. The scenario acquisition unit 110 may select one or more texts based on predetermined criteria. The scenario acquisition unit 110 may select an unselected text from among the one or more texts included in the text information. When the scenario acquisition unit 110 randomly selects one or more texts, it may change the probability distribution used for text selection according to the attributes of the interlocutor S. For example, if the text information includes "clothing store," "flower shop," and "food store," and the interlocutor S has work experience in a clothing store, the probability of "clothing store" appearing may be increased or decreased. If it is decreased, the probability of "clothing store" appearing may be set to zero, which is equivalent to removing some of the texts included in the text information according to the attributes of the interlocutor S, and then randomly selecting texts from the remaining texts according to a predetermined probability distribution.

[0076] The scenario acquisition unit 110 may sequentially select one or more texts based on the order of the texts included in the text information. The scenario acquisition unit 110 may randomly generate a common order of texts for multiple interlocutors S and sequentially select one or more texts according to that order. The scenario acquisition unit 110 may randomly generate a text order for each interlocutor S and sequentially select one or more texts according to that order. When the scenario acquisition unit 110 randomly generates the order of texts, it may randomly generate the order of texts each time all texts have been selected.

[0077] The scenario acquisition unit 110 may select one or more texts to embed in the template information based on the history information of the interlocutor S. The scenario acquisition unit 110 may select one or more texts to embed in the template information from a set of one or more texts included in the text information, excluding the text used in the most recent dialogue with the interlocutor S. The scenario acquisition unit 110 may select one or more texts to embed in the template information from a set of one or more texts included in the text information, excluding the text used in past dialogues with the interlocutor S. The scenario acquisition unit 110 may set a priority for each text included in the text information based on the history information of the interlocutor S and then randomly select one or more texts. For example, the scenario acquisition unit 110 may select a second text to be used to generate the second scenario information from a set of one or more texts included in the text information, excluding the first text used to generate the first scenario information.

[0078] The scenario acquisition unit 110 may select one or more texts to embed in the template information based on information acquired from the dialogue device 10. For example, the scenario acquisition unit 110 may select one or more texts based on information captured by the camera of the dialogue device 10. The information captured by the camera of the dialogue device 10 may include, for example, information about the face, body, clothing, or accessories of the dialogue participant S. Alternatively, for example, the scenario acquisition unit 110 may select one or more texts based on information collected by the microphone of the dialogue device 10. The information collected by the microphone of the dialogue device 10 may include, for example, the voice, speech content, or background sound of the dialogue participant S. Alternatively, for example, the scenario acquisition unit 110 may select one or more texts based on information specific to the dialogue device 10. The information specific to the dialogue device 10 may include, for example, the type of dialogue device 10, its installation location, its affiliation, software configuration, or hardware configuration.

[0079] The scenario acquisition unit 110 may acquire one or more texts to be embedded in the template information by searching. For example, the scenario acquisition unit 110 may search for one or more texts to be embedded in the template information from a predetermined data source. The predetermined data source may include an internal database, a website (e.g., a news site or dictionary site), a search engine, or a social networking service (SNS). For example, the scenario acquisition unit 110 may acquire texts related to seasons or current events by searching.

[0080] The scenario acquisition unit 110 may generate one or more texts to be embedded in the template information based on a machine learning model. The machine learning model may be machine learning model M, or it may be another machine learning model different from machine learning model M. Other machine learning models may include, for example, a large-scale language model, a generative model, or a foundational model. For example, the scenario acquisition unit 110 may generate text related to seasons or current events.

[0081] The scenario acquisition unit 110 may generate a system prompt based on the system prompt template included in the template information and one or more selected texts. The scenario acquisition unit 110 may also generate a system template by rewriting the system prompt template using one or more texts. The scenario acquisition unit 110 may also generate a system template by embedding one or more texts in the placeholders of the system prompt template.

[0082] The scenario acquisition unit 110 may generate voice guidance information based on the voice guidance information template included in the template information and one or more selected texts. The scenario acquisition unit 110 may generate voice guidance information by rewriting the voice guidance information template using one or more texts. The scenario acquisition unit 110 may generate voice guidance information by embedding one or more texts in the placeholders of the voice guidance information template.

[0083] The scenario acquisition unit 110 may generate display guidance information based on the display guidance information template included in the template information and one or more selected texts. The scenario acquisition unit 110 may generate display guidance information by rewriting the display guidance information template using one or more texts. The scenario acquisition unit 110 may generate display guidance information by embedding one or more texts in the placeholders of the display guidance information template.

[0084] The scenario acquisition unit 110 may generate system prompts, voice guidance information, and display guidance information based on the same text. The scenario acquisition unit 110 may generate system prompts, voice guidance information, and display guidance information by rewriting the system prompt template, voice guidance information template, and display guidance information template using the same text. The scenario acquisition unit 110 may generate system prompts, voice guidance information, and display guidance information by embedding the same text in the placeholders of the system prompt template, the voice guidance information template, and the display guidance information template.

[0085] Furthermore, the scenario acquisition unit 110 does not need to generate voice guidance information if the dialogue device 10 is not instructed to speak voice guidance information when starting a dialogue. Also, the scenario acquisition unit 110 does not need to generate display guidance information if the dialogue device 10 is not instructed to display display guidance information when starting a dialogue.

[0086] The scenario acquisition unit 110 may acquire scenario information by reading scenario information stored in the scenario storage unit 101. The scenario acquisition unit 110 may randomly select scenario information to read from the scenario storage unit 101. The scenario acquisition unit 110 may select scenario information to read from the scenario storage unit 101 based on predetermined criteria. The scenario acquisition unit 110 may select unselected scenario information from one or more scenario information stored in the scenario storage unit 101. The scenario acquisition unit 110 may change the selection probability of each scenario information when randomly selecting scenario information based on the attributes of the interlocutor S.

[0087] The scenario acquisition unit 110 may sequentially select scenario information based on the order of the scenario information stored in the scenario storage unit 101. The scenario acquisition unit 110 may randomly generate an order of scenario information common to multiple interlocutors S and sequentially select the scenario information according to that order. The scenario acquisition unit 110 may randomly generate an order of scenario information for each interlocutor S and sequentially select the scenario information according to that order. When the scenario acquisition unit 110 randomly generates the order of scenario information, it may randomly generate the order of scenario information each time all scenario information has been selected.

[0088] The scenario acquisition unit 110 may select scenario information to read from the scenario storage unit 101 based on the history information of the interlocutor S. The history information of the interlocutor S may, for example, include information used in the dialogue between the interlocutor S and the dialogue device 10. The information used in the dialogue may, for example, include information indicating that the dialogue was executed based on the first scenario information. The scenario acquisition unit 110 may select scenario information to read from the set of scenario information stored in the scenario storage unit 101, excluding the scenario information used in the most recent dialogue with the interlocutor S. The scenario acquisition unit 110 may select scenario information to read from the set of scenario information stored in the scenario storage unit 101, excluding the scenario information used in past dialogues with the interlocutor S. The scenario acquisition unit 110 may, based on the history information of the interlocutor S, set a priority for each scenario information stored in the scenario storage unit 101 and then randomly select scenario information. For example, the scenario acquisition unit 110 may select the second scenario information from the set of scenario information stored in the scenario storage unit 101, excluding the first scenario information. Alternatively, the scenario acquisition unit 110 may select the second scenario information from one or more scenario information stored in the scenario storage unit 101, in which at least some of the information relating to dialogue control is common with the first scenario information.

[0089] The scenario acquisition unit 110 may select scenario information to read from the scenario storage unit 101 based on the metadata attached to the scenario information. The metadata attached to the scenario information may include, for example, information indicating the difficulty level of the scenario or arbitrary tag information.

[0090] The scenario acquisition unit 110 may narrow down the scenario information to be read from the scenario storage unit 101 based on the attributes of the interlocutor S or the attributes of the evaluator E. The scenario acquisition unit 110 may also select the scenario information to be read from one or more scenario information narrowed down based on the attributes of the interlocutor S or the attributes of the evaluator E. The attributes of the interlocutor S may include, for example, industry, occupation, position, information listed in the resume, information listed in the work history, etc. The industry, occupation, and position may include the industry, occupation, and position desired by the interlocutor S. The attributes of the evaluator E may include, for example, industry, occupation, information on the personnel the evaluator is looking for, general company information, etc.

[0091] The scenario acquisition unit 110 may acquire predetermined scenario information as first scenario information. The scenario acquisition unit 110 may acquire predetermined scenario information as first scenario information to be used for dialogue with multiple interlocutors S. The scenario acquisition unit 110 may acquire second scenario information that is different from the predetermined scenario information. The scenario acquisition unit 110 may generate second scenario information based on the first scenario information. The scenario acquisition unit 110 may generate second scenario information by changing one or more texts included in the first scenario information.

[0092] The scenario acquisition unit 110 may select the first scenario information from one or more candidates, and randomly select the second scenario information from a number of candidates such that the probability of selecting the same scenario information as the first scenario information is practically eliminated. The scenario acquisition unit 110 may also select the first and second scenario information from multiple candidates such that there is no overlap.

[0093] The scenario acquisition unit 110 may select a first text to be embedded in the template information in order to generate the first scenario information from one or more candidates, and may randomly select a second text to be embedded in the template information in order to generate the second scenario information from a number of candidates such that the probability of selecting the same text as the first text is practically eliminated. The scenario acquisition unit 110 may also select the first text and the second text from multiple candidates such that there is no overlap.

[0094] Figure 3 shows a first example of template information. As shown in Figure 3, the template information may include a system prompt template 300, a voice guidance information template 310, and a display guidance information template 320.

[0095] The system prompt template 300 includes two placeholders 301 (301-1, 301-2) for embedding product categories. The voice guidance information template 310 includes a placeholder 311 for embedding product categories. The display guidance information template 320 includes a placeholder 321 for embedding product categories. Product categories may be nouns or noun phrases. Examples of nouns representing product categories include "hats," "jewelry," "flowers," "clothing," "furniture," "smartphones," and "costume items." Examples of noun phrases representing product categories include "red flowers," "white flowers," "blue flowers," "large flowers," and "small flowers."

[0096] Placeholders 301, 311, and 321 are embedded with the same text indicating the product category. When generating the first scenario information, placeholders 301, 311, and 321 are each embedded with text indicating the first product category (for example, "hats"). When generating the second scenario information, placeholders 301, 311, and 321 are each embedded with text indicating the second product category (for example, "jewelry").

[0097] Figure 4 shows a second example of template information. Figure 4 shows a system prompt template 330 used for the second interaction. The system prompt template 330 includes a placeholder 331 for embedding the product category, a placeholder 332 for embedding the interaction data D, and instruction information 333.

[0098] Placeholder 331 is embedded with text indicating a product category (for example, "hats"). Placeholder 332 is embedded with at least a portion of the dialogue data D generated in a previous dialogue with the interlocutor S. Placeholder 332 may also be embedded with the dialogue data D generated in the first dialogue. Instruction information 333 contains information that instructs the dialogue to proceed in a different flow than the dialogue data D. Placeholder 331 may also be embedded with the product category embedded in the system prompt used in the first dialogue. According to the system prompt template 330, a new dialogue can be executed in a different flow than a previously executed dialogue, following the same scenario for the same product category.

[0099] (Speech acquisition unit) The speech acquisition unit 120 acquires user speech information. The speech acquisition unit 120 may generate user speech information based on the audio signal obtained from the user speech. The speech acquisition unit 120 may acquire the audio signal obtained from the user speech from the dialogue device 10.

[0100] The speech acquisition unit 120 may generate user speech information based on the video signal captured of the interlocutor S. The speech acquisition unit 120 may acquire the video signal captured of the interlocutor S from the dialogue device 10.

[0101] The speech acquisition unit 120 may generate user utterance information based on text data indicating the content of the user utterance. Alternatively, the speech acquisition unit 120 may generate text data indicating the content of the user utterance by performing speech recognition on the audio signal obtained from the user utterance.

[0102] The speech acquisition unit 120 may determine whether the video signal acquired from the dialogue device 10 is a video signal in which the dialoguer S is properly captured. The speech acquisition unit 120 may determine whether the face of the dialoguer S is captured in the video signal. The speech acquisition unit 120 may determine whether the size of the face of the dialoguer S captured in the video signal is within a predetermined range. The speech acquisition unit 120 may use a face detector to detect the face of the dialoguer S captured in the video signal. The face detector may, for example, be constructed using a machine learning model. The speech acquisition unit 120 may use a gaze estimator to determine whether the gaze of the dialoguer S captured in the video signal can be detected.

[0103] When calculating evaluation values ​​for the physical movements of the interlocutor S, the speech acquisition unit 120 may determine whether or not the interlocutor S's body is captured in the video signal. The speech acquisition unit 120 may also determine whether or not the size of the interlocutor S's body captured in the video signal is within a predetermined range. The speech acquisition unit 120 may use a body detector to detect the interlocutor S's body captured in the video signal. The body detector may, for example, be constructed using a machine learning model.

[0104] The speech acquisition unit 120 may notify the interlocutor S if the interlocutor S is not properly captured in the video signal. For example, the speech acquisition unit 120 may send a notification to the dialogue device 10 indicating that the interlocutor S is not properly captured. The dialogue device 10 may display a message on its display device indicating that the interlocutor S is not properly captured. The dialogue device 10 may also speak to indicate that the interlocutor S is not properly captured. The speech acquisition unit 120 may terminate the dialogue with the interlocutor S if the interlocutor S is not properly captured in the video signal.

[0105] The speech acquisition unit 120 may store the acquired user speech information in the dialogue storage unit 102. The speech acquisition unit 120 may store the user speech information in real time, or it may store the user speech information in batches in predetermined processing units. The predetermined processing unit may be, for example, the amount of user speech information data, the number of turns in the dialogue, or the elapsed time of the dialogue.

[0106] The speech acquisition unit 120 may include user utterance information in the dialogue data D stored in the dialogue memory unit 102. The speech acquisition unit 120 may also store the user utterance information in the dialogue memory unit 102 in association with the time information included in the dialogue data D.

[0107] (Speech generation unit) The speech generation unit 130 generates system speech information. The speech generation unit 130 may generate system speech information based on user speech information acquired by the speech acquisition unit 120. The speech generation unit 130 may generate system speech information based on a machine learning model M. The speech generation unit 130 may generate system speech information based on user speech information, input information, and a machine learning model M.

[0108] Furthermore, generating other information based on certain information and a machine learning model may include performing one or more of the following processes: • The process of inputting certain information into a machine learning model to generate other information. • The process of generating information based on existing information, inputting that information into a machine learning model, and then generating other information from that model. • Using information extracted from certain data, a machine learning model is used to perform a generation process that generates other information.

[0109] The input information may be generated based on the scenario information. The scenario information may be set by the scenario acquisition unit 110. The scenario information may be pre-set. The scenario information may be set based on predetermined criteria. The scenario information may be set based on the instructions of the interlocutor S. The scenario information may be set based on the instructions of the evaluator E.

[0110] The speech generation unit 130 may generate model input information based on user utterance information and transmit it to the generation device 30. The model input information may include at least system prompts and dialogue history information, which are examples of input information. The speech generation unit 130 may acquire system prompts from scenario information acquired by the scenario acquisition unit 110. The speech generation unit 130 may select system prompts to acquire from the scenario information according to the progress of the dialogue. The speech generation unit 130 may select system prompts to acquire from the scenario information according to the industry or occupation. The speech generation unit 130 may acquire multiple system prompts and combine them. As an example, the speech generation unit 130 may acquire system prompts according to the industry and system prompts according to the occupation, and generate a combined system prompt.

[0111] The speech generation unit 130 may select a system prompt based on a machine learning model. The machine learning model may be machine learning model M, or another machine learning model different from machine learning model M. Other machine learning models may include, for example, a large-scale language model, a generative model, or a foundational model. The speech generation unit 130 may input information about the dialogue between the interlocutor S and the dialogue device 10, along with multiple system prompts, into the machine learning model to obtain information for selecting one or more system prompts. The information about the dialogue may include dialogue history information. The speech generation unit 130 may also generate a system prompt based on the machine learning model.

[0112] The dialogue history information may be information that links the utterances of the interlocutor S and the utterances of the dialogue device 10 in chronological order. The dialogue history information may include the utterances from the start of the dialogue up to the immediately preceding utterance. The dialogue history information may include a predetermined number of utterances, going backward in time from the immediately preceding utterance. Each utterance may be associated with information indicating the speaker (interlocutor S or dialogue device 10). The information indicating the speaker may be added before or after the utterance. The information indicating the speaker may be added in a way that makes it identifiable by a predetermined symbol. The boundaries of each utterance may be indicated by a predetermined symbol. The dialogue history information may include information indicating the speaker of the target that generates the utterance. The information indicating the speaker of the target that generates the utterance does not need to be associated with utterances.

[0113] The dialogue history information may be, for example, the following string. • "Shop assistant: Welcome, customer:" However, "Welcome" is an example of the content of the preceding utterance. "Shop assistant" and "customer" are examples of information indicating the speaker. ":" is an example of a symbol that separates the speaker from the content of their utterance. "¥n" is an example of a symbol that indicates the boundary of the content of an utterance (in this case, the end of the content of the preceding utterance). In the example above, since no content is associated with "customer," the utterance of the customer (i.e., the dialogue device 10) is the target of generation. In the example above, the dialogue history information only contains the content of the preceding utterance, but if one or more turns have taken place before that, the content of utterances from previous turns may be included before "shop assistant."

[0114] Dialogue history information is not limited to text information, and may include audio information or image information, or it may include text information and at least one of audio information or image information. Specifically, dialogue history information may include information that links the audio information of the dialogue participant S, the image information of the dialogue participant S, the audio information of the dialogue device 10, and the image information of the dialogue device 10 in chronological order.

[0115] The speech generation unit 130 may generate model input information by embedding system prompts and dialogue history information into a predetermined template. The speech generation unit 130 may also generate model input information by processing the template, system prompts, and dialogue history information based on predetermined rules. The speech generation unit 130 may also generate model input information by inputting the template, system prompts, and dialogue history information into a machine learning model. The machine learning model may be machine learning model M, or another machine learning model different from machine learning model M. Other machine learning models may include, for example, a large-scale language model, a generative model, or a foundational model.

[0116] A template for generating model input information may include one or more placeholders for embedding system prompts or dialogue history information. The template may also include instruction information that instructs the generation of utterances. The instruction information may also include information that instructs the output format of the model output information. The output format of the model output information may include, for example, at least one of text data, audio signals, video signals, or control signals. The template may also include constraint information regarding the model output information. The constraint information may include the length of the utterance, what can be said, what can be said if asked, what should not be said even if asked, and the manner of speaking. The instruction information and constraint information may also be predetermined fixed sentences.

[0117] The template may be optimized using any prompt tuning method to obtain good model output information. For example, the template may be optimized by searching for the optimal template using a genetic algorithm or the like, based on a benchmark that evaluates prompts generated using the template.

[0118] Model input information may include text data, image data, or audio data. Text data may, for example, be natural language sentences called prompts. Image data may, for example, be still images or videos. Text data may also be text data obtained by speech recognition of audio data or video data. Text data may also be text obtained by character recognition of image data. Image data may, for example, include images of the user. Audio data may, for example, include the voice of the user speaking. Audio data may also be speech data obtained by speech synthesis of text data.

[0119] The speech generation unit 130 may receive model output information generated by the generation device 30 inputting model input information into the machine learning model M from the generation device 30. The model input information may consist of system prompts and dialogue history information, which may be input to the machine learning model M separately. Alternatively, the model input information may consist of system prompts and dialogue history information combined and input to the machine learning model M.

[0120] The speech generation unit 130 may generate system speech information based on the model output information. If the model output information includes an audio signal indicating system speech or a video signal representing a CG character, the speech generation unit 130 may use at least one of the audio signal or the video signal as system speech information. If the model output information includes text data indicating the content of the speech, the speech generation unit 130 may synthesize an audio signal based on the text data. If the model output information includes information for controlling the expression of the dialogue device 10 (for example, facial expression labels, information on the shape of the mouth when speaking, or posture information), the speech generation unit 130 may synthesize a video signal representing a CG character based on that information.

[0121] The speech generation unit 130 may generate system speech information based on a portion of the model output information. If the model output information is longer than a predetermined data length or a randomly determined data length, the speech generation unit 130 may shorten the model output information so that it is less than or equal to that data length. If the machine learning model M outputs model output information in a stream, the machine learning model M may be instructed to stop subsequent output when the data length is reached.

[0122] As an example, the speech generation unit 130 may acquire N sentences sequentially from the beginning of the model output information. N is a positive integer and may be determined randomly. As an example, the speech generation unit 130 may acquire N' sentences sequentially from the beginning of the model output information so that the total number of sentences is no more than or equal to M. M and N' are positive integers. Sentence boundaries may be specified by predetermined symbols. As an example, periods (such as "。" or ".") and symbols (such as "?" or "!") may be used as delimiters.

[0123] (Speech presentation section) The speech presentation unit 140 presents system utterances to the interlocutor S. The speech presentation unit 140 may present system utterances to the interlocutor S based on system utterance information generated by the speech generation unit 130. The speech presentation unit 140 may also present system utterances to the interlocutor S by controlling the utterances of the dialogue device 10 based on the system utterance information.

[0124] The speech presentation unit 140 may transmit a synthesized voice signal of system utterances to the dialogue device 10. The dialogue device 10 may output voice indicating system utterances from its speaker based on the voice signal received from the control device 20.

[0125] The speech presentation unit 140 may transmit text data indicating the content of the speech to the dialogue device 10. The dialogue device 10 may generate an audio signal by synthesizing the text data received from the control device 20 and output it from the speaker.

[0126] The speech presentation unit 140 may transmit a video signal representing a CG character to the dialogue device 10. The dialogue device 10 may display an image representing a CG character on a display device based on the video signal received from the control device 20.

[0127] The speech presentation unit 140 may transmit information used for synthesizing video signals (for example, facial expression labels, information on the shape of the mouth when speaking, or posture information) to the dialogue device 10. The dialogue device 10 may synthesize video signals based on the information received from the control device 20 and display an image representing a CG character on the display device.

[0128] The speech presentation unit 140 may transmit control signals to the dialogue device 10 to control the representation of the CG character. The dialogue device 10 may control the representation of the CG character based on the control signals received from the control device 20.

[0129] The speech presentation unit 140 may control the dialogue device 10 to utter system utterances that explain the situation of the dialogue (hereinafter also referred to as "explanatory utterances"). The speech presentation unit 140 may cause the dialogue device 10 to utter explanatory utterances based on voice guidance information included in the scenario information acquired by the scenario acquisition unit 110. The speech presentation unit 140 may transmit the voice guidance information included in the scenario information to the dialogue device 10. The speech presentation unit 140 may transmit a voice signal synthesized from the voice guidance information to the dialogue device 10.

[0130] The speech presentation unit 140 may control the dialogue device 10 to utter a system utterance in response to a user utterance. The speech presentation unit 140 may cause the dialogue device 10 to utter a system utterance in response to a user utterance based on the system utterance information generated by the speech generation unit 130. The speech presentation unit 140 may transmit at least one of the text data, audio signals, video signals, or control signals included in the system utterance information to the dialogue device 10.

[0131] The speech presentation unit 140 may store system speech information in the dialogue memory unit 102. The speech presentation unit 140 may store system speech information in real time, or it may store system speech information in batches in predetermined processing units. The predetermined processing unit may be, for example, the amount of system speech information data, the number of dialogue turns, or the elapsed time of the dialogue.

[0132] The speech presentation unit 140 may include system utterance information in the dialogue data D stored in the dialogue memory unit 102. The speech presentation unit 140 may also store the system utterance information in the dialogue memory unit 102 in association with the time information included in the dialogue data D.

[0133] (Dialogue Evaluation Department) The dialogue evaluation unit 150 generates an evaluation result regarding the dialogue participant S. The dialogue evaluation unit 150 may generate an evaluation result regarding the dialogue participant S based on dialogue data D read from the dialogue storage unit 102. The dialogue evaluation unit 150 may generate an evaluation result regarding the dialogue participant S based on dialogue data D that includes at least one of the first dialogue or the second dialogue. The dialogue evaluation unit 150 may generate an evaluation result regarding the dialogue participant S based on dialogue data D that includes a third dialogue different from the first and second dialogues. The evaluation result regarding the dialogue participant S may include a score of 1 or more. The dialogue evaluation unit 150 may select the scores to include in the evaluation result according to the evaluation criteria of evaluator E.

[0134] The dialogue evaluation unit 150 may calculate a score that evaluates the characteristics of the dialogue content of the dialogue participant S. The characteristics of the dialogue content may include, for example, at least one of conciseness, logic, or responsiveness.

[0135] The dialogue evaluation unit 150 may convert the dialogue data D into predetermined data. The predetermined data may differ depending on the characteristics to be evaluated. The dialogue evaluation unit 150 may calculate a score based on the converted dialogue data D. The dialogue evaluation unit 150 may calculate a score based on the dialogue data D before conversion and the converted dialogue data D. Hereinafter, the dialogue data D before conversion will also be referred to as "pre-conversion data". Similarly, the converted dialogue data D (predetermined data) will also be referred to as "post-conversion data".

[0136] The dialogue evaluation unit 150 may convert the pre-conversion data into predetermined post-conversion data based on a machine learning model. The dialogue evaluation unit 150 may also generate post-conversion data based on the pre-conversion data and the machine learning model. The machine learning model may be machine learning model M, or another machine learning model different from machine learning model M. Other machine learning models may include, for example, a large-scale language model, a generative model, or a foundational model.

[0137] The dialogue evaluation unit 150 may generate model input information corresponding to the characteristics to be evaluated and transmit it to the generation device 30. The model input information may include instruction information for converting the pre-conversion data to predetermined post-conversion data. The model input information may also include information indicating the output format of the post-conversion data. The dialogue evaluation unit 150 may receive model output information generated by the generation device 30 inputting the model input information into the machine learning model M from the generation device 30.

[0138] When calculating a score related to conciseness, the dialogue evaluation unit 150 may be configured as follows. Hereinafter, the score related to conciseness will also be referred to as the "conciseness score." The model input information may include instruction information that instructs the summarization of the data before conversion. The model input information may also include instruction information that instructs the deletion of utterances that do not make sense in the dialogue. Utterances that do not make sense in the dialogue may include, for example, dog words, fillers, etc. The model input information may also include instruction information that instructs the addition of information indicating the speaker (i.e., the dialoguer S or the dialogue device 10) to each of the summarized data.

[0139] Model input information may include data indicating the content of utterances. Model input information may include both data indicating the content of user utterances and data indicating the content of system utterances. Model input information may include only data indicating the content of user utterances. Model input information may include system prompts used in the dialogue.

[0140] The model input information may include data indicating the content of utterances in segments that divide the dialogue from start to finish into predetermined units. A predetermined unit may be a dialogue, a role-play, or a turn. A dialogue is a segment that includes a series of utterances from start to finish. A role-play is a segment corresponding to each role-play when the dialogue test includes multiple role-plays. A turn is a segment that includes one user utterance and one system utterance.

[0141] For example, the model input information may include text data representing the content of user utterances and text data representing the content of system utterances. As another example, the model input information may include only text data representing the content of user utterances. The model output information may include text data that summarizes the text data included in the model input information.

[0142] As an example, the model input information may include audio data of user utterances and audio data of system utterances synthesized into speech. As another example, the model input information may include only audio data of user utterances. The model output information may include audio data that summarizes the audio data included in the model input information.

[0143] The dialogue evaluation unit 150 may calculate a conciseness score based on the data volume of the data before conversion and the data volume of the data after conversion. If the data before conversion or the data after conversion is text data, the data volume may include, for example, the number of characters, the number of words, the number of sentences, the data size, etc. If the data before conversion or the data after conversion is audio data, the data volume may include, for example, the length of time, the file size, etc. The data volume of the data before conversion may include at least a portion of the data volume of the data before conversion. The data volume of the data after conversion may include at least a portion of the data volume of the data after conversion.

[0144] If the model input information includes text data, the dialogue evaluation unit 150 may calculate a conciseness score based on the number of characters or sentences in the pre-conversion data and the number of characters or sentences in the post-conversion data. The dialogue evaluation unit 150 may also calculate a conciseness score based on the number of characters or sentences in the text data representing the content of the user utterance in the pre-conversion data and the post-conversion data. As an example, the dialogue evaluation unit 150 may calculate a conciseness score by dividing the number of characters (or sentences) of the user utterance in the summarized data (an example of post-conversion data) by the number of characters (or sentences) of the user utterance in the pre-summarized data (an example of pre-conversion data). As another example, the number of words may be used to calculate the conciseness score.

[0145] If the model input information includes audio data, the dialogue evaluation unit 150 may calculate a conciseness score based on the duration of the pre-conversion data and the duration of the post-conversion data. The dialogue evaluation unit 150 may also calculate a conciseness score based on the duration of the audio data representing user utterances in the pre-conversion data and the post-conversion data. As an example, the dialogue evaluation unit 150 may calculate a conciseness score by dividing the duration of user utterances in the summarized data by the duration of user utterances in dialogue data D.

[0146] The dialogue evaluation unit 150 may calculate a conciseness score based on a machine learning model. The dialogue evaluation unit 150 may generate a conciseness score based on the pre-transformation data, the post-transformation data, and the machine learning model. The dialogue evaluation unit 150 may calculate a conciseness score based on the pre-transformation data and the machine learning model. The dialogue evaluation unit 150 may generate post-transformation data based on the pre-transformation data and the machine learning model, and calculate a conciseness score based on the pre-transformation data and the portion of the post-transformation data that represents user utterances. As an example, the machine learning model may be a model trained to calculate a conciseness score when pre-transformation data and post-transformation data are input. As another example, the machine learning model may be a model trained to calculate a conciseness score when pre-transformation data is input. The machine learning model may be machine learning model M, or it may be another machine learning model different from machine learning model M. Other machine learning models may, as an example, be a large-scale language model, a generative model, or a foundational model.

[0147] Figure 5 shows an example of a prompt that instructs the summarization of dialogue data. As shown in Figure 5, the prompt 400 includes instruction information 401 that instructs summarization, instruction information 402 that instructs the output format, instruction information 403 that instructs the addition of speaker information, and instruction information 404 that instructs outputting the summary result. The prompt 400 also includes the dialogue data 405 to be summarized. The dialogue data 405 may be converted to the output format instructed by instruction information 402 and then embedded in the prompt 400.

[0148] When calculating a score related to logicality, the dialogue evaluation unit 150 may be configured as follows. Hereinafter, the score related to logicality will also be referred to as the "logicality score". The model input information may include instruction information that instructs the determination of the logicality of the data before conversion. The instruction information that instructs the determination of logicality may include instruction information that instructs the determination of whether or not the data before conversion falls under each of the multiple evaluation axes. The multiple evaluation axes may include evaluation axes that define logicality. Examples of evaluation axes that define logicality may include consistency, objectivity, accuracy, structure, and integrity.

[0149] The model input information may include samples for each of the multiple evaluation axes. As an example, the evaluation axis samples may include at least one of the utterances that fall under that evaluation axis, or utterances that do not fall under that evaluation axis. The model input information may also include system prompts used in the dialogue.

[0150] Model input information may include data indicating the content of utterances in one or more turns included in the pre-conversion data. Model input information may include a combination of data indicating the content of system utterances and data indicating the content of user utterances in response to those system utterances. Model input information may be generated for each of the one or more turns included in the pre-conversion data. Model output information may include converted data containing truth values ​​indicating whether or not a turn falls under multiple evaluation axes.

[0151] The dialogue evaluation unit 150 may calculate a logical score based on the number of times each of the multiple evaluation axes is determined to be true. For example, the dialogue evaluation unit 150 may obtain converted data for a predetermined number of turns, determining whether each of the multiple evaluation axes applies, and calculate a logical score by dividing the number of evaluation axes determined to be true in the converted data by the number of turns.

[0152] The dialogue evaluation unit 150 may correct the logicality score. For example, the dialogue evaluation unit 150 may add to the logicality score based on a calculation formula that includes a predetermined correction term. As another example, the dialogue evaluation unit 150 may set at least one of an upper limit or a lower limit so that the logicality score falls within a predetermined range.

[0153] The dialogue evaluation unit 150 may calculate a logical score based on a machine learning model. The dialogue evaluation unit 150 may generate a logical score based on the pre-transformation data, the post-transformation data, and the machine learning model. The dialogue evaluation unit 150 may calculate a logical score based on the pre-transformation data and the machine learning model. The dialogue evaluation unit 150 may generate post-transformation data based on the pre-transformation data and the machine learning model, and calculate a logical score based on the pre-transformation data and the post-transformation data. As an example, the machine learning model may be a model that has been trained to calculate a logical score when pre-transformation data and post-transformation data are input. As another example, the machine learning model may be a model that has been trained to calculate a logical score when pre-transformation data is input. The machine learning model may be machine learning model M, or it may be another machine learning model different from machine learning model M. Other machine learning models may, as an example, be a large-scale language model, a generative model, or a foundational model.

[0154] Figure 6 shows an example of a prompt that instructs a determination of logical consistency. As shown in Figure 6, prompt 410 includes instruction information 411 that instructs an evaluation of logical consistency, instruction information 412 (412-1 to 412-5) that instructs a determination of whether or not it falls under the evaluation axis, instruction information 413 (413-1 to 413-5) that instructs the output of the reason why it falls under or does not fall under the evaluation axis, and instruction information 414 that instructs the output format. Prompt 410 also includes dialogue data 415 to be evaluated. The dialogue data 415 may be converted to the output format instructed by instruction information 414 and then embedded in prompt 410.

[0155] The dialogue evaluation unit 150 may calculate a score multiple times for a single dialogue data D. The dialogue evaluation unit 150 may calculate the final score using statistical values ​​(e.g., mean, median, maximum, minimum, variance, standard deviation, etc.) of the multiple scores calculated multiple times. The dialogue evaluation unit 150 may calculate a score multiple times using different machine learning models. The dialogue evaluation unit 150 may calculate a score multiple times while changing parameters related to the stability (variance) of the machine learning model. If the machine learning model is a language model with a temperature parameter, the dialogue evaluation unit 150 may calculate a score multiple times while changing the temperature parameter.

[0156] The dialogue evaluation unit 150 may generate evaluation results for the dialogue participant S at any time. The dialogue evaluation unit 150 may generate evaluation results in batch processing or in ad-hoc processing. For example, the dialogue evaluation unit 150 may generate evaluation results for each of the accumulated dialogue data D at predetermined time intervals. The dialogue evaluation unit 150 may also generate evaluation results for the dialogue data D related to a dialogue when the dialogue has ended.

[0157] The dialogue evaluation unit 150 may store the evaluation results regarding the dialogue participant S in the evaluation result storage unit 103. The dialogue evaluation unit 150 may store the evaluation results in the evaluation result storage unit 103 in association with identification information that can identify the dialogue. For example, the dialogue evaluation unit 150 may store the evaluation results in the evaluation result storage unit 103 in association with time information.

[0158] The dialogue evaluation unit 150 may generate an evaluation result regarding the dialogue participant S based on the scenario information acquired by the scenario acquisition unit 110. For example, the dialogue evaluation unit 150 may embed at least a portion of the scenario information into the model input information that is input to the machine learning model to calculate the score.

[0159] The scenario information that the dialogue evaluation unit 150 embeds into the model input information may be the same as the scenario information acquired by the scenario acquisition unit 110. At least some of the information in the scenario information that the dialogue evaluation unit 150 embeds into the model input information may overlap with the scenario information acquired by the scenario acquisition unit 110. For example, the scenario information that the dialogue evaluation unit 150 embeds into the model input information may be the same as the scenario information that the scenario acquisition unit 110 used to generate the scenario information. This allows the dialogue evaluation unit 150 to have the machine learning model calculate a score that takes into account the purpose of the dialogue indicated in the scenario information (for example, a cafe manager wants to consult about his sales problems, or a customer who has come to the store wants to buy a watch that suits him). In addition, at least some of the model input information in the dialogue evaluation unit 150 and information about the score calculated by the dialogue evaluation unit 150 may be included in the scenario information in the scenario acquisition unit 110. This allows the dialogue system 1000 to perform a dialogue that takes into account the evaluation in the dialogue evaluation unit 150.

[0160] (Result output section) The result output unit 160 outputs the evaluation results for the interlocutor S. The result output unit 160 may also output the evaluation results generated by the dialogue evaluation unit 150. The result output unit 160 may also output the evaluation results read from the evaluation result storage unit 103.

[0161] The result output unit 160 may output external evaluation values ​​for the interlocutor S along with the evaluation results for the interlocutor S. The result output unit 160 may read the external evaluation values ​​for the interlocutor S stored in the evaluation result storage unit 103 along with the evaluation results for the interlocutor S, and include the external evaluation values ​​in the evaluation results for the interlocutor S.

[0162] The result output unit 160 may include the dialogue data D used to evaluate the dialogue participant S in the evaluation results for the dialogue participant S. For example, the result output unit 160 may include the recorded data contained in the dialogue data D, or the features extracted from the recorded data, in the evaluation results for the dialogue participant S.

[0163] The result output unit 160 may transmit the evaluation results regarding the interlocutor S to the terminal device 40. The result output unit 160 may transmit the evaluation results regarding the interlocutor S to the terminal device 40 in response to an output request from the terminal device 40. If there are multiple evaluation results regarding the interlocutor S, the result output unit 160 may transmit the trend of the evaluation values ​​or statistical information shown in the evaluation results to the terminal device 40. The terminal device 40 may present the evaluation results regarding the interlocutor S to the evaluator E.

[0164] The result output unit 160 may transmit the evaluation results regarding the interlocutor S to the dialogue device 10. The dialogue device 10 may present the evaluation results regarding the interlocutor S to the interlocutor S. The result output unit 160 may transmit the evaluation results regarding the interlocutor S to an external information processing device or information processing system. The external information processing device or information processing system may present the evaluation results regarding the interlocutor S to users other than the interlocutor S. Users other than the interlocutor S may include the evaluator E.

[0165] The result output unit 160 may, for example, send screen data displaying the evaluation results to the terminal device 40. The screen data may, for example, be electronic data written in HTML (Hyper Text Markup Language) or the like. The result output unit 160 may also send screen data displaying the evaluation results to the dialogue device 10.

[0166] The result output unit 160 may, for example, send the evaluation results to the account of evaluator E or interlocutor S via email or a messaging service. The result output unit 160 may also send a link to a screen displaying the evaluation results to the account of evaluator E or interlocutor S via email or a messaging service.

[0167] The functional configuration of the control device 20 shown in Figure 2 is just one example, and it goes without saying that there are various functional configurations depending on the application and purpose. The division of memory units, such as the scenario memory unit 101, dialogue memory unit 102, and evaluation result memory unit 103 shown in Figure 2, is just one example. The division of processing units, such as the scenario acquisition unit 110, utterance acquisition unit 120, utterance generation unit 130, utterance presentation unit 140, dialogue evaluation unit 150, and result output unit 160 shown in Figure 2, is just one example.

[0168] For example, at least two of the scenario memory unit 101, the dialogue memory unit 102, or the evaluation result memory unit 103 may be integrated into a single memory unit. Alternatively, at least one of the scenario memory unit 101, the dialogue memory unit 102, or the evaluation result memory unit 103 may be divided into multiple memory units.

[0169] For example, at least two of the scenario acquisition unit 110, utterance acquisition unit 120, utterance generation unit 130, utterance presentation unit 140, dialogue evaluation unit 150, or result output unit 160 may be integrated into one processing unit. Alternatively, at least one of the scenario acquisition unit 110, utterance acquisition unit 120, utterance generation unit 130, utterance presentation unit 140, dialogue evaluation unit 150, or result output unit 160 may be divided into multiple processing units.

[0170] <Flow of the dialogue method> The dialogue method performed by the dialogue system 1000 will be explained with reference to Figure 7. Figure 7 is a processing flow showing an example of a dialogue method. Below, an example will be described in which a CG character interacts with a dialoguer S who is operating a personal computer, which is an example of a dialogue device 10.

[0171] In step S101, the interlocutor S performs an operation to initiate a dialogue with the dialogue device 10. For example, the interlocutor S may select a control device 20 to engage in dialogue with from a list of control devices 20 displayed on the display device of the dialogue device 10. When the interlocutor S selects a control device 20, the dialogue device 10 may perform a process to connect to the control device 20.

[0172] The interlocutor S may perform a test of the microphone or camera of the dialogue device 10. The interlocutor S may turn on the microphone switch of the dialogue device 10 and check the microphone volume level. At this time, the dialogue device 10 and the control device 20 may cooperate to instruct the interlocutor S to read a specific sentence aloud, the interlocutor S reads the specified sentence aloud, the spoken sentence is recognized by speech recognition, and if the recognized text is the same as or similar to the instructed sentence, the control device 20 may determine that the microphone is working properly. The interlocutor S may turn on the camera switch of the dialogue device 10 and check the camera image displayed on the display device. At this time, the dialogue device 10 may use a face or body detector to determine whether the interlocutor S is properly captured in the camera image. The dialogue device 10 may control not to start the dialogue until the interlocutor S is properly captured in the camera image. Properly capturing the interlocutor S in the camera image may include, for example, the face being larger than a predetermined size, the face facing forward, and the body being within a predetermined range.

[0173] In step S102, the control device 20 starts recording the dialogue between the interlocutor S and the dialogue device 10. Specifically, the control device 20 requests the speech acquisition unit 120 to store user utterance information in the dialogue memory unit 102. The control device 20 also requests the speech presentation unit 140 to store system utterance information in the dialogue memory unit 102. When the speech acquisition unit 120 acquires user utterance information, it enables the function to store the user utterance information in the dialogue memory unit 102. When the speech presentation unit 140 outputs system utterance information, it enables the function to store the system utterance information in the dialogue memory unit 102.

[0174] In step S103, the speech presentation unit 140 of the control device 20 acquires voice guidance information set by the scenario acquisition unit 110. The speech presentation unit 140 synthesizes the voice guidance information into speech. This generates a voice signal indicating an explanatory utterance. The speech presentation unit 140 transmits the voice signal indicating the explanatory utterance to the dialogue device 10. The speech presentation unit 140 may transmit the voice guidance information to the dialogue device 10 together with the voice signal or by itself. The speech presentation unit 140 may also transmit display guidance information to the dialogue device 10 together with the voice signal or by itself. In other words, the speech presentation unit 140 notifies the dialogue participant S of the guidance information. The notification of guidance information may be done by any means such as voice, image, or text. The speech presentation unit 140 also notifies the dialogue participant S of the first guidance information before the execution of the first dialogue. The speech presentation unit 140 notifies the dialogue participant S of the second guidance information before the execution of the second dialogue.

[0175] In step S104, the dialogue device 10 receives an audio signal indicating an explanatory utterance from the control device 20. Based on the audio signal indicating the explanatory utterance, the dialogue device 10 outputs an explanatory utterance from the speaker. If the dialogue device 10 receives audio guidance information from the control device 20, it may output an audio signal synthesized from the audio guidance information from the speaker. Also, if the dialogue device 10 receives display guidance information from the control device 20, it may display the display guidance information on the display device.

[0176] In step S105, the dialogue device 10 displays a CG character on the display device. The dialogue device 10 transmits the audio signal picked up by the microphone and the video signal captured by the camera to the control device 20. Thereafter, the dialogue device 10 continues to transmit the audio signal and video signal until the dialogue ends.

[0177] The interlocutor S speaks in response to the explanatory utterance output in step S104. The dialogue device 10 receives the user utterance from the interlocutor S. The audio signal that the dialogue device 10 transmits to the control device 20 includes the audio of the user utterance spoken by the interlocutor S. The video signal that the dialogue device 10 transmits to the control device 20 includes an image of the interlocutor S's facial expression or actions at the time the user utterance was spoken.

[0178] In step S106, the speech acquisition unit 120 of the control device 20 receives an audio signal from the dialogue device 10 that captures user utterances. The speech acquisition unit 120 also receives a video signal from the dialogue device 10 that captures the dialoguer S.

[0179] The speech acquisition unit 120 performs speech recognition on the audio signal obtained from the user's utterance. This generates text data representing the content of the user's utterance. The speech acquisition unit 120 sends the text data representing the content of the user's utterance to the speech generation unit 130.

[0180] The speech acquisition unit 120 generates user speech information based on the audio signal obtained from the user's speech, the video signal obtained from capturing the dialogue subject S, and the text data indicating the content of the user's speech. The speech acquisition unit 120 stores the user speech information in the dialogue storage unit 102.

[0181] In step S107, the speech generation unit 130 of the control device 20 receives text data indicating the content of the user utterance from the speech acquisition unit 120. The speech generation unit 130 acquires the system prompt set by the scenario acquisition unit 110. The speech generation unit 130 reads the dialogue data D from the dialogue storage unit 102. Based on the text data indicating the content of the user utterance and the dialogue data D read from the dialogue storage unit 102, the speech generation unit 130 generates dialogue history information.

[0182] The speech generation unit 130 generates model input information based on system prompts and dialogue history information. The speech generation unit 130 may also generate model input information by embedding system prompts and dialogue history information into a predetermined template. The speech generation unit 130 transmits the model input information to the generation device 30.

[0183] The generation device 30 receives model input information from the control device 20. The generation device 30 inputs the received model input information into the machine learning model M. The machine learning model M performs a predetermined task based on the model input information and outputs the resulting model output information. The model output information may include at least text data indicating the content of the system utterance. The generation device 30 transmits the model output information to the control device 20.

[0184] In step S108, the speech generation unit 130 of the control device 20 receives model output information from the generation device 30. The speech generation unit 130 generates system speech information based on at least a portion of the received model output information. The speech generation unit 130 may also synthesize text data indicating the content of the system speech into speech. This generates an audio signal indicating the system speech. The audio signal indicating the system speech may be included in the model output information.

[0185] The speech generation unit 130 may generate a video signal representing a CG character. The speech generation unit 130 may synthesize a video signal representing a CG character based on information included in the model output information. The video signal representing a CG character may be included in the model output information. In other words, the video signal representing a CG character may be directly generated by the machine learning model M.

[0186] The speech generation unit 130 sends system speech information to the speech presentation unit 140. The speech presentation unit 140 receives system speech information from the speech generation unit 130. The speech presentation unit 140 stores the system speech information in the dialogue memory unit 102. The speech presentation unit 140 transmits the system speech information to the dialogue device 10.

[0187] In step S109, the dialogue device 10 receives system utterance information from the control device 20. Based on the system utterance information, the dialogue device 10 utters a system utterance. The dialogue device 10 may output sound indicating the system utterance from its speaker. The dialogue device 10 may display an image representing a CG character on a display device. If the dialogue device 10 is a robot, it may control the robot based on the system utterance information. The dialogue device 10 may turn off the microphone switch while uttering a system utterance. In other words, the dialogue device 10 may record the utterance of the interlocutor S with the microphone until it utters a system utterance.

[0188] In step S110, the control device 20 determines whether the termination conditions for ending the dialogue have been met. The termination conditions may include, for example, the number of turns in the dialogue or the elapsed time since the start of the dialogue exceeding a predetermined threshold. In Figure 7, the termination determination process is placed only immediately after step S108, but if the determination is based on elapsed time, the termination determination process may be performed at any timing from immediately before step S106 to immediately after step S108. In this case, the termination determination may not be performed until a predetermined time has elapsed from the start to the end of each utterance by the dialogue participant S. By doing so, it is possible to end the dialogue at a time close to the predetermined time while preventing the conversation from ending abruptly.

[0189] The control device 20 may, at any timing between immediately before step S106 and immediately after step S108, use a face or body detector to check whether the face or body of the person speaking S is visible in the camera of the dialogue device 10. If the face or body cannot be detected for a certain period of time, the dialogue is terminated and the process proceeds to step S111. Before the end of the dialogue, the dialogue device 10 may output a message to the person speaking S in text or voice indicating that the face or body could not be detected.

[0190] The control device 20 may determine whether or not to terminate the dialogue based on the content of the dialogue. For example, the control device 20 may input dialogue history information into a machine learning model and determine whether or not to terminate the dialogue based on the model output information output from the machine learning model. The machine learning model that determines the termination of the dialogue may be machine learning model M, or it may be another machine learning model different from machine learning model M. Other machine learning models may include, for example, a large-scale language model, a generative model, or a foundational model.

[0191] If the control device 20 determines that the termination condition has been met (YES), it proceeds to step S111. If the control device 20 determines that the termination condition has been met, it may transmit an audio signal indicating a predetermined utterance to the dialogue device 10. The predetermined utterance may be an utterance indicating that the dialogue has ended. The dialogue device 10 may receive an audio signal indicating a predetermined utterance from the control device 20 and output the predetermined utterance from its speaker.

[0192] On the other hand, if it is determined that the termination condition is not met (NO), the dialogue device 10 returns to step S105, and the control device 20 returns to step S106. When the control device 20 returns to step S106, the speech acquisition unit 120 of the control device 20 acquires user utterance information regarding the next user utterance obtained by the dialogue device 10 in step S105. After that, the control device 20 executes the processes from step S106 to step S110 again based on the new user utterance information. In this way, the dialogue system 1000 repeatedly receives user utterances and presents system utterances until it is determined in step S110 that the termination condition has been met. The speech generation unit 130 may change the system prompt used to generate system utterances each time the process is repeated.

[0193] In step S111, the control device 20 terminates recording of the dialogue between the interlocutor S and the dialogue device 10. Specifically, the control device 20 requests the speech acquisition unit 120 to stop storing user utterance information. The control device 20 also requests the speech presentation unit 140 to stop storing system utterance information. The speech acquisition unit 120 disables the function of storing user utterance information in the dialogue storage unit 102. The speech presentation unit 140 disables the function of storing system utterance information in the dialogue storage unit 102. As a result, dialogue data D is stored in the dialogue storage unit 102.

[0194] <Scenario Control Method Flow> The scenario control method executed by the dialogue system 1000 will be explained with reference to Figure 8. Figure 8 is a processing flow showing an example of the scenario control method. The scenario control method is a method for controlling the scenario information used in the dialogue method shown in Figure 7.

[0195] In step S1, the scenario acquisition unit 110 of the control device 20 acquires first scenario information to be used for the first dialogue between the interlocutor S and the dialogue device 10. The scenario acquisition unit 110 may acquire predetermined scenario information as the first scenario information. The scenario acquisition unit 110 may select the first scenario information from the scenario information stored in the scenario storage unit 101. The scenario acquisition unit 110 may generate the first scenario information based on the template information and text information stored in the scenario storage unit 101. The scenario acquisition unit 110 may generate the first scenario information by embedding the first text contained in the text information into the placeholder of the template information.

[0196] In step S2, the control device 20 performs a first dialogue between the interlocutor S and the dialogue device 10. Based on the first scenario information acquired in step S1, the control device 20 performs the first dialogue by executing the dialogue method shown in Figure 7. That is, the control device 20 performs the first dialogue using at least the system prompts and machine learning model M included in the first scenario information. While performing the first dialogue, the control device 20 stores the dialogue data D related to the first dialogue in the dialogue storage unit 102.

[0197] In step S2, the first dialogue based on the first scenario information does not necessarily have to be completed. For example, even if the first dialogue is interrupted or terminated before completion, such as when the connection between the dialogue device 10 and the control device 20 is disconnected, the process may proceed from step S2 to step S3.

[0198] In step S3, the scenario acquisition unit 110 of the control device 20 acquires second scenario information to be used for the second dialogue with the interlocutor S. The scenario acquisition unit 110 acquires the second scenario information based on the first scenario information in step S1. The second scenario information differs from the first scenario information in at least some respects. In addition, the second scenario information shares at least some respects with the first scenario information regarding dialogue control.

[0199] The scenario acquisition unit 110 may select a second scenario information different from the first scenario information from the scenario information stored in the scenario storage unit 101. The scenario acquisition unit 110 may generate the second scenario information based on the template information used to generate the first scenario information and a second text different from the first text used to generate the first scenario information. The scenario acquisition unit 110 may generate the second scenario information by embedding the second text contained in the text information into the placeholder of the template information.

[0200] In step S4, the control device 20 performs a second dialogue with the interlocutor S. Based on the second scenario information acquired in step S3, the control device 20 performs the second dialogue by executing the dialogue method shown in Figure 7. That is, the control device 20 performs the second dialogue using at least the system prompts and machine learning model M included in the second scenario information. While performing the second dialogue, the control device 20 stores the dialogue data D related to the second dialogue in the dialogue storage unit 102.

[0201] Step S3 may be executed at any time before step S4. For example, step S3 may be executed before step S2. For example, step S3 may be executed immediately after step S1. For example, step S3 may be executed before step S1. For example, steps S1 and S3 may be executed simultaneously or in parallel. For example, step S3 may be executed immediately before step S4.

[0202] <Evaluation Method Flow> The evaluation method performed by the dialogue system 1000 will be explained with reference to Figure 9. Figure 9 is a processing flow showing an example of a dialogue method. Below, an example of evaluating the conciseness score, which is an example of the characteristics of the utterance content, will be explained.

[0203] In step S11, evaluator E performs an operation to start the evaluation on the terminal device 40. For example, evaluator E may select the dialogue participant S to be evaluated from the list of dialogue participants S displayed on the display device of the dialogue device 10. If dialogue participant S has had multiple conversations, evaluator E may select the dialogue data to be evaluated from the dialogue data D that records the conversations with dialogue participant S.

[0204] The terminal device 40 transmits an evaluation request to the control device 20 in response to an operation by the evaluator E. The evaluation request includes information indicating the dialogue subject S or dialogue data D to be evaluated. The dialogue evaluation unit 150 of the control device 20 reads the dialogue data D to be evaluated from the dialogue storage unit 102 in response to the evaluation request from the terminal device 40.

[0205] In step S12, the dialogue evaluation unit 150 of the control device 20 generates model input information for evaluating conciseness based on the dialogue data D. The model input information may include instruction information that instructs the summarization of the dialogue data D. It may include both text data showing the content of user utterances and text data showing the content of system utterances. The model input information may include only text data showing the content of user utterances. The model input information may include system prompts used in the dialogue. The dialogue evaluation unit 150 transmits the model input information to the generation device 30.

[0206] The generation device 30 receives model input information from the control device 20. The generation device 30 inputs the received model input information into the machine learning model M. The machine learning model M performs a predetermined task based on the model input information and outputs the resulting model output information. The model output information may include at least converted data that summarizes the dialogue data D. The generation device 30 transmits the model output information to the control device 20.

[0207] In step S13, the interactive evaluation unit 150 of the control device 20 receives model output information from the generation device 30. The interactive evaluation unit 150 obtains converted data from the received model output information.

[0208] The dialogue evaluation unit 150 obtains data indicating the content of the user utterance from the dialogue data D acquired in step S11. The dialogue evaluation unit 150 obtains data summarizing the content of the user utterance from the converted data. The dialogue evaluation unit 150 calculates a conciseness score based on the data obtained from the dialogue data D and the data obtained from the converted data. As an example, the dialogue evaluation unit 150 divides the number of characters (or sentences) of the user utterance in the dialogue data D by the number of characters (or sentences) of the user utterance in the converted data. This allows the dialogue evaluation unit 150 to obtain a conciseness score. The dialogue evaluation unit 150 stores the conciseness score in the evaluation result storage unit 103.

[0209] In steps S12 and S13, the dialogue evaluation unit 150 may calculate scores that evaluate other characteristics. For example, these other characteristics may include logic or responsiveness. Also, in steps S12 and S13, the dialogue evaluation unit 150 may calculate multiple scores that evaluate each of the multiple characteristics.

[0210] In step S14, the result output unit 160 of the control device 20 reads the evaluation result for the interlocutor S from the evaluation result storage unit 103. The evaluation result may include at least the score calculated in step S13. The result output unit 160 transmits the evaluation result for the interlocutor S to the terminal device 40. The terminal device 40 presents the evaluation result for the interlocutor S to the evaluator E. The result output unit 160 may also transmit the evaluation result for the interlocutor S to the dialogue device 10. The dialogue device 10 may present the evaluation result for the interlocutor S to the interlocutor S.

[0211] [Example 1] The control device 20 may execute the first and second dialogues based on the same system prompt. In this case, the control device 20 may configure the machine learning model M such that the flow of the first dialogue and the flow of the second dialogue are different. In this modified example, the scenario information may include configuration information about the machine learning model M as information related to dialogue control.

[0212] For example, the control device 20 may use different machine learning models for the first dialogue and the second dialogue. In this case, the scenario information may include setting information indicating the machine learning model to be used. The first scenario information and the second scenario information may include setting information indicating different machine learning models.

[0213] For example, the control device 20 may have different settings information regarding the randomness of the output of the machine learning model M when used for the first dialogue and when used for the second dialogue. The settings information regarding randomness may include, as an example, a temperature parameter. The temperature parameter is a parameter in which the randomness of the output of the machine learning model M increases when the value is high, and decreases when the value is low. In this case, the scenario information may also include settings information regarding randomness. The values ​​of the settings information regarding randomness may be different for the first scenario information and the second scenario information.

[0214] According to this modified version, even using the same system prompt, the flow of the first dialogue and the flow of the second dialogue can be configured to be different. According to this modified version, because the same system prompt is used, the dialogue with each of multiple test takers can be compared and tested. Furthermore, according to this modified version, since the flow of the dialogue changes with each dialogue, the influence of the number of times a test taker has taken the test on the evaluation results can be reduced, and fairness among test takers can be improved.

[0215] [Differentiation 2] The control device 20 may store information used to control the dialogue device 10. This information may include, for example, text data used to synthesize voice signals representing system utterances, the start and end times of the synthesized voice signals, labels used to control the dialogue device 10's facial expressions or bodily movements (e.g., smiling, waving, nodding, etc.), and the start and end times of videos showing the actions based on those labels. The time may be in Japan Standard Time (JST), Coordinated Universal Time (UTC), or the elapsed time since the start of recording the dialogue.

[0216] The information used to control the dialogue device 10 may be stored in the memory device of the dialogue device 10. The information used to control the dialogue device 10 may be stored in the memory device of the generation device 30. The information used to control the dialogue device 10 may be stored in an information processing device or memory device other than the dialogue device 10, the control device 20, or the generation device 30.

[0217] The information used to control the dialogue device 10 may be used together with the audio signal or video signal to calculate evaluation values. For example, by using the information used to synthesize the audio signal, the speech section of the dialogue device 10 can be identified from the recorded data of the entire dialogue based on the start and end times of playback of the audio signal. Also, by using the information used to synthesize the audio signal, the content of the dialogue device 10's speech can be obtained as text without using speech recognition. For example, by using information about the dialogue device 10's facial expressions or body movements, additional information such as what the dialoguer S responded to those facial expressions or body movements, what facial expressions they made, and what tone of voice they used can be used for evaluation.

[0218] [Difference 3] The speech generation unit 130 may generate system utterances using a technique called Retrieval Augmented Generation (RAG). Retrieval Augmented Generation is a technique that includes reference information obtained by searching external data sources in the prompt in order to obtain good generation results from a machine learning model.

[0219] The control device 20 may pre-store reference information, including background information related to the dialogue device 10. The speech generation unit 130 may generate a query based on user utterance information and search for reference information based on the query. The speech generation unit 130 may generate a system prompt based on the retrieved reference information. The speech generation unit 130 may include the generated system prompt in the model input information and transmit the model input information to the generation device 30.

[0220] As an example, we will describe an example of performing a dialogue in a scenario where the interlocutor S is a doctor and the dialogue device 10 is the patient. The reference information includes the circumstances under which the symptoms occurred, current symptoms, medical history, chronic illnesses, medication history, vital signs, facial expressions, and, if there is an injury, text describing the details. The speech generation unit 130 generates a query based on the question from the interlocutor S and retrieves the reference information related to the question from the interlocutor S based on the query. The speech generation unit 130 generates a system prompt that includes the retrieved reference information and generates a system utterance that responds to the question from the interlocutor S based on the machine learning model M. As a result, the dialogue device 10 can respond appropriately to the question from the interlocutor S.

[0221] For the sake of simplicity, this explanation assumes that the reference information is text-based. However, the reference information is not limited to text; it may also include audio or image information, etc.

[0222] [Differentiation Example 4] In each of the above embodiments, examples were described in which a dialogue is performed between a dialogue participant S and a dialogue device 10 using voice information and image information. The above embodiments are not limited to dialogues using voice information and image information, and may also be applied to dialogues using text information, for example. Dialogues using text information may include, for example, text chat, email, messaging services, etc.

[0223] [Other embodiments] The dialogue system 1000 according to this embodiment can be used for dialogue testing using scenarios. Here, the dialogue system 1000 is not limited to actual examinations or tests where pass or fail is determined in a single attempt, but may also be applied to simulated examinations for actual examinations, training or workshops to acquire dialogue skills, etc. In this case, the history information of the dialogue participant S may include scenarios used in simulated examinations, training or workshops to acquire dialogue skills, etc. When the dialogue system 1000 is used for actual examinations or tests, a different scenario may be selected based on the history information including these. For example, the same template information may be used in common for actual examinations or tests and for simulated examinations, training or workshops to acquire dialogue skills, etc., but the text embedded in the placeholders may be different. Conversely, in order to confirm that it can be done as practiced, a scenario may be randomly selected from those used in simulated examinations, training or workshops to acquire dialogue skills, etc., and used for the actual examination or test. If the same template information is used, the text information embedded in the placeholders used in the actual examination or test may consist only of text used in simulated examinations, training or workshops to acquire conversational skills, etc.

[0224] The first and second dialogues may be conducted as dialogue tests with different purposes. For example, the first dialogue may be conducted during the training phase, and the second dialogue during the testing phase. Another example is that the first dialogue may be conducted during the training phase, and the second dialogue during the actual test.

[0225] In this embodiment, an example was described in which the first scenario information and the second scenario information have at least some information in common. However, the first scenario information and the second scenario information may be configured so that they have no common information (i.e., the first scenario information and the second scenario information are completely different). Alternatively, the first scenario information and the second scenario information may be configured so that the circumstances (background information) that serve as prerequisites for the dialogue are different.

[0226] In this embodiment, an example was described in which the same CG character is used in the first dialogue and the second dialogue. However, the CG character may be changed between the first dialogue and the second dialogue. For example, the first CG character used in the first dialogue and the second CG character used in the second dialogue may be configured to be different. Alternatively, for example, the same CG character may be used in the first dialogue and the second dialogue, but the characteristics of the CG character (e.g., voice quality, facial expression, or emotion) may be different.

[0227] The evaluation results according to this embodiment can be used to evaluate test takers in dialogue tests. However, the evaluation results may be used for other purposes. For example, the evaluation results may be used as feedback information in training or workshops. They can also be applied to business communication tools, for example, by evaluating the conciseness of speech during meetings. Furthermore, they can be applied to, for example, evaluating students' presentation skills in educational support systems, or managing the dialogue quality of agents in customer support.

[0228] The dialogue system 1000 according to this embodiment is not limited to dialogue in a specific language. In calculating the conciseness score, the score may be defined in processing units according to the language (for example, in the case of English, the conciseness score may be calculated in units of word count). The characteristics of the dialogue content to be evaluated may be selected according to the attributes of the dialogue participants (e.g., age or occupation). When the evaluation results are used as feedback information in training or workshops, the feedback of the evaluation results may be given in real time during the dialogue.

[0229] In this embodiment, an example was described in which the dialogue system 1000 performs a first dialogue and a second dialogue based on scenario information and a machine learning model. Here, performing a dialogue based on scenario information and a machine learning model may include inputting at least some of the information contained in the scenario information into the machine learning model to perform the dialogue. The scenario information may include information that is not used to perform the dialogue using the machine learning model.

[0230] In this embodiment, an example was described in which the dialogue system 1000 executes a first dialogue and a second dialogue based on the same machine learning model M. However, the machine learning model used to execute the first dialogue and the machine learning model used to execute the second dialogue may be configured to be different. In this case, the first scenario information and the second scenario information may be different.

[0231] In this embodiment, an example was described in which the dialogue system 1000 acquires second scenario information based on first scenario information. Here, acquiring second scenario information based on first scenario information may include acquiring second scenario information based on the acquisition result of first scenario information, acquiring second scenario information based on the situation in which a first dialogue was performed using the first scenario information, and acquiring second scenario information based on the selection result of the first scenario information.

[0232] In this embodiment, an example was described in which the dialogue system 1000 acquires second scenario information based on the history information of the dialogue participant S. Here, acquiring second scenario information based on the history information of the dialogue participant S may also include acquiring second scenario information based on first scenario information.

[0233] Obtaining second scenario information based on first scenario information may include any of the following: - Randomly select a second scenario from multiple scenario options. If the second scenario matches the first scenario, select the second scenario again. - Randomly select a second text to embed in the template information. If the second text matches the first text used to generate the first scenario information, select the second text again. - If the first scenario information consists of a system prompt template and the first text, and the second scenario information consists of the same system prompt template and the second text, the first text and the second text must be different. The means by which the first text and the second text are different may be arbitrary. For example, this may include selecting the first text and the second text based on different seed information.

[0234] The history information of the interlocutor S may include the history of dialogues performed using the dialogue system 1000. The dialogue execution history may include, for example, the fact that a dialogue based on the first scenario information was performed, or that an attempt was made to perform a dialogue based on the first scenario information. The dialogue execution history may also include, for example, the number of times the dialogue has been performed. For example, the dialogue system 1000 may determine which scenario information to use depending on the number of times the dialogue has been performed. As an example, the dialogue system 1000 may use the first scenario information for the first dialogue with interlocutor S, and use the second scenario information for the second dialogue with interlocutor S.

[0235] In this embodiment, an example was described in which the dialogue system 1000 obtains scenario information by generating scenario information. Here, generating scenario information may also include generating input information to be input to a machine learning model in order to perform the dialogue.

[0236] In this embodiment, an example was described in which the dialogue system 1000 acquires second scenario information, which is different from the first scenario information, to be used for a second dialogue with the dialogue participant S, based on the first scenario information. Here, acquiring second scenario information, which is different from the first scenario information, to be used for a second dialogue with the dialogue participant S, based on the first scenario information, may include selecting the second scenario information from a set of multiple scenario information, excluding the first scenario information.

[0237] <Summary> As is clear from the above description, the dialogue system 1000 according to the first aspect of this disclosure acquires first scenario information to be used for a first dialogue with the dialogue participant S, executes the first dialogue with the dialogue participant S based on the first scenario information and a machine learning model M, acquires second scenario information different from the first scenario information to be used for a second dialogue with the dialogue participant S based on the first scenario information, executes the second dialogue with the dialogue participant S after the completion of the first dialogue based on the second scenario information and a machine learning model M, and the first scenario information and the second scenario information have at least some information in common regarding dialogue control.

[0238] The dialogue system 1000 may acquire second scenario information based on the history information of the interlocutor S. Based on the history information of the interlocutor S may include based on the first scenario information.

[0239] At least some of the information relating to dialogue control may include information indicating the relationship between the interlocutor S and the interlocutor, preconditions for the dialogue, or constraints on the utterance. At least some of the information relating to dialogue control may include system prompt templates. At least some of the information relating to dialogue control may include information indicating the relationship between the interlocutor S and the interlocutor.

[0240] The relationship between the interlocutor S and the interlocutor may include at least one of the following: an interlocutor S providing services or goods to the interlocutor; an interlocutor S instructing the interlocutor; an interlocutor paying remuneration to the interlocutor S; or an interlocutor S possessing information A providing information A to an interlocutor who does not possess information A. The relationship between the interlocutor S and the interlocutor may also include at least one of the following: an interlocutor S being the examinee and the interlocutor being the interviewer; an interlocutor S being the shop assistant and the interlocutor being the customer; an interlocutor S being the superior and the interlocutor being the subordinate; an interlocutor S being the teacher and the interlocutor being the student; an interlocutor S being the speaker and the interlocutor being the audience; an interlocutor S being the doctor and the interlocutor being the patient; or an interlocutor S being the person making the request and the interlocutor being the person making the request.

[0241] The dialogue system 1000 may generate first scenario information based on predetermined template information and one or more first words. The dialogue system 1000 may generate second scenario information based on predetermined template information and one or more second words different from the one or more first words.

[0242] The dialogue system 1000 may randomly select one or more first words from a predetermined set of words. The dialogue system 1000 may also randomly select one or more second words from the set of words excluding one or more first words.

[0243] The dialogue system 1000 may select a first scenario from a plurality of scenarios that share at least some information regarding dialogue control. The dialogue system 1000 may randomly select the first scenario from a plurality of scenarios. The dialogue system 1000 may select the first and second scenarios from a plurality of scenarios based on predetermined criteria. The dialogue system 1000 may select the second scenario from a set of scenarios excluding the first scenario from the plurality of scenarios.

[0244] The first scenario information may include at least a first system prompt and first guidance information. The second scenario information may include at least a second system prompt and second guidance information. The first system prompt may be generated by embedding a first word in a placeholder of a system prompt template. The first guidance information may be generated by embedding a first word in a placeholder of a guidance template. The second system prompt may be generated by embedding a second word different from the first word in a placeholder of a system prompt template. The second guidance information may be generated by embedding a second word in a placeholder of a guidance template. The dialogue system 1000 may notify the dialogue participant S of the first guidance information before executing the first dialogue. The dialogue system 1000 may notify the dialogue participant S of the second guidance information before executing the second dialogue.

[0245] The dialogue system 1000 may perform a first dialogue by inputting a system prompt to a machine learning model that operates based on first configuration information included in the first scenario information. The dialogue system 1000 may perform a second dialogue by inputting a system prompt to a machine learning model that operates based on second configuration information included in the second scenario information.

[0246] The dialogue system 1000 may perform a first dialogue by inputting a system prompt included in the first scenario information into a machine learning model. The dialogue system 1000 may perform a second dialogue by inputting a system prompt, information about the first dialogue, and information instructing the machine learning model to perform a dialogue different from the first dialogue. The second scenario information may include information instructing the model to perform a dialogue different from the first dialogue.

[0247] The dialogue system 1000 may acquire dialogue data that includes at least information about the first utterance spoken by the dialogue participant S in the second dialogue. The dialogue system 1000 may convert at least a portion of the dialogue data into predetermined data by inputting it into a second machine learning model. The dialogue system 1000 may calculate a predetermined score for the dialogue data based on the predetermined data. The machine learning model and the second machine learning model may be the same model.

[0248] The dialogue system 1000 according to a second aspect of this disclosure acquires dialogue data including information about a first utterance spoken by at least a first speaker, converts at least a portion of the dialogue data into predetermined data by inputting it into a machine learning model, and calculates a predetermined score for the dialogue data based on the predetermined data.

[0249] The dialogue data may include information about the second utterance spoken by the second speaker. The specified data may include data summarizing the dialogue data. The dialogue data and the specified data may include text data indicating the content of the utterances. The specified score may be calculated based on the amount of data in the dialogue data and the amount of data in the specified data. The specified score may include a score regarding the conciseness of the utterances spoken by the first speaker. The dialogue data and the specified data may include audio data indicating the utterances. The specified score may be calculated based on the amount of data in the dialogue data and the amount of data in the specified data.

[0250] The specified data may include truth values ​​that evaluate whether the dialogue data satisfies specified characteristics on one or more evaluation axes. The truth values ​​may be calculated by inputting the dialogue data into a machine learning model. The specified score may be calculated based on the number of evaluation axes and the number of characteristics that were evaluated as being satisfied. The specified score may include a score regarding the logic of the utterances made by the first speaker.

[0251] The dialogue system 1000 may calculate a predetermined score by inputting predetermined data into a machine learning model.

[0252] As a result, according to one embodiment of this disclosure, a device that interacts with a user can be controlled. In one aspect, according to this embodiment, since the interaction with the user is performed using scenario information in which at least some of the information regarding the interaction control is common, the interaction with each of multiple test takers can be examined in a comparative manner. In another aspect, according to this embodiment, since the first scenario information used for the first interaction and the second scenario information used for the second interaction are different, the influence of the number of times a test taker has taken the test on the evaluation result can be reduced, and fairness among test takers can be improved.

[0253] [Hardware configuration of information processing equipment] In the embodiments described above, some or all of the devices (interaction device 10, control device 20, generation device 30, and terminal device 40) may be composed of hardware, or they may be composed of information processing by software (programs) executed by a CPU (Central Processing Unit), GPU (Graphics Processing Unit), etc. If the information processing is composed of software, the software that realizes at least some of the functions of each device in the embodiments described above may be stored on a non-temporary storage medium (non-temporary computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or USB (Universal Serial Bus) memory, and the information processing of the software may be executed by loading it into a computer. Alternatively, the software may be downloaded via a communication network. Furthermore, all or part of the processing of the software may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), so that the information processing by the software is executed by hardware.

[0254] The storage medium for the software may be a removable medium such as an optical disc, or a fixed storage medium such as a hard disk or memory. Furthermore, the storage medium may be located inside the computer (main memory, auxiliary storage, etc.) or outside the computer.

[0255] Figure 10 is a block diagram showing an example of the hardware configuration of each device (dialogue device 10, control device 20, generation device 30, and terminal device 40) in the embodiment described above. Each device may be implemented as a computer 7, for example, comprising a processor 71, main memory 72 (memory), auxiliary memory 73 (memory), network interface 74, and device interface 75, which are connected via a bus 76.

[0256] The computer 7 in Figure 10 has one of each component, but it may have multiple identical components. Also, although Figure 10 shows one computer 7, the software may be installed on multiple computers, and each of these computers may execute the same or different parts of the software's processing. In this case, it may be a distributed computing configuration in which each computer communicates via a network interface 74 or the like to execute processing. In other words, each device in the above-described embodiment (dialogue device 10, control device 20, generation device 30, and terminal device 40) may be configured as a system that realizes its function by one or more computers executing instructions stored in one or more storage devices. Alternatively, it may be configured so that information transmitted from the terminal is processed by one or more computers located on the cloud, and the processing results are transmitted to the terminal.

[0257] The various calculations performed by each device (dialogue device 10, control device 20, generation device 30, and terminal device 40) in the embodiments described above may be performed in parallel using one or more processors, or using multiple computers via a network. Alternatively, the various calculations may be distributed to multiple processing cores within a processor and performed in parallel. Furthermore, some or all of the processing and means of this disclosure may be implemented by at least one of a processor and a storage device located on a cloud that can communicate with computer 7 via a network. Thus, each device in the embodiments described above may be in the form of parallel computing using one or more computers.

[0258] The processor 71 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that performs either control or calculations of a computer. The processor 71 may also be a general-purpose processor, a dedicated processing circuit designed to perform specific calculations, or a semiconductor device that includes both a general-purpose processor and a dedicated processing circuit. Furthermore, the processor 71 may include optical circuits or quantum computing-based calculation functions.

[0259] The processor 71 may perform calculations based on data and software input from various devices within the computer 7, and may output calculation results and control signals to these devices. The processor 71 may also control the various components of the computer 7 by executing the computer 7's OS (Operating System) or applications.

[0260] Each of the devices in the above-described embodiment (dialogue device 10, control device 20, generation device 30, and terminal device 40) may be implemented by one or more processors 71. Here, the processor 71 may refer to one or more electronic circuits arranged on one chip, or to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, each electronic circuit may communicate by wire or wireless.

[0261] The main memory 72 may store instructions executed by the processor 71 and various data, and the information stored in the main memory 72 may be read by the processor 71. The auxiliary storage device 73 is a storage device other than the main memory 72. These storage devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. In the above-described embodiment, the storage device for saving various data in each device (interaction device 10, control device 20, generation device 30, and terminal device 40) may be implemented by the main memory 72 or the auxiliary storage device 73, or by the built-in memory of the processor 71. For example, each storage unit in the above-described embodiment may be implemented by the main memory 72 or the auxiliary storage device 73.

[0262] In the embodiments described above, if each device (interaction device 10, control device 20, generation device 30, and terminal device 40) consists of at least one storage device (memory) and at least one processor connected to (coupled with) this at least one storage device, then at least one processor may be connected to one storage device. Also, at least one storage device may be connected to one processor. Furthermore, the configuration may include at least one processor among a plurality of processors being connected to at least one storage device among a plurality of storage devices. This configuration may also be realized by storage devices and processors included in a plurality of computers. Furthermore, the configuration may include a storage device integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache).

[0263] The network interface 74 is an interface for connecting to the communication network 8 wirelessly or via a wired connection. The network interface 74 can be any appropriate interface, such as one conforming to existing communication standards. Information may be exchanged between the computer 7 and an external device 9A connected via the communication network 8 through the network interface 74. The communication network 8 may be a WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or a combination thereof, as long as information is exchanged between the computer 7 and the external device 9A. An example of a WAN is the Internet, an example of a LAN is IEEE 802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication).

[0264] The device interface 75 is an interface such as USB that connects directly to the external device 9B.

[0265] External device 9A is a device connected to computer 7 via a network. External device 9B is a device directly connected to computer 7.

[0266] External device 9A or external device 9B may, for example, be an input device. The input device may be a camera, microphone, motion capture device, various sensors, keyboard, mouse, touch panel, etc., and provides the acquired information to the computer 7. Alternatively, it may be a device equipped with an input unit, memory, and processor, such as a personal computer, tablet terminal, or smartphone.

[0267] Furthermore, external device 9A or external device 9B may, for example, be an output device. The output device may be a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or a speaker that outputs sound, etc. It may also be a device equipped with an output unit, memory, and a processor, such as a personal computer, tablet terminal, or smartphone.

[0268] Furthermore, external devices 9A and 9B may be storage devices (memory). For example, external device 9A may be network storage, and external device 9B may be storage such as an HDD.

[0269] Furthermore, the external device 9A or external device 9B may be a device that has some of the functions of the components of each device (interaction device 10, control device 20, generation device 30, and terminal device 40) in the above-described embodiment. In other words, the computer 7 may transmit some or all of the processing results to the external device 9A or external device 9B, or may receive some or all of the processing results from the external device 9A or external device 9B.

[0270] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. Furthermore, any element may have multiple instances, such as aa, abb, aabbcc, etc. In addition, it is also possible to add other elements other than the enumerated elements (a, b, and c), such as abcd which has d.

[0271] In this specification (including the claims), when expressions such as "using data as input / based on data / according to / in accordance with data" (including similar expressions) are used, unless otherwise specified, this includes using the data itself or using data that has been processed in some way (e.g., data with added noise, normalized data, features extracted from the data, intermediate representations of the data, etc.). Furthermore, when it is stated that some result is obtained "using data as input / based on data / according to / in accordance with data" (including similar expressions), unless otherwise specified, this includes cases where the result is obtained based solely on the data in question or where the result is influenced by other data, factors, conditions, and / or states other than the data in question. Furthermore, when it is stated that "data is output" (including similar expressions), unless otherwise specified, this includes cases where the data itself is used as output or where data that has been processed in some way (e.g., data with added noise, normalized data, features extracted from the data, intermediate representations of various types of data, etc.) is used as output.

[0272] In this specification (including the claims), the terms “connected” and “coupled” are intended to be non-restrictive terms that include any direct connection / coupling, indirect connection / coupling, electrical connection / coupling, communicative connection / coupling, operational connection / coupling, physical connection / coupling, etc. The terms should be interpreted as appropriate in the context in which they are used, but any form of connection / coupling that is not intentionally or naturally excluded should be interpreted non-restrictively as being included in the terms.

[0273] In this specification (including the claims), when the expression "A configured to B" is used, it may include that the physical structure of element A has a configuration capable of performing operation B, and that the permanent or temporary setting / configuration of element A is configured to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and that it is configured to actually perform operation B by the setting of a permanent or temporary program (instruction). Furthermore, if element A is a dedicated processor, dedicated arithmetic circuit, etc., it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0274] In this specification (including the claims), when terms meaning containing or possessing (for example, "comprising / including", "having", etc.) are used, such terms are intended as open-ended terms that include the case of containing or possessing objects other than the subject indicated by the object of the relevant term. Where the object of these terms meaning containing or possessing is an expression that does not specify a quantity or suggests the singular (an expression with the article "a" or "an"), such expression should be interpreted as not being limited to a specific number.

[0275] In this specification (including the claims), even if expressions such as "one or more", "at least one", etc. are used in one place, and expressions that do not specify a quantity or suggest the singular (expressions with the article "a" or "an") are used in other places, the latter expressions are not intended to mean "one". In general, expressions that do not specify a quantity or suggest the singular (expressions with the article "a" or "an") should always be interpreted as not being limited to a specific number.

[0276] In this specification, when it is stated that a specific advantage / result is obtained for a specific configuration of a certain embodiment, unless there is a particular reason, it should be understood that the same advantage / result can also be obtained for one or more other embodiments having the relevant configuration. However, it should be understood that the presence or absence of the advantage / result generally depends on various factors, conditions and / or states, and the relevant configuration does not necessarily guarantee that the advantage / result is obtained. The advantage / result is merely obtained by the relevant configuration described in the embodiment when various factors, conditions and / or states are satisfied, and the invention according to the claims that define the relevant configuration or a similar configuration does not necessarily achieve the relevant advantage / result.

[0277] In this specification (including the claims), when a plurality of hardware units perform a predetermined process, the respective hardware units may cooperate to perform the predetermined process, or a part of the hardware units may perform all of the predetermined process. Alternatively, a part of the hardware units may perform a part of the predetermined process, and another hardware unit may perform the rest of the predetermined process. In this specification (including the claims), when expressions such as "one or more hardware units perform a first process, and the one or more hardware units perform a second process" (including similar expressions) are used, the hardware unit that performs the first process and the hardware unit that performs the second process may be the same or different. That is, the hardware unit that performs the first process and the hardware unit that performs the second process only need to be included in the one or more hardware units. Note that the hardware may include electronic circuits, devices including electronic circuits, and the like.

[0278] In this specification (including the claims), when a plurality of storage devices (memories) store data, each individual storage device among the plurality of storage devices may store only a part of the data, or may store the entire data. Alternatively, a configuration in which a part of the plurality of storage devices stores the data may be included.

[0279] In this specification (including the claims), terms such as "first", "second" and the like are merely used as a way to distinguish between two or more elements, and are not necessarily intended to impose technical meanings such as temporal aspect, spatial aspect, order, and quantity on the subject. Therefore, for example, references to a first element and a second element do not necessarily mean that only two elements can be employed therein, that the first element must precede the second element, that the first element must exist for the second element to exist, and the like.

[0280] While embodiments of this disclosure have been described in detail above, this disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, and partial deletions are possible, provided that they do not depart from the conceptual idea and spirit of the present invention derived from the claims and their equivalents. For example, where numerical values ​​or mathematical formulas are used in the description of the embodiments described above, these are provided for illustrative purposes only and do not limit the scope of this disclosure. Similarly, the sequence of operations shown in the embodiments is also illustrative and does not limit the scope of this disclosure.

[0281] Furthermore, the following forms are possible for disclosure technology.

[0282] (Note 1) At least one memory, Equipped with at least one processor, The aforementioned at least one processor is Obtain the first scenario information to be used for the first interaction with the user, Based on the first scenario information and the machine learning model, the first interaction with the user is performed. To use in a second interaction with the user, obtain a second scenario information different from the first scenario information, After the completion of the first dialogue, the second dialogue with the user is performed based on the second scenario information and the machine learning model. The first scenario information and the second scenario information share at least some information regarding dialogue control. Information processing system.

[0283] (Note 2) The aforementioned at least one processor is Based on the user's history information, the second scenario information is obtained. The information processing system described in Appendix 1.

[0284] (Note 3) Based on the user's history information includes based on the first scenario information. The information processing system described in Appendix 2.

[0285] (Note 4) At least some of the information relating to the dialogue control includes one of the following: information indicating the relationship between the user and the dialogue partner, preconditions in the dialogue, or constraints on utterances. The information processing system described in Appendix 1.

[0286] (Note 5) At least some of the information relating to the aforementioned dialogue control includes a system prompt template, An information processing system as described in any one of the items in Appendix 1 to Appendix 4.

[0287] (Note 6) At least some of the information relating to the dialogue control includes at least information indicating the relationship, The aforementioned relationship includes at least one of the following: a relationship in which the user provides services or goods to the conversation partner; a relationship in which the user instructs the conversation partner; a relationship in which the conversation partner pays remuneration to the user; or a relationship in which the user who possesses information A provides information A to the conversation partner who does not possess information A. The information processing system described in Appendix 4.

[0288] (Note 7) At least some of the information relating to the dialogue control includes at least information indicating the relationship, The aforementioned relationship includes at least one of the following: the user is the examinee and the other party is the interviewer; the user is the store clerk and the other party is the customer; the user is the boss and the other party is the subordinate; the user is the teacher and the other party is the student; the user is the speaker and the other party is the audience; the user is the doctor and the other party is the patient; or the user is the person being requested and the other party is the requester. The information processing system described in Appendix 4.

[0289] (Appendix 8) The at least one processor is configured to generate the first scenario information based on predetermined template information and one or more first words, generate the second scenario information based on the predetermined template information and one or more second words different from the one or more first words, The information processing system according to any one of appendices 1 to 7.

[0290] (Appendix 9) The at least one processor is configured to randomly select the one or more first words from a plurality of predetermined words. The information processing system according to appendix 8.

[0291] (Appendix 10) The at least one processor is configured to randomly select the one or more second words from a set obtained by excluding the one or more first words from the plurality of words. The information processing system according to appendix 9.

[0292] (Appendix 11) The at least one processor is configured to select the first scenario information from a plurality of scenario information sharing at least part of information related to the dialogue control. The information processing system according to any one of appendices 1 to 10.

[0293] (Appendix 12) The at least one processor is configured to randomly select the first scenario information from the plurality of scenario information. The information processing system according to appendix 11.

[0294] (Appendix 13) The at least one processor is configured to Based on predetermined criteria, the first scenario information and the second scenario information are selected from the plurality of scenario information. The information processing system described in Appendix 11.

[0295] (Note 14) The aforementioned at least one processor is The second scenario information is selected from the set of multiple scenario information excluding the first scenario information. The information processing system described in Appendix 11.

[0296] (Note 15) The first scenario information includes at least a first system prompt and first guidance information, The second scenario information includes at least a second system prompt and second guidance information, The first system prompt is generated by embedding the first word in the placeholder of the system prompt template, The first guidance information is generated by embedding the first word in the placeholder of the guidance template, The second system prompt is generated by embedding a second word different from the first word in the placeholder of the system prompt template, The second guidance information is generated by embedding the second word in the placeholder of the guidance template, The aforementioned at least one processor is Before performing the first dialogue, the first guidance information is notified to the user. The second guidance information is notified to the user before the execution of the second dialogue. An information processing system as described in any one of the items in Appendix 1 through Appendix 14.

[0297] (Note 16) The aforementioned at least one processor is The first interaction is executed by inputting a system prompt to the machine learning model which operates based on the first configuration information included in the first scenario information. The second interaction is performed by inputting the system prompt to the machine learning model which operates based on the second configuration information included in the second scenario information. An information processing system as described in any one of the items in Appendix 1 through Appendix 15.

[0298] (Note 17) The aforementioned at least one processor is The first dialogue is executed by inputting the system prompts included in the first scenario information into the machine learning model. The second dialogue is executed by inputting the system prompt, information about the first dialogue, and information instructing the machine learning model to perform a dialogue different from the first dialogue. The second scenario information includes information that instructs to perform a dialogue different from the first dialogue, An information processing system as described in any one of the items in Appendix 1 through Appendix 16.

[0299] (Note 18) The aforementioned at least one processor is At least dialogue data is obtained that includes information about the first utterance spoken by the user in the second dialogue. At least a portion of the aforementioned dialogue data is input into a second machine learning model to convert it into predetermined data. Based on the predetermined data, a predetermined score is calculated for the dialogue data. An information processing system as described in any one of the items in Appendix 1 through Appendix 17.

[0300] (Note 19) The aforementioned machine learning model and the aforementioned second machine learning model are the same model. The information processing system described in Appendix 18.

[0301] (Note 20) At least one memory, Equipped with at least one processor, The aforementioned at least one processor is We obtain dialogue data that includes information about the first utterance spoken by at least one speaker, At least a portion of the aforementioned dialogue data is input into a machine learning model to convert it into predetermined data. Based on the predetermined data, a predetermined score is calculated for the dialogue data. Information processing system.

[0302] (Note 21) The aforementioned dialogue data includes information about the second utterance spoken by the second speaker. The information processing system described in Appendix 20.

[0303] (Note 22) The aforementioned predetermined data includes data that summarizes the dialogue data. The information processing system described in Appendix 20 or 21.

[0304] (Note 23) The aforementioned dialogue data and the aforementioned predetermined data include text data indicating the content of the utterance. The predetermined score is calculated based on the amount of data in the dialogue data and the amount of data in the predetermined data. The information processing system described in Appendix 22.

[0305] (Note 24) The predetermined score includes a score relating to the conciseness of the utterance by the first speaker. An information processing system as described in any one of the items in Appendix 20 to Appendix 23.

[0306] (Note 25) The aforementioned dialogue data and the aforementioned predetermined data include voice data indicating speech, The predetermined score is calculated based on the amount of data in the dialogue data and the amount of data in the predetermined data. The information processing system described in Appendix 22.

[0307] (Note 26) The predetermined data includes truth values ​​obtained by evaluating whether the dialogue data satisfies predetermined characteristics using one or more evaluation axes. An information processing system as described in any one of the items in Appendix 20 to Appendix 25.

[0308] (Note 27) The truth value is calculated by inputting the dialogue data into the machine learning model. The information processing system described in Appendix 26.

[0309] (Note 28) The predetermined score is calculated based on the number of evaluation axes and the number of characteristics that are evaluated to satisfy the criteria. The information processing system described in Appendix 26 or 27.

[0310] (Note 29) The predetermined score includes a score relating to the logicality of the utterance by the first speaker. An information processing system as described in any one of the items in Appendix 26 to Appendix 28.

[0311] (Note 30) The aforementioned at least one processor is The predetermined score is calculated by inputting the predetermined data into the machine learning model. An information processing system as described in any one of the items in Appendix 20 to Appendix 29.

[0312] (Note 31) Using the information processing system described in any one of the items from Appendix 1 to Appendix 19, Obtain the first scenario information to be used for the first interaction with the user, Based on the first scenario information and the machine learning model, the first interaction with the user is performed. To use in a second interaction with the user, obtain a second scenario information different from the first scenario information, After the completion of the first dialogue, the second dialogue with the user is performed based on the second scenario information and the machine learning model. The first scenario information and the second scenario information share at least some information regarding dialogue control. Information processing methods.

[0313] (Note 32) Using the information processing system described in any one of the items in Appendix 20 to Appendix 30, We obtain dialogue data that includes information about the first utterance spoken by at least one speaker, At least a portion of the aforementioned dialogue data is input into a machine learning model to convert it into predetermined data. Based on the predetermined data, a predetermined score is calculated for the dialogue data. Information processing methods.

[0314] (Note 33) At least one processor is made to execute the information processing method described in Appendix 31. program.

[0315] (Note 34) At least one processor is made to execute the information processing method described in Appendix 32. program. [Explanation of symbols]

[0316] 10: Dialogue device 20: Control device 30:Generation device 40: Terminal device 101: Scenario Memory Department 102: Dialogue Memory Unit 103: Evaluation Result Storage Unit 110: Scenario Acquisition Department 120: Speech acquisition unit 130: Speech generation unit 140: Utterance presentation section 150: Dialogue Evaluation Department 160: Result output section 1000: Dialogue System D: Dialogue data M: Machine learning model

Claims

1. At least one memory, It comprises at least one processor, The aforementioned at least one processor is Obtain the first scenario information to be used for the first interaction with the user, Based on the first scenario information and the machine learning model, the first interaction with the user is performed. To obtain a second scenario information different from the first scenario information, which will be used for a second dialogue with the user, After the completion of the first dialogue, the second dialogue with the user is performed based on the second scenario information and the machine learning model. The first scenario information and the second scenario information share at least some information related to dialogue control. Information processing system.

2. The aforementioned at least one processor is Based on the user's history information, the second scenario information is obtained. The information processing system according to claim 1.

3. Based on the user's history information includes based on the first scenario information. The information processing system according to claim 2.

4. At least some of the information relating to the dialogue control includes one of the following: information indicating the relationship between the user and the dialogue partner, preconditions in the dialogue, or constraints on utterances. The information processing system according to claim 1.

5. At least some of the information relating to the aforementioned dialogue control includes a system prompt template, The information processing system according to claim 1.

6. At least some of the information relating to the dialogue control includes at least information indicating the relationship, The aforementioned relationship includes at least one of the following: a relationship in which the user provides services or goods to the conversation partner; a relationship in which the user instructs the conversation partner; a relationship in which the conversation partner pays remuneration to the user; or a relationship in which the user who possesses information A provides information A to the conversation partner who does not possess information A. The information processing system according to claim 4.

7. At least some of the information relating to the dialogue control includes at least information indicating the relationship, The aforementioned relationship includes at least one of the following relationships: the user is the examinee and the other party is the interviewer; the user is the store clerk and the other party is the customer; the user is the boss and the other party is the subordinate; the user is the teacher and the other party is the student; the user is the speaker and the other party is the audience; the user is the doctor and the other party is the patient; or the user is the person being requested and the other party is the requester. The information processing system according to claim 4.

8. The aforementioned at least one processor is Based on predetermined template information and one or more first words, the first scenario information is generated. Based on the predetermined template information and one or more second words different from the one or more first words, the second scenario information is generated. The information processing system according to claim 1.

9. The aforementioned at least one processor is Select one or more first words randomly from a predetermined set of words. The information processing system according to claim 8.

10. The aforementioned at least one processor is From the set of multiple words excluding the one or more first words, one or more second words are randomly selected. The information processing system according to claim 9.

11. The aforementioned at least one processor is The first scenario information is selected from a plurality of scenario information in which at least some of the information relating to the dialogue control is common. The information processing system according to claim 1.

12. The aforementioned at least one processor is The first scenario information is randomly selected from the aforementioned multiple scenario information. The information processing system according to claim 11.

13. The aforementioned at least one processor is Based on predetermined criteria, the first scenario information and the second scenario information are selected from the plurality of scenario information. The information processing system according to claim 11.

14. The aforementioned at least one processor is The second scenario information is selected from the set of multiple scenario information excluding the first scenario information. The information processing system according to claim 11.

15. The first scenario information includes at least a first system prompt and first guidance information, The second scenario information includes at least a second system prompt and second guidance information, The first system prompt is generated by embedding the first word in the placeholder of the system prompt template, The first guidance information is generated by embedding the first word in the placeholder of the guidance template, The second system prompt is generated by embedding a second word different from the first word in the placeholder of the system prompt template, The second guidance information is generated by embedding the second word in the placeholder of the guidance template, The aforementioned at least one processor is Before performing the first dialogue, the first guidance information is notified to the user. The second guidance information is notified to the user before the execution of the second dialogue. The information processing system according to claim 1.

16. The aforementioned at least one processor is The first interaction is executed by inputting a system prompt to the machine learning model which operates based on the first configuration information included in the first scenario information. The second interaction is performed by inputting the system prompt to the machine learning model which operates based on the second configuration information included in the second scenario information. The information processing system according to claim 1.

17. The aforementioned at least one processor is The first dialogue is executed by inputting the system prompts included in the first scenario information into the machine learning model. The second dialogue is executed by inputting the system prompt, information regarding the first dialogue, and information instructing the machine learning model to execute a dialogue different from the first dialogue. The second scenario information includes information that instructs to perform a dialogue different from the first dialogue, The information processing system according to claim 1.

18. The aforementioned at least one processor is At least dialogue data is obtained that includes information about the first utterance spoken by the user in the second dialogue, At least a portion of the aforementioned dialogue data is input into a second machine learning model to convert it into predetermined data. Based on the predetermined data, a predetermined score is calculated for the dialogue data. The information processing system according to any one of claims 1 to 17.

19. The aforementioned machine learning model and the aforementioned second machine learning model are the same model. The information processing system according to claim 18.

20. At least one memory, It comprises at least one processor, The aforementioned at least one processor is Dialogue data is obtained that includes information about the first utterance spoken by at least one speaker. At least a portion of the aforementioned dialogue data is input into a machine learning model to convert it into predetermined data. Based on the predetermined data, a predetermined score is calculated for the dialogue data. Information processing system.

21. The aforementioned dialogue data includes information about the second utterance spoken by the second speaker. The information processing system according to claim 20.

22. The aforementioned predetermined data includes data that summarizes the dialogue data. The information processing system according to claim 20.

23. The aforementioned dialogue data and the aforementioned predetermined data include text data indicating the content of the utterance. The predetermined score is calculated based on the amount of data in the dialogue data and the amount of data in the predetermined data. The information processing system according to claim 22.

24. The predetermined score includes a score relating to the conciseness of the utterance by the first speaker. The information processing system according to claim 20.

25. The aforementioned dialogue data and the aforementioned predetermined data include voice data indicating speech, The predetermined score is calculated based on the amount of data in the dialogue data and the amount of data in the predetermined data. The information processing system according to claim 22.

26. The predetermined data includes truth values ​​obtained by evaluating whether the dialogue data satisfies predetermined characteristics using one or more evaluation axes. The information processing system according to claim 20.

27. The truth value is calculated by inputting the dialogue data into the machine learning model. The information processing system according to claim 26.

28. The predetermined score is calculated based on the number of evaluation axes and the number of characteristics that are evaluated to satisfy the aforementioned characteristics. The information processing system according to claim 26.

29. The predetermined score includes a score relating to the logicality of the utterance made by the first speaker. The information processing system according to claim 26.

30. The aforementioned at least one processor is By inputting the predetermined data into the machine learning model, the predetermined score is calculated. The information processing system according to claim 20.

31. Using the information processing system described in any one of claims 1 to 17, Obtain the first scenario information to be used for the first interaction with the user, Based on the first scenario information and the machine learning model, the first interaction with the user is performed. To obtain a second scenario information different from the first scenario information, which will be used for a second dialogue with the user, After the completion of the first dialogue, the second dialogue with the user is performed based on the second scenario information and the machine learning model. The first scenario information and the second scenario information share at least some information related to dialogue control. Information processing methods.

32. Using the information processing system described in any one of claims 20 to 30, Dialogue data is obtained that includes information about the first utterance spoken by at least one speaker. At least a portion of the aforementioned dialogue data is input into a machine learning model to convert it into predetermined data. Based on the predetermined data, a predetermined score is calculated for the dialogue data. Information processing methods.

33. The information processing method described in claim 31 is performed on at least one processor. program.

34. To cause at least one processor to execute the information processing method described in claim 32, program.

Images