Information processing apparatus, information processing method, and program

Abstract

To provide a technique capable of appropriately evaluating interaction contents in role playing.SOLUTION: A setting unit configured to set information related to a generated AI model of a customer role based on the setting information related to the business negotiation; and an instructing unit configured to instruct the generated AI model of a customer role to evaluate content of utterance in a business negotiation between the generated AI model and a person based on the information related to the content of the utterance.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, interactive AI services using generative artificial intelligence (AI) have become available.

[0003] For example, Patent Document 1 discloses a technology that uses such a generation AI to provide an answer to a user's question while referring to an appropriate document. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-76413 Summary of the Invention [Problem to be solved by the invention]

[0005] There are also known techniques for performing role-playing using generative AI. For example, one such role-playing is one for improving dialogue skills. Participants in such role-playing would like to know the level of their dialogue skills in the role-playing.

[0006] In view of the above, one object of the present disclosure is to provide a technique that can appropriately evaluate the content of dialogue in role-playing. [Means for solving the problem]

[0007] An information processing device according to one embodiment of the present disclosure includes an acquisition unit that acquires setting information related to a business negotiation, a setting unit that sets information related to a generative AI model for a customer role based on the setting information related to the business negotiation, and an instruction unit that instructs the generative AI model to evaluate the content of an utterance based on information related to the content of an utterance during a business negotiation between the generative AI model for the customer role and a person. [Effects of the Invention]

[0008] According to one aspect of the present disclosure, the content of dialogue in role-playing can be appropriately evaluated. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a schematic configuration example of a mock negotiation evaluation system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the device according to the embodiment. [Figure 3] FIG. 10 is a block diagram illustrating an example of a functional configuration of a user device according to the embodiment. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of the mock negotiation evaluation device according to the embodiment. [Figure 5A] FIG. 2 is a block diagram illustrating an example of a functional configuration of a database device according to the embodiment. [Figure 5B] FIG. 2 is a diagram illustrating an example of a configuration of a knowledge database according to the embodiment. [Figure 6] FIG. 2 is a block diagram illustrating an example of a functional configuration of a generating device according to the embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of a prompt according to the embodiment. [Figure 8] FIG. 10 is a diagram showing an example of an evaluation result according to the embodiment. [Figure 9] FIG. 10 is a sequence diagram illustrating an example of the operation of the mock negotiation evaluation system according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] (Embodiment) <Summary> According to one embodiment of the present disclosure, an information processing device (e.g., a mock business negotiation evaluation device 20 described later) includes an acquisition unit, a setting unit, and an instruction unit. The acquisition unit acquires setting information related to a business negotiation (e.g., a mock business negotiation). The setting unit sets information related to a generative AI model (e.g., a trained large-scale language model) playing the role of a customer based on the setting information related to the business negotiation. The instruction unit instructs the generative AI model to evaluate the content of the utterance based on information related to the content of the utterance during the business negotiation between the generative AI model playing the customer and a person.

[0011] The information processing device sets information about the generative AI model for the customer role and instructs the generative AI model for the customer role to evaluate the content of utterances in business negotiations between the thus set generative AI model for the customer role and the person. In this way, by having the generative AI model act as the customer, the content of the utterances can be appropriately evaluated. In particular, if the person is a salesperson, the customer's voice is very important in business, so the evaluation results will be even more valuable information for the salesperson.

[0012] <Configuration of the mock business negotiation evaluation system> FIG. 1 is a block diagram illustrating a schematic configuration example of a negotiation evaluation system according to an embodiment of the present disclosure.

[0013] As shown in FIG. 1, the simulated business negotiation evaluation system 1 includes a user device 10, a simulated business negotiation evaluation device 20, a database (DB) device 30, and a generation device 40. Two or more of the user device 10, the simulated business negotiation evaluation device 20, and the generation device 40 are communicatively connected to each other via a network 50. The simulated business negotiation evaluation device 20 and the DB device 30 are communicatively connected directly to each other. Alternatively, the simulated business negotiation evaluation device 20 and the DB device 30 may be communicatively connected to each other via the network 50. The network 50 may be a LAN (Local Area Network), the Internet, an intranet, a VPN (Virtual Private Network), a mobile communication network, or a combination thereof.

[0014] The mock business negotiation evaluation system 1 provides the user of the user device 10 who conducted the mock business negotiation with an evaluation of the mock business negotiation using a generative AI model (e.g., a learned large-scale language model (LLM)). For example, the user of the user device 10 may be a sales representative user (an example of a person). In the following description, the user of the user device 10 is assumed to be a sales representative user.

[0015] In this embodiment, it is assumed that the DB device 30 stores the interactions during the mock negotiation between the sales representative user and the generated AI model stored in the generation device 40 (i.e., information indicating the content of the sales representative user's utterances (e.g., text information on the content of the utterances) and the response (history) of the generated AI model to the same). Also, in this embodiment, it is assumed that the mock negotiation evaluation device 20 identifies the information of the user of the user device 10 based on the login information of the user of the user device 10.

[0016] The user device 10 may be realized by any computer or information processing device such as a tablet, a smartphone, or a PC (Personal Computer).

[0017] The user device 10 is configured to be connectable to the simulated business negotiation evaluation device 20, for example, via a dedicated application or a web browser installed on the user device 10. For example, the following processing of the user device 10 may be performed via the dedicated application or the web browser.

[0018] The user device 10 receives data for evaluating the mock negotiations from the mock negotiation evaluation device 20 via the network 50. The data may include, for example, a list of mock negotiations conducted by the sales representative user, screen data including interactions between the sales representative user and the generated AI model stored in the generation device 40 in each mock negotiation, and an evaluation result generation request button associated with each mock negotiation.

[0019] In response to the sales representative user pressing the evaluation result generation request button, the user device 10 sends an evaluation result generation request including identification information of the mock business negotiation associated with the evaluation result generation request button to the mock business negotiation evaluation device 20 via the network 50.

[0020] The user device 10 receives the evaluation results of the content of the utterances made in the mock business negotiation from the mock business negotiation evaluation device 20 via the network 50 and outputs them to the outside.

[0021] 1 shows one user device 10, there may be multiple user devices 10. The user device 10 may also be called an information terminal, a user terminal, a terminal, a client, a client device, etc.

[0022] The simulated negotiation evaluation device 20 may be realized by a computer such as a server or an information processing device.

[0023] The simulated business negotiation evaluation device 20 is installed in, for example, a cloud environment. Alternatively, the simulated business negotiation evaluation device 20 may be installed in an on-premise environment.

[0024] The mock business negotiation evaluation device 20 transmits data for evaluating the mock business negotiation to the user device 10 via the network 50.

[0025] The simulated business negotiation evaluation device 20 receives an evaluation result generation request from the user device 10 via the network 50.

[0026] The simulated business negotiation evaluation device 20 transmits a search request to the DB device 30 to search the knowledge DB stored in the DB device 30, and acquires various information related to the generation of evaluation results from the DB device 30. Such information includes, for example, setting information related to the simulated business negotiation. Acquiring setting information related to the simulated business negotiation may be considered as determining setting information related to the simulated business negotiation.

[0027] The simulated business negotiation evaluation device 20 sets information about the generative AI model for the customer role (stored in the generation device 40) based on the setting information about the simulated business negotiation. Setting information about the generative AI model for the customer role may be considered as determining information about the generative AI model for the customer role.

[0028] The simulated business negotiation evaluation device 20 instructs the generation AI model to evaluate the utterance content based on information about the utterance content in the simulated business negotiation between the generation AI model playing the customer role and the sales representative user. Instructing the generation AI model to evaluate the utterance content may be interpreted as generating a prompt for evaluating the utterance content and transmitting the generated prompt to the generation device 40 via the network 50. The prompt may be interpreted as an instruction (sentence), a question (sentence), etc., as appropriate.

[0029] The mock business negotiation evaluation device 20 receives the evaluation results of the speech content in the mock business negotiation from the generation device 40 via the network 50.

[0030] The mock business negotiation evaluation device 20 transmits the evaluation results of the speech content in the mock business negotiation to the user device 10 via the network 50.

[0031] The DB device 30 may be realized by a computer such as a server or an information processing device.

[0032] The DB device 30 is installed in, for example, a cloud environment, or may be installed in an on-premise environment.

[0033] The DB device 30 stores a knowledge DB. In response to a search request from the simulated business negotiation evaluation device 20, the DB device 30 retrieves various pieces of information related to the simulated business negotiation, which are search results, from the knowledge DB and returns the information to the simulated business negotiation evaluation device 20.

[0034] The generating device 40 may be realized by a computer such as a server or an information processing device.

[0035] The generation device 40 is installed in, for example, a cloud environment, or may be installed in an on-premise environment.

[0036] The generation device 40 receives prompts from the simulated business negotiation evaluation device 20 via the network 50. The prompts include, for example, prompts for evaluating the content of utterances made during the simulated business negotiation between the customer role generation AI model and the sales representative user.

[0037] The generation device 40 inputs the received prompt into a generation AI model stored in the generation device 40, and generates a response to the received prompt (for example, an evaluation result of the content of the utterance in the mock business negotiation, etc.).

[0038] The generation device 40 transmits the generated response to the simulated business negotiation evaluation device 20 via the network 50.

[0039] In this embodiment, the generation device 40 is a device that provides an existing interactive AI service (generated AI (model)) using a trained LLM. The simulated business negotiation evaluation device 20 uses an API (Application Programming Interface) for using the interactive AI service to send prompts to the generation device 40 and receive responses to the prompts from the generation device 40. Alternatively, the generation device 40 may be a device that provides an interactive AI service using a trained LLM that has been generated by training the LLM described above using training data such as text data published on the web.

[0040] <Configuration of various devices> 2 is a block diagram showing an example of the hardware configuration of the device according to this embodiment. The device 1000 shown in FIG. 2 may include a user device 10, a simulated negotiation evaluation device 20, a DB device 30, and a generation device 40.

[0041] 2, the device 1000 may include a storage device 1001, a processing device 1002, a user interface (UI) device 1003, a communication device 1004, and a bus 1005. The storage device 1001, the processing device 1002, the UI device 1003, and the communication device 1004 may be connected to each other via the bus 1005.

[0042] The programs or instructions that realize the functions and processes described above and below of the apparatus 1000 (e.g., the user device 10, the simulated negotiation evaluation apparatus 20, the DB apparatus 30, and the generation apparatus 40) may be downloaded from some external device (e.g., a server) via a network or the like. Such programs or instructions may also be provided from a removable storage medium such as a CD-ROM (Compact Disc Read Only Memory) or a flash memory.

[0043] Storage device 1001 may be implemented by RAM (Random Access Memory), flash memory, a hard disk drive, or the like, and stores installed programs or instructions as well as files, data, and the like used to execute the programs or instructions. Storage device 1001 may also include a non-transitory storage medium.

[0044] The processing device 1002 may be realized, for example, by a general-purpose processor or controller (circuit), or by a dedicated processor or controller (circuit). When realized by a general-purpose processor, the processing device 12 may be realized by one or more central processing units (CPUs), graphics processing units (GPUs), processing circuitry, etc., each of which may be composed of one or more processor cores. In this case, the processing device 1002 performs the functions and processes of the device 1000 described above and below in accordance with programs or instructions stored in the storage device 1001, data such as parameters used to execute the programs or instructions, and the like. Such a program, when executed by a computer or processing device 1002, may cause the computer or processing device 1002 to function as the device 1000 according to the present disclosure. Furthermore, such a program, when executed by the computer or processing device 1002, may cause the computer or processing device 1002 to execute a method for evaluating a simulated sales negotiation according to the present disclosure.

[0045] The UI device 1003 may be composed of input devices such as a keyboard, a mouse, a camera, a microphone, etc., output devices such as a display, a speaker, a headset, a printer, etc., and input / output devices such as a haptic device, a touch panel, etc., and realizes an interface between the user of the device 1000 and the device 1000. For example, the user of the device 1000 operates the device 1000 by manipulating a graphical user interface (GUI) displayed on a display or a touch panel using a keyboard, a mouse, a stylus, the user's finger, etc.

[0046] The communication device 1004 may be realized by various communication circuits that execute communication processes with external devices, the Internet, LAN, VPN, mobile communication networks, and other communication networks.

[0047] The above-described hardware configuration of the device 1000 is merely an example, and the device 1000 according to the present disclosure may be realized by any other appropriate hardware configuration.

[0048] FIG. 3 is a block diagram showing an example of the functional configuration of a user device according to this embodiment.

[0049] 3, the user device 10 includes a request sending unit 101, an evaluation result acquiring unit 102, an output control unit 103, and an output unit 151. The request sending unit 101, the evaluation result acquiring unit 102, and the output control unit 103 may be realized as hardware, software, or a combination of hardware and software by a processing device 1002 of the user device 10. The output unit 151 may be realized by an output device or an input / output device (e.g., a display, a speaker, etc.) included in a UI device 1003 of the user device 10.

[0050] The request sending unit 101 sends an evaluation result generation request to the simulated negotiation evaluation device 20 via the communication device 1004 of the user device 10 in response to the sales representative user pressing the evaluation result generation request button of the user device 10.

[0051] In response to the evaluation result generation request, the evaluation result acquisition unit 102 acquires the evaluation result by receiving the evaluation result of the content of the utterances in the exchange in the mock negotiation between the sales representative user and the generated AI model from the mock negotiation evaluation device 20 via the communication device 1004 of the user device 10. The evaluation result acquisition unit 102 outputs the acquired evaluation result to the output control unit 103.

[0052] The output control unit 103 outputs the evaluation result output by the evaluation result acquisition unit 102 to an output device or an input / output device of the user device 10. For example, if the evaluation result is text information, the output control unit 103 may output the evaluation result to a display. Also, for example, if the evaluation result is audio information, or if the user device 10 has a TTS (text-to-speech) function and the evaluation result, which is text information, is converted into audio information, the output control unit 103 may output the audio information to a speaker.

[0053] Under the control of the output control unit 103, the output unit 151 outputs the evaluation result to the outside of the user device 10 (to the sales representative user).

[0054] FIG. 4 is a block diagram showing an example of the functional configuration of the mock negotiation evaluation device according to this embodiment.

[0055] 4, the simulated business negotiation evaluation device 20 includes an acquisition unit 201, a setting unit 202, an instruction unit 203, and an evaluation result processing unit 204. The acquisition unit 201, the setting unit 202, the instruction unit 203, and the evaluation result processing unit 204 may be realized by the processing device 1002 of the simulated business negotiation evaluation device 20 as hardware, software, or a combination of hardware and software.

[0056] The acquisition unit 201 receives an evaluation result generation request from the user device 10 via the communication device 1004 of the mock business negotiation evaluation device 20, and in response to the request, searches the knowledge DB 361 (see FIGS. 5A and 5B) stored in the DB device 30 to acquire various information related to the evaluation of the mock business negotiation. The acquired information includes setting information related to the mock business negotiation.

[0057] The setting information regarding the mock business negotiation may include, for example, information regarding the content of the business negotiation. The information regarding the content of the business negotiation may include, for example, information regarding the content of the mock business negotiation between the customer role generation AI model and the sales representative user (for example, information indicating the content of the contract).

[0058] In addition, the setting information regarding the mock business negotiation may include, for example, information regarding the AI ​​model for generating the customer role.

[0059] Information regarding the customer role generation AI model may include, for example, information indicating one or a combination of the customer's business activities, the company or department to which the customer belongs, the customer's job title, etc.

[0060] Furthermore, the information about the generative AI model playing the customer role may include, for example, customer attribute information, information about the customer's personality, etc. For example, the information about the generative AI model playing the customer role may include (1) information indicating the relationship between the sales representative user and the generative AI model playing the customer role, and / or (2) information indicating the customer's values. The information indicating the relationship (1) may include, for example, information indicating good / average / bad, information indicating the degree of intimacy, etc. The information indicating the values ​​(2) may include, for example, information indicating the degree of importance placed on each of the following items: quality / function, price, reputation, track record, level of customer service, ability to respond to customer requests, etc. (e.g., quality / function: 80%, price: 70%, reputation: 40%, etc.). Note that the information indicating the above degrees may be expressed in a manner that is easy for the generative AI model to understand. For example, the knowledge DB 361 may store information indicating that the degree "80 to 100%" is to be read as "very important," or "very important" may be entered in a prompt (described later), transmitted to the generating device 40, and input to the generative AI model. Furthermore, in relation to the information indicating values ​​(2), information regarding conditions for each item may be stored in the knowledge DB 361. For example, in the case of a simulated business negotiation regarding a smartphone, the knowledge DB 361 may store "quality / function MUST condition: maximum communication speed (downstream) 2.4 Gbps, quality / function WANT condition: maximum communication speed (downstream) 2.6 Gbps, price MUST condition: 200,000 yen or less, price WANT condition: 180,000 yen or less, ..."

[0061] The setting information regarding the mock business negotiation may also include, for example, setting information regarding role-playing. The setting information regarding role-playing may include, for example, the duration of the mock business negotiation, the difficulty level, and the like.

[0062] The above information may be information specified by a sales representative user, or may be information pre-stored (for example, by a system administrator).

[0063] The acquisition unit 201 acquires some or all of the above-mentioned information as setting information related to the mock negotiation, and outputs the acquired setting information related to the mock negotiation together with the identification information of the mock negotiation included in the evaluation result generation request to the setting unit 202 and the instruction unit 203. The acquisition unit 201 also outputs various information acquired from the knowledge DB 361 other than the setting information to the setting unit 202 and the instruction unit 203.

[0064] The setting unit 202 sets (or determines) information about the generative AI model for the customer role (stored in the generation device 40) based on the setting information about the simulated business negotiation output by the acquisition unit 201. Here, the generative AI model in which information about the generative AI model for the customer role has been set may mean an AI model for the customer role in which some or all of the setting information about the simulated business negotiation has been set as information about the role of the generative AI model in the prompts input to the generative AI model for the customer role. In other words, setting information about the generative AI model for the customer role may mean determining part of the setting information about the simulated business negotiation as information about the role of the generative AI model to be described in the prompts input to the generative AI model for the customer role. The setting unit 202 outputs information about the generative AI model for the customer role to the instruction unit 203.

[0065] The instruction unit 203 instructs the generative AI model to evaluate the utterance content based on information about the utterance content in the simulated business negotiation between the sales representative user and the generative AI model playing the role of a customer, to which the setting information output by the setting unit 202 is set. Instructing the generative AI model to evaluate the utterance content may be interpreted as generating a prompt for evaluating the utterance content and transmitting the generated prompt to the generation device 40 via the communication device 1004 of the simulated business negotiation evaluation device 20. The instruction unit 203 outputs to the evaluation result processing unit 204 a notice that the generative AI model has been instructed to evaluate the utterance content. Here, a prompt refers to an instruction or question input by a user or the like in a dialogue with a generative AI model such as an LLM. The instruction unit 203 may generate a prompt for evaluating the utterance content using RAG (Retrieval-Augmented Generation), a type of prompt expansion technology. A common application example of RAG is a technique in which a user or the like searches for similar documents in a search system before making a request to an LLM and then requests those documents together with the LLM. This allows you to have LLM generate answers based on specific documents, such as company information.

[0066] More specifically, the instruction unit 203 acquires information about the content of utterances in the mock negotiation from various pieces of information output by the acquisition unit 201, based on the identification information of the mock negotiation output by the acquisition unit 201. The information about the content of utterances may include the content of utterances made by the sales representative user and / or information associated therewith. The associated information may include, for example, date and time, information indicating the emotions of the sales representative user (for example, information acquired by a trained emotion estimation model). The information acquired by the trained emotion estimation model may include, for example, information indicating the degree of anger, sadness, joy, surprise, etc. output from the trained emotion estimation model after inputting the analysis results of the sales representative user's voice or facial expression during the mock negotiation into the trained emotion estimation model, or information indicating a degree of positivity or negativity calculated based on this information. Additionally or alternatively, the information about the content of utterances may include the content of utterances made by the customer role generation AI model (output as text information)) and / or information associated therewith (for example, date and time).

[0067] In addition, at this time, the instruction unit 203 may acquire information regarding the scoring criteria from the various information output by the acquisition unit 201, and enter the information regarding the scoring criteria in the prompt, as well as information instructing the user to score the content of the utterances in the mock business negotiation. Acquiring information regarding the scoring criteria may be considered as determining information regarding the scoring criteria. There may be one piece of information regarding the scoring criteria for the entire mock business negotiation, or there may be information regarding the level of each item (evaluation item) related to the customer's values. In the latter case, the information regarding the scoring criteria may be, for example, "WANT conditions for quality and functionality are met: 3 (good), WANT conditions for quality and functionality are not met but MUST conditions are met: 2 (average), MUST conditions for quality and functionality are not met: 1 (poor), WANT conditions for price are met: 3 (good), WANT conditions for price are not met but MUST conditions are met: 2 (average), MUST conditions for price are not met: 1 (poor), ..." The instruction unit 203 may instruct the generative AI model to score the utterance content for each of the multiple evaluation items based on information regarding the utterance content and information regarding multiple scoring criteria corresponding to each of the multiple evaluation items.

[0068] Additionally or alternatively, the instruction unit 203 may cause the customer role generating AI model to evaluate trust and / or consent based on information about the content of the utterance.

[0069] Regarding trust, for example, the instructing unit 203 may enter in the prompt information instructing the sales representative user to evaluate the degree of trust (1: untrustworthy, 2: average, 3: trustworthy) from the perspective of the customer role generation AI model. As a more specific example, in a mock business negotiation, questions are asked about multiple items regarding the functions, price, contents, etc. of a product (e.g., a smartphone), and correct answer examples for the questions are prepared in advance in the knowledge DB 361. Then, the instructing unit 203 may enter in the prompt information instructing the user to evaluate trust based on a comparison between the content of the sales representative user's utterance and this content and the following criteria: Example criteria: If everything is correct, it's 3 (trustworthy). If there are insufficient explanations for some items, but there is no contradiction with the correct answers for all items, give a rating of 2 (normal). If there are discrepancies in some items, rate them as 1 (unreliable). Example questions and answers: (Question 1) What is the communication speed of Smartphone A? (Correct Answer 1) Maximum communication speed (downstream) 2.4Gbps, maximum communication speed (upstream) 159Mbps (Question 2) What is the price of Smartphone A? (Correct Answer 2) 169,590 yen for 128GB or less, 192,140 yen for 256GB More specifically, the instruction unit 203 may instruct the generative AI model to evaluate (or score) the content of the utterance (in terms of trustworthiness) based on examples of correct answers to the questions and the answers (of the sales representative user) to the questions contained in the information about the content of the utterance.

[0070] Regarding the sense of satisfaction, for example, the instruction unit 203 may enter information in the prompt instructing the user to evaluate whether the opinions of the generation AI model playing the customer have been sufficiently heard (1: insufficient, 2: normal, 3: sufficient). As a more specific example, a list of items to be heard from the sales representative user in the mock business negotiation is prepared in advance in the knowledge DB 361. Then, the instruction unit 203 may enter information in the prompt instructing the user to evaluate the sense of satisfaction by checking how many of these items were heard within the time limit in the mock business negotiation based on the content of the sales representative user's utterances and the following criteria: Example criteria: 3 (sufficient) if the number of cases is greater than or equal to a predetermined first threshold (e.g., 7 cases) If the number of items is less than the first threshold and greater than or equal to the second threshold (e.g., 3 items), the value is 2 (normal). 1 (insufficient) if the number of cases is less than a predetermined second threshold Examples of things we would like to hear about: Requirement 1: Maximum communication speed (downstream) 2.6Gbps Request 2: Price under 200,000 yen The instruction unit 203 may instruct the generative AI model to evaluate (or score) the content of the utterance (in terms of satisfaction) based on the requests made in the mock business negotiation and the answers to the requests contained in the information related to the content of the utterance.

[0071] Although the above describes examples of evaluation based on whether conditions are met, whether there is a sense of reliability, whether there is a sense of satisfaction, etc., the present disclosure is not limited to these. For example, the instruction unit 203 may instruct to perform a comprehensive evaluation by combining any two or more of the above examples.

[0072] The instruction unit 203 may also generate a prompt instructing the generation AI model to create a report based on the scoring results. For example, the instruction unit 203 may instruct the generative AI model to score the utterance content based on information about the utterance content and information about the scoring criteria, and to create a report based on the scoring results of the utterance content. More specifically, for example, reference sentences based on the scoring results may be prepared in advance in the knowledge DB 361. Then, the instruction unit 203 may write information in the prompt instructing the generation AI model to create a report so that the reference sentences based on the scoring results are included in the report. Alternatively or additionally, the instruction unit 203 may write information in the prompt instructing the generation AI model to create a report including suggestions on what should be improved based on the above criteria, correct examples, scoring results, etc.

[0073] The evaluation result processing unit 204 receives the evaluation results of the utterance content from the generation device 40 via the communication device 1004 of the simulated business negotiation evaluation device 20. The evaluation results of the utterance content may be a scoring result of the utterance content, or may be a report showing the scoring result together with a more detailed evaluation result of the utterance content. The evaluation result processing unit 204 transmits the evaluation results of the utterance content to the user device 10 via the communication device 1004 of the simulated business negotiation evaluation device 20. The evaluation result processing unit 204 may convert the received evaluation results, which are text information, into voice information using TTS technology, and transmit the evaluation results, which are the converted voice information, to the user device 10.

[0074] 5A is a block diagram showing an example of the functional configuration of the DB device according to this embodiment. As shown in FIG. 5A, the DB device 30 includes a knowledge DB storage unit 351 that stores a knowledge DB 361.

[0075] 5B is a diagram showing an example of the configuration of the knowledge DB according to this embodiment. As shown in FIG. 5B, the knowledge DB 361 stores setting information related to the mock negotiation described above, information related to the content of utterances made during the mock negotiation, information related to the scoring criteria, and other information.

[0076] FIG. 6 is a block diagram showing an example of the functional configuration of the generating device according to this embodiment.

[0077] 6, the generation device 40 includes an acquisition unit 401, a generation unit 402, and an AI model storage unit 451. The acquisition unit 401 and the generation unit 402 may be realized as hardware, software, or a combination of hardware and software by a processing device 1002 of the generation device 40. The AI ​​model storage unit 451 may be realized by the storage device 1001 of the generation device 40.

[0078] The acquisition unit 401 acquires prompts by receiving them from the simulated business negotiation evaluation device 20 via the communication device 1004 of the generation device 40. The acquired prompts include, for example, prompts instructing the user to evaluate the content of an utterance. The acquisition unit 401 outputs the acquired prompts to the generation unit 402.

[0079] The generation unit 402 generates a response to the prompt (the evaluation result of the utterance content in the mock business negotiation) by inputting the prompt output by the acquisition unit 401 into the generated AI model stored in the AI ​​model storage unit 451. The generation unit 402 transmits the generated response to the mock business negotiation evaluation device 20 via the communication device 1004 of the generation device 40.

[0080] The AI ​​model storage unit 451 stores a generated AI model that outputs a response to a prompt when the prompt is input.

[0081] <prompt> Next, an example of a prompt generated by the instruction unit 203 of the simulated business negotiation evaluation device 20 will be described.

[0082] 7 is a diagram showing an example of a prompt according to this embodiment. The exemplary prompt 700 shown in FIG. 7 is generated by the instructing unit 203 of the simulated business negotiation evaluation device 20 based on various information acquired from the knowledge DB 361.

[0083] Information 701 included in prompt 700 is information indicating the customer's values. Information 702 included in prompt 700 is information about the customer's attributes and personality. Information 703 included in prompt 700 is information about the interaction between the generative AI model and the sales representative user in the mock business negotiation. Information 704 included in prompt 700 is information about the scoring criteria, information instructing scoring, and information instructing the creation of a report.

[0084] <Evaluation results> Next, an example of an evaluation result generated by the generating unit 402 of the generating device 40 will be described.

[0085] Fig. 8 is a diagram showing an example of an evaluation result according to this embodiment. The exemplary evaluation result 800 shown in Fig. 8 is generated by the generation unit 402 of the generation device 40 based on the prompt 700 shown in Fig. 7, transmitted from the generation device 40 to the simulated business-negotiation evaluation device 20, and transmitted from the simulated business-negotiation evaluation device 20 to the user device 10. The evaluation result 800 includes a scoring result portion 801 and a report portion 802 generated in accordance with the prompt 700.

[0086] <Operation of the mock sales negotiation evaluation system> Next, an example of the operation of the mock negotiation evaluation system 1 will be described with reference to FIG.

[0087] FIG. 9 is a sequence diagram showing an example of the operation of the mock negotiation evaluation system according to this embodiment.

[0088] In step S101, the user device 10 (request sending unit 101) sends an evaluation result generation request to the simulated negotiation evaluation device 20 in response to the sales representative user pressing the evaluation result generation request button.

[0089] In step S102, the mock business negotiation evaluation device (acquisition unit 201) searches the knowledge DB 361 based on the received evaluation result generation request to acquire various pieces of information related to the mock business negotiation evaluation.

[0090] In step S103, the mock business negotiation evaluation device (acquisition unit 201) acquires setting information related to the mock business negotiation from various types of information.

[0091] In step S104, the simulated business negotiation evaluation device (setting unit 202) sets information about the customer role generation AI model based on the setting information about the business negotiation.

[0092] In step S105, the simulated business negotiation evaluation device (instruction unit 203) instructs the generation AI model to evaluate the utterance content based on information regarding the utterance content in the simulated business negotiation between the generation AI model playing the customer role and the sales representative user (sends an utterance content evaluation instruction (prompt) to the generation device 40). The simulated business negotiation evaluation device (instruction unit 203) may also instruct the generation AI model to evaluate the utterance content based on information regarding the scoring criteria obtained from various information.

[0093] In step S106, the generation device 40 (generation unit 402) inputs the prompt into the generative AI model and generates an evaluation result that is a response corresponding to the prompt.

[0094] In step S107, the generation device 40 (generation unit 402) transmits the evaluation result to the simulated negotiation evaluation device 20.

[0095] In step S108, the simulated negotiation evaluation device 20 (the evaluation result processing unit 204) transmits the evaluation result to the user device 10.

[0096] In step S109, the user device 10 (output unit 151) outputs the evaluation result to the outside.

[0097] As described above, according to an embodiment of the present disclosure, the simulated business negotiation evaluation device 20 sets information about a generative AI model for a customer role and instructs the generative AI model for a customer role to evaluate the content of utterances made during a business negotiation between the thus-set generative AI model for a customer role and a sales representative user. In this way, by having the generative AI model act as a customer, the content of utterances can be appropriately evaluated. Since customer feedback is very important in business, the results of this evaluation are even more valuable information for sales representatives.

[0098] <Modification> A modification of the above embodiment will now be described.

[0099] The mock business negotiation evaluation device 20 may provide a mock business negotiation with the generative AI model to the sales representative user of the user device 10. As a result, the interaction between the generative AI model and the sales representative user in the mock business negotiation may be stored in the knowledge DB 361. Then, immediately after the mock business negotiation is conducted, the mock business negotiation evaluation device 20 may transmit the evaluation results of the utterance content to the user device 10, as described in the above embodiment.

[0100] Some or all of the functions of the user device 10, the simulated business-negotiation evaluation device 20, the DB device 30, and / or the generation device 40 may be included in other devices. For example, some or all of the above-mentioned functions of the simulated business-negotiation evaluation device 20, the DB device 30, and / or the generation device 40 may be included in the user device 10. For example, the user device 10 may include the acquisition unit 201, the setting unit 202, and the instruction unit 203 of the simulated business-negotiation evaluation device 20, the knowledge DB storage unit 351 of the DB device 30, or the generation unit 402 of the generation device 40. Furthermore, for example, the simulated business-negotiation evaluation device 20 may include the generation unit 402 of the generation device 40.

[0101] The names of the above-mentioned devices (user device 10, simulated business negotiation evaluation device 20, etc.) and functional units (request sending unit 101, acquisition unit 201, etc.) constituting the devices are merely examples and may be changed as appropriate. Furthermore, two or more functional units constituting the devices may be combined and integrated, or one functional unit may be divided into multiple functional units.

[0102] The order of steps in the sequence diagrams shown above is merely an example and may be changed as appropriate, or some steps may be performed in parallel.

[0103] As described above, according to one aspect of the present disclosure, the content of dialogue in role-playing can be appropriately evaluated.

[0104] <Summary> An information processing device according to one embodiment of the present disclosure includes an acquisition unit that acquires setting information related to a business negotiation, a setting unit that sets information related to a generative AI model for a customer role based on the setting information related to the business negotiation, and an instruction unit that instructs the generative AI model to evaluate the content of an utterance based on information related to the content of an utterance during a business negotiation between the generative AI model for the customer role and a person.

[0105] In one example, the instruction unit instructs the generative AI model to score the speech content further based on information regarding scoring criteria.

[0106] In one example, the instruction unit instructs the generative AI model to score the speech content and to create a report based on the results of scoring the speech content.

[0107] In one example, the instruction unit instructs the generative AI model to create a report including a sentence based on the sentence according to the result of scoring the speech content.

[0108] In one example, the instruction unit instructs the generative AI model to score the utterance content for each of the multiple evaluation items based on information regarding the utterance content and multiple scoring criteria corresponding to each of the multiple evaluation items.

[0109] In one example, the instruction unit instructs the generative AI model to evaluate the content of the utterance based on example correct answers to the question and the answers to the questions included in the information related to the content of the utterance.

[0110] In one example, the instruction unit instructs the generative AI model to evaluate the content of the utterance based on the requests made in the business negotiation and the answers to the requests contained in the information related to the content of the utterance.

[0111] An information processing method according to one embodiment of the present disclosure includes an information processing device that acquires setting information related to a business negotiation, sets information related to a generative AI model for a customer role based on the setting information related to the business negotiation, and instructs the generative AI model to evaluate the content of the utterance based on information related to the content of the utterance during the business negotiation between the generative AI model for the customer role and a person.

[0112] A program according to one embodiment of the present disclosure is a program for causing a computer to perform the following steps: acquire setting information regarding a business negotiation; set information regarding a generative AI model for a customer role based on the setting information regarding the business negotiation; and instruct the generative AI model to evaluate the content of an utterance based on information regarding the content of an utterance during a business negotiation between the generative AI model for the customer role and a person.

[0113] <Other> In the above-described embodiment, ROM and RAM are exemplified as storage device 1001, but other suitable storage media may also be used, such as flexible disks, magneto-optical disks (e.g., compact disks, digital versatile disks, Blu-ray (registered trademark) discs), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy (registered trademark) disks, magnetic strips, databases, servers, or other suitable storage media.

[0114] In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0115] In the above-described embodiment, input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added. Output information may be deleted. Input information may be transmitted to another device.

[0116] In the above-described embodiment, the determination may be made by a value (0 or 1) represented using one bit, by a Boolean value (true or false), or by a comparison of a numerical value (e.g., a comparison with a predetermined value).

[0117] The order of the illustrated procedures, sequences, flowcharts, etc. in the above-described embodiments may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0118] Each function illustrated in the above-described embodiments is realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, by wire, wirelessly, etc.) and these multiple devices. A functional block may also be realized by combining software with the single device or the multiple devices.

[0119] The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0120] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0121] In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0122] The information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information.

[0123] In the above-described embodiments, the user device 10, the simulated business opportunity evaluation device 20, the DB device 30, and the generation device 40 may be mobile stations (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other appropriate term. In the present disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0124] In the above-described embodiments, the terms "connected," "coupled," or any variation thereof refers to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read with "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using one or more wires, cables, and / or printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0125] In the above-described embodiments, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0126] The terms "determining" and "determining" as used in this disclosure may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0127] In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or," as used in this disclosure, is not intended to be an exclusive or.

[0128] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0129] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0130] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Furthermore, notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0131] Although the embodiments have been described above with reference to the drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims. It is understood that such modifications or alterations also fall within the technical scope of the present disclosure. Furthermore, the components or features in the embodiments (including modifications or alterations) may be combined in any manner without departing from the spirit of the present disclosure. [Explanation of symbols]

[0132] 1. Mock negotiation evaluation system 10 User Devices 101 Request sending unit 102 Evaluation result acquisition unit 103 Output control section 151 Output section 20. Business negotiation simulation evaluation device 201 Acquisition Department 202 Settings 203 Instruction section 204 Evaluation result processing unit 30 DB device 351 Knowledge DB storage section 361 Knowledge DB 40 Generator 401 Acquisition Department 402 Generator 451 AI model memory unit

Claims

1. an acquisition unit for acquiring setting information related to the business negotiation; A setting unit that sets information about a customer role generation AI model based on setting information about the business negotiation; An instruction unit that instructs the generative AI model to evaluate the utterance content based on information regarding the utterance content in a business negotiation between the generative AI model of the customer role and a person; An information processing device comprising:

2. The instruction unit instructs the generative AI model to score the utterance content based on information about the utterance content and information about scoring criteria. The information processing device according to claim 1 .

3. The instruction unit instructs the generative AI model to score the utterance content based on information about the utterance content and information about the scoring criteria, and to create a report according to the result of scoring the utterance content. The information processing device according to claim 2 .

4. The instruction unit instructs the generative AI model to score the utterance content for each of the plurality of evaluation items based on information about the utterance content and a plurality of scoring criteria corresponding to each of the plurality of evaluation items. The information processing device according to claim 1 .

5. The instruction unit instructs the generative AI model to evaluate the utterance content based on a correct answer example for the question and an answer to the question included in the information related to the utterance content. The information processing device according to claim 1 .

6. The instruction unit instructs the generative AI model to evaluate the utterance content based on the request items in the business negotiation and the answers to the request items included in the information related to the utterance content. The information processing device according to claim 1 .

7. The information processing device Get configuration information about the deal, Based on the setting information regarding the business negotiation, information regarding a customer role generation AI model is set, Instruct the generative AI model to evaluate the utterance content based on information regarding the utterance content during the business negotiation between the generative AI model of the customer role and the person. Information processing methods.

8. Obtaining configuration information regarding the business opportunity; Setting information about a customer role generation AI model based on the setting information about the business negotiation; Instructing the generative AI model to evaluate the utterance content based on information regarding the utterance content in a business negotiation between the generative AI model of the customer role and a person; A program that causes a computer to execute the following.

Images