System

Abstract

To provide a system.SOLUTION: A system includes the means of: acquiring a voice input of a user from a smart device a stuff member wears, and converting voice data to text data, using a voice recognition system; transmitting the converted text data to a generative AI, and generating proper responses on the basis of the tex data; using a voice synthesis system to convert the response to voice data; analyzing reactions to the response by an emotion engine; evaluating properness of the response on the basis of an emotion state of the user obtained from an analysis result of the emotion engine, and providing the stuff member with feedback.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] In VR simulations for the hospitality industry, patterns must be created individually for each business type, making it difficult to create a large number of patterns. [Means for solving the problem]

[0005] By incorporating generative AI, a complainer model can be created that can interactively respond to the subject's comments. It is also possible to evaluate the subject's response. In addition to the content of the speech, accuracy can be improved by including biological information such as the subject's gaze and tone of voice in the response generation / evaluation. [Brief description of the drawings]

[0006] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. [Figure 19] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system according to the embodiment 2 of the second embodiment in the case where an emotion engine is combined. [Figure 20] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when combined with an emotion engine. [Figure 21] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the embodiment 3 of the form example 3 in the case where an emotion engine is combined. [Figure 22] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, a generative AI is incorporated into a VR simulation system for the customer service industry. This generative AI creates a complaint model that can interactively respond to the test subject's comments. Specifically, when the test subject says, "Please tell me the details of this product," the generative AI generates a response such as, "This product is ~~~ and has ~~~ characteristics."

[0029] "Example 2"

[0030] Furthermore, in one embodiment of the present invention, the subject's response is evaluated. This evaluation is based on the appropriateness of the response generated by the generative AI and the subject's reaction. For example, if the subject responds with "Thank you, that's enough," the response is highly evaluated.

[0031] "Example 3"

[0032] In one embodiment of the present invention, biological information such as the subject's gaze and tone of voice is included in the response generation / evaluation. This makes it possible to more accurately grasp the subject's emotional state and intentions and generate a response accordingly. For example, if the subject's tone of voice becomes higher, it is determined that the subject is in an excited state and an appropriate response is generated.

[0033] The process flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: Launch the VR simulation system for hospitality.

[0036] Step 2: Incorporate generative AI. This AI creates a complaint model that can respond interactively to the subject's comments.

[0037] Step 3: If the subject says, "Please tell me more about this product," the generative AI generates a response such as, "This product is ~~ and has ~~ characteristics."

[0038] "Example 2"

[0039] Step 1: Evaluate the appropriateness of the response generated by the generative AI.

[0040] Step 2: Evaluate the subject's response. For example, if the subject responds, "Thank you, that's enough," the response is rated high.

[0041] "Example 3"

[0042] Step 1: Obtain biometric information such as the subject's gaze and tone of voice.

[0043] Step 2: Include the acquired biometric information in the response generation / evaluation, which allows for a more accurate understanding of the subject's emotional state and intentions.

[0044] Step 3: If the subject's voice tone becomes higher, determine that the subject is excited and generate an appropriate response.

[0045] Example 1

[0046] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0047] Conventional simulation systems for the customer service industry have a problem in that responses to the subjects' comments are fixed, making interactive responses difficult. In addition, evaluations do not take into account the subjects' biometric information, making it difficult to simulate a real customer service situation. Furthermore, there is an issue that practical training is difficult because responses cannot be generated in real time.

[0048] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for incorporating a generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for converting the subject's remarks into text data using a voice recognition technology, a means for generating a prompt sentence in the generative AI model and generating an appropriate response, a means for converting the generated response text into voice data using a voice synthesis technology, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, and a means for providing the generated voice data to the subject. This enables interactive response in real time to the subject's remarks, and realizes a simulation close to an actual customer service situation. Furthermore, since evaluation can be performed taking biological information into account, more practical training is possible.

[0049] "Generative AI" is an artificial intelligence technology that interactively generates responses to what a subject says.

[0050] The "complainer model" is a model for simulating a complaint in response to a statement made by a subject.

[0051] "Voice recognition technology" is a technology that converts voice data into text data.

[0052] A "prompt sentence" is an instruction sentence that causes a generative AI model to generate an appropriate response.

[0053] "Speech synthesis technology" is a technology that converts text data into voice data.

[0054] "Biometric information" is information obtained from the subject's body, such as their gaze or tone of voice.

[0055] An "evaluation means" is a means for evaluating the subject's response.

[0056] "Interactive" refers to a state in which two-way communication is possible.

[0057] "Real time" refers to a state in which processing is carried out immediately without delay.

[0058] This invention provides a customer service industry VR simulation system that incorporates generative AI to provide a customer complaint model that responds interactively to the comments of test subjects. A specific embodiment of this system will be described below.

[0059] First, the device collects the subject's speech through a microphone. The collected voice data is converted into text data using voice recognition technology. Specifically, the voice data is converted into text data using Google (registered trademark) Speech-to-Text API. In this conversion process, the voice signal is analyzed and the corresponding text is generated.

[0060] Next, the server generates a prompt sentence for the generative AI model based on the text data received from the terminal. This prompt sentence includes instructions for the generative AI model to generate an appropriate response. For example, the server generates a prompt sentence such as "The user said, 'Please tell me the details of this product.' Please generate a response."

[0061] The server then inputs the prompt sentence into a generative AI model (e.g., GPT-4 (registered trademark)) to generate an appropriate response. The generative AI model analyzes the prompt sentence and generates a natural response according to the context. For example, it generates a response such as, "This product is made using the latest technology, and has features such as high durability and ease of use."

[0062] The generated response text is converted to voice data by the server using voice synthesis technology. Specifically, Amazon Polly is used to convert text data into voice data. In this conversion process, text data is converted into voice signals to generate natural voice.

[0063] Finally, the device plays back the voice data received from the server and provides the response to the subject, allowing the subject to hear the response generated by the generative AI model.

[0064] As a concrete example, the process is as follows when a subject says "Tell me more about this product" in a VR simulation. The device converts the subject's speech into text using the Google Speech-to-Text API, and the server generates a prompt saying, "The user said, 'Tell me more about this product.' Please generate a response." The generative AI model (GPT-4) generates a response saying, "This product is made using the latest technology, and is durable and easy to use," and the server converts this into voice data using Amazon Polly. Finally, the device plays back the voice data and provides the response to the subject.

[0065] In this way, the server, the terminal, and the user can cooperate to realize an interactive response.

[0066] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0067] Step 1: Audio collection and recognition

[0068] The device collects the subject's speech through a microphone and converts the collected voice data into text data using the Google Speech-to-Text API. This conversion process involves analyzing the voice signal and generating the corresponding text.

[0069] Specific behavior:

[0070] Input: Subject's speech (audio data)

[0071] Data processing: Send the voice data to the Google Speech-to-Text API and analyze the voice signal.

[0072] Output: Text data (e.g. "Please tell me the details of this product")

[0073] Step 2: Generate a prompt

[0074] The server generates a prompt sentence for the generative AI model based on the text data received from the terminal. The prompt sentence includes instructions for generating an appropriate response for the generative AI model.

[0075] Specific behavior:

[0076] Input: Text data (e.g. "Please tell me the details of this product")

[0077] Data processing: Analyze text data and generate prompts suitable for generative AI models

[0078] Output: Prompt statement (e.g. "The user says 'Tell me more about this product.' Generate a response.")

[0079] Step 3: Generate a response

[0080] The server inputs a prompt sentence into a generative AI model (e.g., GPT-4) and generates an appropriate response. The generative AI model analyzes the prompt sentence and generates a natural response according to the context.

[0081] Specific behavior:

[0082] Input: Prompt statement (e.g. "The user says 'Tell me more about this product.' Generate a response.")

[0083] Data computation: A generative AI model analyzes the prompt and generates an appropriate response

[0084] Output: Response text (e.g. "This product is made using the latest technology, is durable, and has easy-to-use features.")

[0085] Step 4: Text-to-speech response

[0086] The server converts the generated response text into voice data using speech synthesis technology. Specifically, it uses Amazon Polly to convert text data into voice data.

[0087] Specific behavior:

[0088] Input: Response text (e.g. "This product is made using the latest technology, is durable, and has easy-to-use features.")

[0089] Data processing: Send text data to Amazon Polly and convert it into voice data

[0090] Output: Audio data

[0091] Step 5: Providing a response

[0092] The device plays the voice data received from the server and provides the response to the subject, allowing the subject to hear the response generated by the generative AI model.

[0093] Specific behavior:

[0094] Input: Audio data

[0095] Data processing: Playing back audio data

[0096] Output: Audio response to the subject (e.g., "This product is made using the latest technology, is durable, and has easy-to-use features.")

[0097] In this way, the server, the terminal, and the user work together to realize interactive responses.

[0098] (Application example 1)

[0099] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] In conventional customer service work, it is difficult for store clerks to respond quickly and appropriately to questions and complaints from customers. It is especially difficult for new or inexperienced store clerks to maintain the quality of customer service, which can lead to a decrease in customer satisfaction and an increase in complaints. It is also difficult for store clerks to accurately grasp the emotional state of customers and respond accordingly. To solve these problems, a system is needed that allows store clerks to receive appropriate responses in real time.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0102] In this invention, the server includes a means for incorporating a generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for converting the customer's remarks into text using voice recognition, and a means for the generative AI to generate an appropriate response based on the customer's remarks and display it on a display device. This enables store clerks to respond quickly and appropriately to questions and complaints from customers, which is expected to improve customer satisfaction and reduce complaints.

[0103] "Generative AI" refers to artificial intelligence that automatically generates responses and content based on input data.

[0104] The "complainer model" is a simulation model for responding appropriately to complaints and questions from customers.

[0105] "Subject" means a person who is trained and evaluated using the system.

[0106] "Biometric information" refers to physical data such as a subject's gaze, tone of voice, heart rate, etc.

[0107] "Speech recognition" is a technology that converts speech into text data.

[0108] "Display device" means a device for visually displaying a generated response.

[0109] "Real-time" refers to immediate processing or response without delay.

[0110] "Customer satisfaction" refers to the degree of satisfaction a customer feels toward a service or product.

[0111] "Response generation" refers to the process of generating an appropriate response to a customer's utterance.

[0112] "Evaluation" refers to the process of measuring the quality of a subject's actions or responses and providing feedback.

[0113] A system for carrying out this invention incorporates a generative AI and creates a complainer model that can interactively respond to comments made by subjects. The system includes a means for evaluating the response of the subject, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for converting the customer's comments into text using voice recognition, and a means for the generative AI to generate an appropriate response based on the customer's comments and display it on a display device.

[0114] Hardware and Software Used

[0115] Hardware: Smart glasses, microphone

[0116] Software: OpenAI(R) API, SpeechRecognition library, Smart Glasses SDK

[0117] System processing overview

[0118] The server first uses a microphone to capture the customer's speech as audio. It then uses the SpeechRecognition library to convert the speech into text data. This text data is input into a generative AI (OpenAI API) that generates an appropriate response based on the customer's speech. The generated response is displayed on the smart glasses' display device, allowing the store clerk to check it in real time.

[0119] Examples

[0120] For example, if a customer asks, "Please tell me more about this product," the generative AI will generate a response such as, "This product uses the latest technology, is lightweight, and is easy to carry." This response is displayed on the smart glasses' display, and the store clerk can explain it to the customer while checking it.

[0121] Examples of prompt statements

[0122] Customer Question: What are the details of this product?

[0123] Salesperson's response: This product uses the latest technology, is lightweight and easy to carry.

[0124] In this way, the system helps store clerks to respond quickly and appropriately to customer inquiries and complaints, improving customer satisfaction and reducing complaints.

[0125] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0126] Step 1:

[0127] The server uses a microphone to capture the customer's speech as audio: the input is the customer's voice, and the output is audio data, which is converted to text in the next step.

[0128] Step 2:

[0129] The server converts the voice data into text data using the SpeechRecognition library. The input is the voice data obtained in step 1, and the output is text data. This text data is input to the generative AI.

[0130] Step 3:

[0131] The server inputs text data into the generative AI (OpenAI API) and generates an appropriate response based on the customer's utterance. The input is the text data obtained in step 2, and the output is the generated response text. This response text is displayed on a display device in the next step.

[0132] Step 4:

[0133] The server sends the generated response text to the display device of the smart glasses for display. The input is the response text obtained in step 3, and the output is the text displayed on the display of the smart glasses. This allows the user (store clerk) to provide an appropriate response to the customer in real time.

[0134] Step 5:

[0135] The user (store clerk) checks the response displayed on the smart glasses display and provides explanations and responses to the customer based on that. The input is the response text displayed in step 4, and the output is a verbal explanation or response to the customer. This is expected to improve customer satisfaction.

[0136] Example 2

[0137] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0138] Conventional systems using generative AI had insufficient means to evaluate the appropriateness of responses to the subject's comments and the subject's reactions, making it difficult to effectively generate and evaluate interactive responses. In addition, it was difficult to include biological information such as the subject's gaze and tone of voice in response generation and evaluation, making it difficult to accurately estimate the subject's emotional state. Furthermore, the process of the user inputting a reaction to a generated response and evaluating that reaction was complicated, which led to problems with not improving the user experience.

[0139] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0140] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for the user to input a prompt sentence through the terminal, a means for the terminal to transmit the input prompt sentence to the server, a means for the server to generate a response to the prompt sentence using the generative AI model, a means for the server to transmit the generated response to the terminal, a means for the terminal to display the generated response to the user, a means for the user to input a reaction to the generated response, a means for the terminal to transmit the user's reaction to the server, a means for the server to analyze and evaluate the user's reaction, a means for the server to transmit the evaluation result to the terminal, and a means for the terminal to display the evaluation result to the user. This enables interactive response generation and evaluation using generative AI, making it possible to accurately estimate the subject's emotional state and improve the user experience.

[0141] "Generative AI" is a system that uses artificial intelligence technology to automatically generate responses to user input.

[0142] The "complainer model" is a model that responds interactively to comments made by subjects and incorporates generative AI.

[0143] A "subject" refers to a user who makes statements and responds to the system.

[0144] "Evaluation" is the process of analyzing the appropriateness of the responses generated by the generative AI and the subjects' reactions to determine their quality and usefulness.

[0145] "Biometric information" refers to physical data such as the subject's gaze, tone of voice, and heart rate, which are used to generate and evaluate responses.

[0146] A "prompt sentence" refers to a question or instruction that a user inputs to a generative AI.

[0147] "Terminal" refers to a device through which a user can input prompts and display generated responses and evaluation results.

[0148] "Server" refers to a computer system that executes a generative AI model, generates responses to prompts, and evaluates them.

[0149] "Response" refers to the reply or reaction that a generative AI generates based on a prompt.

[0150] "Analysis" refers to the process of processing the user's reactions and biometric information as data and understanding and evaluating its content.

[0151] "Evaluation Results" refers to the conclusions or feedback obtained after the Server analyzes the User's reactions and generated responses.

[0152] This invention is a system that uses generative AI to generate responses to utterances by subjects and evaluates the appropriateness of the responses and the reactions of the subjects. A specific embodiment of this system will be described below.

[0153] Hardware and Software Used

[0154] The server uses a high-performance computing system equipped with an NVIDIA GPU. For software, it uses deep learning frameworks such as TENSORFLOW (registered trademark) and PyTorch to execute generative AI models. It also uses natural language processing (NLP) libraries spaCy and NLTK to analyze data.

[0155] The terminal is a device for a user to input a prompt sentence and display the generated response and the evaluation result. The terminal can be a general personal computer or a smartphone.

[0156] System Operation

[0157] 1. The user enters a prompt through the terminal.

[0158] The user enters a prompt sentence into the input field of the terminal, for example, "If the generating AI responded, 'Thank you, that's enough,' how would you rate that response?"

[0159] 2. The terminal sends the prompt text entered to the server

[0160] The terminal sends the entered prompt text to the server, using HTTP or WebSocket as the communication protocol.

[0161] 3. The server uses the generative AI model to generate a response to the prompt.

[0162] The server runs a generative AI model built using TensorFlow and PyTorch to generate a response to the prompt, such as "That's a very appropriate response."

[0163] 4. The server generates a response and sends it to the device.

[0164] The server then sends the generated response to the terminal, again using HTTP or WebSocket as the communication protocol.

[0165] 5. The terminal displays the generated response to the user.

[0166] The terminal displays the response received from the server to the user. For example, it displays "That response is very appropriate" on the screen.

[0167] 6. The user enters a response to the generated response

[0168] The user enters their reaction to the generated response, for example, "Thanks, that's good enough."

[0169] 7. The device sends the user's response to the server

[0170] The terminal sends the user's response to the server, again using HTTP or WebSocket as the communication protocol.

[0171] 8. The server analyzes the user's response and gives an evaluation.

[0172] The server uses natural language processing libraries (e.g., spaCy or NLTK) to analyze the user's response and rate its appropriateness. For example, a response like "Thanks, that's enough" is highly rated.

[0173] 9. The server sends the evaluation results to the device.

[0174] The server then transmits the evaluation results to the terminal, again using HTTP or WebSocket as the communication protocol.

[0175] 10. The terminal displays the evaluation results to the user.

[0176] The terminal displays the evaluation result received from the server to the user. For example, it displays "Your response is very appropriate" on the screen.

[0177] In this way, a system is realized in which the server, terminal, and user work together to generate and evaluate responses using a generative AI model. This system enables interactive response generation and evaluation using generative AI, accurately estimating the subject's emotional state and improving the user experience.

[0178] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0179] Step 1:

[0180] The user enters a prompt through the terminal.

[0181] Specifically, the user inputs a prompt sentence into the input field of the terminal. For example, the user inputs, "If the generating AI responded, 'Thank you, that's enough,' how would you evaluate that response?" The input data is the prompt sentence in text format.

[0182] Step 2:

[0183] The terminal sends the prompt text entered to the server.

[0184] Specifically, the terminal sends the input prompt text to the server using HTTP or WebSocket. The input data is the prompt text entered by the user, and the output data is the prompt text sent to the server.

[0185] Step 3:

[0186] The server uses a generative AI model to generate a response to the prompt.

[0187] Specifically, the server runs a generative AI model built using TensorFlow and PyTorch to generate a response to the prompt. The input data is the prompt, and the output data is the generated response. For example, it generates a response such as "That response is very appropriate."

[0188] Step 4:

[0189] The server sends the generated response to the terminal.

[0190] Specifically, the server sends the generated response to the terminal using HTTP or WebSocket. The input data is the generated response, and the output data is the response sent to the terminal.

[0191] Step 5:

[0192] The terminal displays the generated response to the user.

[0193] Specifically, the terminal displays the response received from the server to the user. The input data is the response sent from the server, and the output data is the response displayed to the user. For example, "That response is very appropriate" is displayed on the screen.

[0194] Step 6:

[0195] The user enters a reaction to the generated response.

[0196] As a specific operation, the user inputs his / her reaction to the generated response. The input data is the user's reaction, and the output data is the reaction input to the terminal. For example, the user inputs "Thank you, that's enough."

[0197] Step 7:

[0198] The device transmits the user's response to the server.

[0199] Specifically, the terminal transmits the user's response to the server using HTTP or WebSocket. The input data is the user's response, and the output data is the response transmitted to the server.

[0200] Step 8:

[0201] The server analyzes the user's response and provides an evaluation.

[0202] Specifically, the server uses a natural language processing library (e.g., spaCy or NLTK) to analyze the user's response and evaluate its appropriateness. The input data is the user's response, and the output data is the evaluation result. For example, a response like "Thanks, that's enough" is highly evaluated.

[0203] Step 9:

[0204] The server transmits the evaluation results to the terminal.

[0205] Specifically, the server sends the evaluation results to the terminal using HTTP or WebSocket. The input data is the evaluation results, and the output data is the evaluation results sent to the terminal.

[0206] Step 10:

[0207] The terminal displays the evaluation results to the user.

[0208] As a specific operation, the terminal displays the evaluation result received from the server to the user. The input data is the evaluation result sent from the server, and the output data is the evaluation result displayed to the user. For example, "Your response is very appropriate" is displayed on the screen.

[0209] (Application example 2)

[0210] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0211] Conventional customer service systems did not provide sufficient support for store clerks to respond appropriately to customers, making it difficult to improve customer satisfaction. In addition, there was a lack of means to evaluate the appropriateness of responses based on customer reactions, making it difficult to improve the response skills of store clerks. Furthermore, there was a need for a system that could efficiently handle customer service in physical stores.

[0212] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0213] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for generating responses to support customer service in a physical store, a means for evaluating the appropriateness of the response based on the customer's reaction, and a means including an application to be installed on a smartphone. This enables support for store clerks to respond appropriately to customers, thereby improving customer satisfaction and the response skills of store clerks.

[0214] "Generative AI" is a system that uses artificial intelligence technology to generate appropriate responses to user input.

[0215] The "complainer model" is a model for appropriately responding to complaints and inquiries that arise in customer service.

[0216] "Subject" refers to a user who participates in the evaluation and testing of the system.

[0217] "Biometrics" refers to physical data such as a subject's gaze, tone of voice, heart rate, etc.

[0218] "Brick and mortar store" refers to a sales or service location that is located in a physical location.

[0219] "Customer service" refers to the way store clerks respond to and provide services to customers.

[0220] "Response appropriateness" is the measure of how well the generated response meets the customer's requirements and expectations.

[0221] A "smartphone" is a multi-function mobile device that, in addition to the functions of a mobile phone, can connect to the Internet and run applications.

[0222] An "Application" is a software program designed to provide a particular function or service.

[0223] A system for implementing this invention includes a means for incorporating a generative AI and creating a complainer model that can interactively respond to comments made by a subject, a means for evaluating the subject's response, a means for including biometric information such as the subject's gaze and tone of voice in response generation / evaluation, a means for generating responses to assist with customer service in physical stores, a means for evaluating the appropriateness of the response based on the customer's reaction, and a means for including an application to be installed on a smartphone.

[0224] System Programming

[0225] This system is implemented using Python. OpenAI's GPT-3 (registered trademark) is used as the generative AI model. The server receives the prompt entered by the store clerk and sends it to the generative AI model to generate a response. The generated response is displayed on the smartphone, providing assistance to the store clerk in providing an appropriate response to the customer.

[0226] Processing Description

[0227] The server receives the prompt entered by the store clerk and sends it to the generative AI model using the OpenAI API. The generative AI model generates an appropriate response based on the prompt and returns the response to the server. The server sends the generated response to the smartphone, and the store clerk responds to the customer. The customer's response is collected through the smartphone's microphone and camera and sent to the server. The server evaluates the appropriateness of the response based on the customer's response and provides feedback to the store clerk.

[0228] Hardware and Software Used

[0229] Hardware: Smartphones, servers

[0230] Software: Python, OpenAI API

[0231] Examples

[0232] For example, if a sales associate types, "Tell me what products you're looking for," the generative AI model responds with, "What kind of products are you looking for? For example, clothing or electronics?" If the customer responds, "Thanks, that's fine," this response is scored as "highly rated."

[0233] Example prompt:

[0234] "Tell us what product you're looking for."

[0235] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0236] Step 1:

[0237] The store clerk enters the prompt text into his / her smartphone.

[0238] Input: Prompt text to be entered by the salesperson (e.g. "Tell me about the product you are looking for.")

[0239] Output: The prompt text is sent to the server.

[0240] Specific actions: The store clerk opens the application on his / her smartphone, enters a prompt in the text box, and presses the send button.

[0241] Step 2:

[0242] The server receives the prompt text and sends it to the generative AI model.

[0243] Input: Prompt text sent from smartphone

[0244] Output: The prompt sent to the generative AI model

[0245] Specific operation: The server creates a request to send the prompt text received from the smartphone to the OpenAI API and sends it to the API endpoint.

[0246] Step 3:

[0247] A generative AI model generates a response based on the prompt.

[0248] Input: Prompt text sent from the server

[0249] Output: The generated response text

[0250] Specific operation: The generative AI model (e.g. GPT-3) analyzes the received prompt and generates an appropriate response, which is returned to the server as an API response.

[0251] Step 4:

[0252] The server receives the generated response and sends it to the smartphone.

[0253] Input: Response text returned by the generative AI model

[0254] Output: Response text sent to smartphone

[0255] Specific operation: The server analyzes the response text received from the generative AI model and creates a message to send to the smartphone. The message is sent to the smartphone application.

[0256] Step 5:

[0257] The smartphone displays the response text, and the store clerk responds to the customer.

[0258] Input: The response text sent by the server

[0259] Output: The response the salesperson gives to the customer

[0260] Specific operation: The smartphone application displays the received response text on the screen. The store clerk uses the displayed response as a reference to respond appropriately to the customer.

[0261] Step 6:

[0262] The smartphone collects customer responses and sends them to the server.

[0263] Input: Customer response (voice, gaze, facial expression, etc.)

[0264] Output: Customer response data sent to the server

[0265] How it works: The smartphone's microphone and camera are used to collect customer responses, and the collected data is sent to a server.

[0266] Step 7:

[0267] The server evaluates the appropriateness of the response based on the customer's reaction and provides feedback to the store attendant.

[0268] Input: Customer response data

[0269] Output: Response appropriateness assessment and feedback

[0270] Specific operation: The server analyzes the received customer response data and evaluates the appropriateness of the response. The evaluation result is sent to the store clerk's smartphone as feedback.

[0271] Example 3

[0272] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0273] In conventional interactive systems, responses to the subject's utterances were uniform, making it difficult to accurately grasp the subject's emotional state and intentions. As a result, it was not possible to generate appropriate responses according to the subject's emotions, resulting in a poor user experience. Furthermore, it was not possible to utilize biological information such as the subject's gaze or tone of voice, resulting in low response accuracy.

[0274] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0275] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for collecting gaze data and tone of voice data of the subject using a high-resolution camera and microphone, an image processing means for analyzing the collected gaze data, a voice analysis means for analyzing the collected tone of voice data, and a means for grasping the emotional state and intention of the subject based on the analyzed gaze data and tone of voice data and generating an appropriate response. This makes it possible to accurately grasp the emotional state and intention of the subject and generate an appropriate response accordingly.

[0276] "Generative AI" is an artificial intelligence system that generates responses in real time based on the subject's statements and biometric information.

[0277] The "complainer model" is a model that responds interactively to comments made by subjects, and is constructed using generative AI.

[0278] "Biometric information" is information obtained from a subject's body, such as their gaze or tone of voice, and is used to understand their emotional state and intentions.

[0279] A "high-resolution camera" is a camera for collecting gaze data from subjects, and is a device that has high resolution and can obtain detailed gaze information.

[0280] A "microphone" is a voice input device for collecting tone data of the subject's voice.

[0281] "Image processing means" refers to technology for analyzing collected gaze data and identifying the subject's gaze direction and movement.

[0282] The "voice analysis means" is a technology for analyzing collected voice tone data and detecting changes in the subject's voice tone.

[0283] The "response generation means" is a technology for understanding the subject's emotional state and intentions based on analyzed gaze data and tone of voice data, and generating an appropriate response.

[0284] This invention is a system that incorporates generative AI and can interactively respond to the subject's comments. Specifically, it is a system that collects biological information such as the subject's gaze and tone of voice, and generates / evaluates a response based on that information. A specific embodiment of this system is described below.

[0285] Hardware and Software Used

[0286] Hardware:

[0287] High-resolution camera: Used to collect subject's gaze data.

[0288] Microphone: Used to collect subject's vocal tone data.

[0289] Server: Equipped with high-performance CPU and GPU, it performs data analysis and response generation.

[0290] software:

[0291] OpenCV: An image processing library for analyzing gaze data.

[0292] PyAudio: An audio processing library for analyzing vocal tone data.

[0293] TensorFlow: A machine learning framework for running generative AI models.

[0294] System processing flow

[0295] Data collection:

[0296] The server activates a high-resolution camera to capture the user's face. The camera tracks the user's eye movements in real time to collect gaze data. The server also activates a microphone to record the user's voice. The microphone collects data for analyzing the tone of the user's voice in real time. The user speaks naturally into the camera and microphone while interacting with the system.

[0297] Data Analysis:

[0298] The server uses OpenCV to analyze the video data obtained from the camera. Specifically, it detects the position of the user's pupils and identifies the direction of gaze. For example, if the user is looking at the top right of the screen, that direction is identified. The server uses PyAudio to analyze the recorded audio data. Specifically, it analyzes the frequency components of the audio and detects changes in the tone of the voice. For example, if the tone of the user's voice becomes higher, it is determined that the user is excited.

[0299] Response generation:

[0300] The server uses TensorFlow to input gaze data and tone of voice data into the generative AI model. The generative AI model understands the user's emotional state and intentions based on this data. For example, if it determines that the user is excited, it generates a response that corresponds to that state. The server uses the generative AI model to generate a response appropriate to the user's state. For example, if the user is excited, it generates a response to respond calmly and chooses words to calm the user.

[0301] Examples of concrete examples and prompts

[0302] Examples:

[0303] Suppose a user is using an online customer support system. When the user asks a question, the system analyzes the user's gaze and tone of voice to determine whether the user is excited. For example, if the user's tone of voice becomes higher, the system determines that the user is excited and generates a response to help the user stay calm.

[0304] Example prompt:

[0305] "When a user is using your online customer support system, analyze their gaze and tone of voice to understand their emotional state and generate a response accordingly."

[0306] In this way, the server can generate a response based on the biometric information of the user, and can take a more accurate and appropriate action. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0307] Step 1:

[0308] Data collection

[0309] The server activates a high-resolution camera and microphone to collect the user's gaze data and tone of voice data. Specifically, the camera captures the user's face and tracks eye movements in real time, and the microphone records the user's voice and collects data for real-time tone of voice analysis.

[0310] Input: User's face video and audio

[0311] Output: Gaze data and tone of voice data

[0312] Step 2:

[0313] Analysis of gaze data

[0314] The server uses OpenCV to analyze the video data obtained from the camera. Specifically, it detects the position of the user's pupils and identifies the direction of the user's gaze. For example, if the user is looking at the upper right corner of the screen, that direction is identified.

[0315] Input: Gaze data

[0316] Output: Gaze direction information

[0317] Step 3:

[0318] Analysis of voice tone data

[0319] The server uses PyAudio to analyze the recorded voice data. Specifically, it analyzes the frequency components of the voice and detects changes in the tone of the voice. For example, if the user's voice tone becomes higher, it is determined that the user is excited.

[0320] Input: Voice tone data

[0321] Output: Voice tone change information

[0322] Step 4:

[0323] Emotional state estimation

[0324] The server understands the user's emotional state and intentions based on the analyzed gaze data and tone of voice data. Specifically, it determines whether the user is excited by combining gaze direction information and tone of voice change information.

[0325] Input: Gaze direction information, voice tone change information

[0326] Output: Emotional state information

[0327] Step 5:

[0328] Response Generation

[0329] The server uses TensorFlow to input emotional state information into a generative AI model to generate an appropriate response. For example, if the server determines that the user is excited, it generates a response according to that state.

[0330] Input: Emotional state information

[0331] Output: Response message

[0332] Step 6:

[0333] Providing a response

[0334] The server provides the generated response message to the user, either by displaying it in text form or by responding audibly using speech synthesis.

[0335] Input: Response message

[0336] Output: Response to the user

[0337] In this way, the server can generate a response based on the user's biometric information, enabling more accurate and appropriate action to be taken.

[0338] (Application example 3)

[0339] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0340] Conventional customer service systems have the problem that it is difficult to accurately grasp the emotional state and intentions of customers, and they are unable to generate appropriate responses. In addition, there is a lack of technology that utilizes biometric information such as gaze and tone of voice, and effective means to improve customer satisfaction are required. Furthermore, when introducing customer service robots into brick-and-mortar stores, flexible responses according to the customer's emotions and intentions are required.

[0341] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0342] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for grasping the subject's emotional state and intention and generating a response accordingly, and a means for being installed in a customer service robot in a brick-and-mortar store. This makes it possible to accurately grasp the customer's emotional state and intention and generate the optimal response accordingly.

[0343] "Generative AI" is an artificial intelligence system that interactively generates responses based on the subjects' statements and actions.

[0344] The "complainer model" is a model for generating appropriate responses to the statements or actions of customers or subjects.

[0345] An "evaluation means" is a means for evaluating the subject's response or behavior.

[0346] "Biometric information" is information obtained from the subject's body, such as their gaze or tone of voice.

[0347] The "response generating means" is a means for generating an appropriate response based on the subject's utterances and biometric information.

[0348] "Emotional state" refers to the subject's current emotional state.

[0349] "Intention" refers to what the subject is thinking and intending to do.

[0350] A "brick and mortar store" is a store that exists in a physical location.

[0351] A "customer service robot" is a robot that provides service to customers in a physical store.

[0352] A system for implementing this invention includes a means for incorporating generative AI and creating a complainer model that can interactively respond to comments made by a subject, a means for evaluating the subject's response, a means for including biometric information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for grasping the subject's emotional state and intentions and generating a response accordingly, and a means for being installed in a customer service robot in a physical store.

[0353] System Programming

[0354] The program of this system uses the following hardware and software:

[0355] Hardware: Camera (for gaze detection), microphone (for voice recognition)

[0356] Software: OpenCV (for face recognition and gaze detection), SpeechRecognition (for voice recognition), Transformers (for sentiment analysis)

[0357] Processing flow

[0358] The server captures video and audio in real time using a camera and microphone. It uses OpenCV to detect faces and eyes from the camera footage, and SpeechRecognition to convert the audio captured from the microphone into text. The converted text is analyzed for sentiment using a generative AI model from Transformers. Based on the analysis results, an appropriate response is generated.

[0359] Examples

[0360] For example, if a customer is excited, it is determined that the customer's voice tone is high and their gaze is focused on the product. In this case, the customer service robot will respond, "Sir, what product are you looking for today?" On the other hand, if a customer is in trouble, it is determined that the customer's voice tone is low and their gaze is unsteady. In this case, the customer service robot will respond, "If there's anything I can help you with, I'd like to help you."

[0361] Examples of prompt statements

[0362] "Generate responses where the customer's voice tone is high and their gaze is focused on the product."

[0363] In this way, customer service robots in physical stores will be able to accurately grasp the emotional state and intentions of customers and provide the most appropriate response accordingly.

[0364] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0365] Step 1:

[0366] The server uses a camera and microphone to capture video and audio in real time.

[0367] Input: Camera video, microphone audio

[0368] Output: Video data, audio data

[0369] Specific operation: Captures the customer's face and gaze through a camera, and records what the customer says through a microphone.

[0370] Step 2:

[0371] The server uses OpenCV to detect faces and eyes from camera footage.

[0372] Input: Video data

[0373] Output: Face and eye position information

[0374] Specific operation: Apply OpenCV's facial recognition algorithm to identify the face and eye positions from video data.

[0375] Step 3:

[0376] The server uses SpeechRecognition to convert the audio data into text.

[0377] Input: Audio data

[0378] Output: Text data

[0379] What it does: Apply a speech recognition algorithm to convert the audio data into text.

[0380] Step 4:

[0381] The server uses Transformers' generative AI models to parse sentiment from text data.

[0382] Input: Text data

[0383] Output: Emotion analysis results

[0384] How it works: Input text data into a generative AI model, and output an emotional state (e.g. positive, negative, neutral).

[0385] Step 5:

[0386] The server generates an appropriate response based on the face and eye position information and emotion analysis results.

[0387] Input: Face and eye position information, emotion analysis results

[0388] Output: Response text

[0389] Specific operation: Determine the direction of gaze from face and eye position information, combine it with emotion analysis results, and generate an appropriate response using a generative AI model.

[0390] Step 6:

[0391] The server transmits the generated response text to a customer service robot, which provides a response to the customer.

[0392] Input: Response text

[0393] Output: Audio response

[0394] Specific operation: The response text is input to the speech synthesis system and output as a voice. The customer service robot plays this voice and provides the response to the customer.

[0395] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0396] "Example 1"

[0397] One embodiment of the present invention is to incorporate an emotion engine into the generative AI. This emotion engine recognizes the emotional state of the subject from the tone of voice, facial expression, eye movement, etc., and feeds this information back to the generative AI. The generative AI generates a response to the subject based on this emotion information. For example, if the subject shows an emotion of anger, the generative AI receives that information and generates a response that soothes the subject.

[0398] "Example 2"

[0399] The emotion engine also has the ability to adjust the evaluation based on the subject's emotional state. Specifically, if the subject is happy, the response is rated high. Conversely, if the subject is unhappy, the response is rated low. In this way, the emotion engine grasps the subject's emotional state in real time and adjusts the evaluation accordingly.

[0400] "Example 3"

[0401] In addition, the emotion engine also has the ability to adjust responses based on the subject's emotional state. For example, if the subject shows signs of surprise, the generative AI will receive that information and generate a response that will reassure the subject. In this way, the emotion engine understands the subject's emotional state in real time and adjusts responses accordingly.

[0402] The process flow of each embodiment will be described below.

[0403] "Example 1"

[0404] Step 1: The generative AI collects information from the subject's tone of voice, facial expressions, eye movements, etc.

[0405] Step 2: Send the collected information to the emotion engine.

[0406] Step 3: The emotion engine recognizes the subject's emotional state and feeds that information back to the generative AI.

[0407] Step 4: The generative AI generates a response to the subject based on the feedback information. "Example 2"

[0408] Step 1: The generative AI sends the subject’s emotional state to the emotion engine.

[0409] Step 2: The emotion engine adjusts the rating based on the subject's emotional state.

[0410] Step 3: The adjusted evaluation is fed back to the generative AI.

[0411] Step 4: The generative AI generates a response to the subject based on the feedback information. "Example 3"

[0412] Step 1: The generative AI sends the subject’s emotional state to the emotion engine.

[0413] Step 2: The emotion engine adjusts the response based on the subject's emotional state.

[0414] Step 3: The adjusted response is fed back to the generative AI.

[0415] Step 4: The generative AI generates a response for the subject based on the feedback information.

[0416] Example 1

[0417] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0418] In conventional customer service simulation systems, responses to comments made by subjects were fixed, making interactive responses difficult. In addition, responses could not be generated that took into account the subject's emotional state, making it impossible to realize a realistic customer service simulation. Furthermore, evaluations of the subjects' responses were subjective, making it difficult to make objective evaluations.

[0419] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0420] In this invention, the server includes a means for incorporating a generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for capturing the subject's remarks as voice data, a means for converting the voice data into text data, a means for inputting a prompt sentence into the generative AI model and generating a response, a means for providing the generated response by voice, a means for capturing the subject's emotional state and feeding it back to the generative AI model, a means for updating the response based on the emotional information, and a means for providing the updated response by voice. This enables interactive response in real time according to the subject's remarks and emotional state, and realizes a more realistic customer service simulation.

[0421] "Generative AI" is an artificial intelligence technology that interactively generates responses to what a subject says.

[0422] The "complainer model" is a model that generates responses to statements made by subjects and simulates how to handle complaints.

[0423] "Subject" refers to a user of the simulation system.

[0424] "Biometric information" refers to physical information such as the subject's gaze, tone of voice, and facial expressions.

[0425] "Audio data" refers to audio signals that capture the subject's speech.

[0426] "Text data" refers to character information obtained by analyzing voice data.

[0427] A "prompt sentence" refers to the text data input into a generative AI model.

[0428] "Response" refers to the reply that a generative AI model generates based on a prompt.

[0429] "Emotional state" is information that indicates the subject's emotions and is estimated from gaze, tone of voice, facial expression, etc.

[0430] "Feedback" refers to feeding information such as emotional state back to the generative AI model.

[0431] "Updating" refers to the generative AI model regenerating a response based on fed-back information.

[0432] This invention incorporates generative AI into a VR simulation system for the customer service industry to interactively generate responses to the subjects' comments, realizing a realistic customer service simulation. This system captures the subjects' comments as voice data, converts the voice data into text data, and inputs prompts into the generative AI model to generate responses. It also captures the subject's emotional state and feeds that information back to the generative AI model to generate responses according to the emotions.

[0433] Hardware and Software Used

[0434] Hardware

[0435] VR headset (e.g. Oculus Rift)

[0436] Built-in microphone

[0437] Built-in camera

[0438] software

[0439] Speech recognition software (e.g. Google Cloud Speech-to-Text API)

[0440] Generative AI models (e.g. OpenAI's GPT-4)

[0441] Text-to-speech software (e.g. Google Text-to-Speech API)

[0442] Emotion engine (e.g. Affectiva SDK)

[0443] System Operation

[0444] 1. Capturing what users say

[0445] The user puts on the VR headset and speaks within the simulation. For example, the user might say, "Please tell me more about this product." The device uses a built-in microphone to capture the user's speech as voice data.

[0446] 2. Converting voice data to text

[0447] The server uses speech recognition software to convert the captured voice data into text, such as "Please tell me more about this product."

[0448] 3. Generating the Response

[0449] The server inputs the converted text data as a prompt to the generative AI model, which then generates a response such as, "This product is made using the latest technology and is lightweight and durable. It also features a long battery life and an easy-to-use design."

[0450] 4. Providing a Response

[0451] The device uses speech synthesis software to convert the generated text response into audio data and provides it to the user, who hears the response through the VR headset's speakers.

[0452] 5. Emotional state capture and feedback

[0453] The device uses a camera and microphone to capture the user's facial expressions and tone of voice. The captured data is sent to the emotion engine, which analyzes the user's emotional state. The server inputs additional prompts into the generative AI model based on the emotion information received from the emotion engine.

[0454] 6. Response Updates

[0455] For example, if the user is angry, the server adds the information "The user is angry" and re-prompts the generative AI model. The generative AI model updates the response to say, "We are sorry, customer. We sincerely apologize for the inconvenience. This product is made with the latest technology, making it lightweight and durable. It also features a long battery life and an easy-to-use design."

[0456] 7. Providing an Updated Response

[0457] The device then converts the updated text response into voice data using speech synthesis software again and provides it to the user, who hears the updated response through the VR headset's speakers.

[0458] Examples

[0459] For example, if a user says "Tell me more about this product" and indicates anger, the following specific actions will be taken:

[0460] 1. The user says, "Tell me more about this product."

[0461] 2. The device captures the audio data and sends it to the server.

[0462] 3. The server converts the voice data into text data.

[0463] 4. The server inputs the prompt sentence into the generative AI model and generates an initial response.

[0464] 5. The device provides an initial voice response to the user.

[0465] 6. The device captures the user's anger emotion and sends it to the server.

[0466] 7. The server inputs additional prompts to the generative AI model based on the emotion information and updates the response.

[0467] 8. The terminal provides the updated response audibly to the user.

[0468] In this way, the system generates and delivers appropriate responses depending on the user's utterances and emotional state.

[0469] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0470] Step 1:

[0471] The user speaks within the VR simulation.

[0472] Specifically, the user puts on the VR headset and says, "Tell me more about this product." The input is the user's voice, and the output is voice data.

[0473] Step 2:

[0474] The device captures the user's speech as audio data.

[0475] Specifically, the device uses a built-in microphone to capture the user's speech as voice data. The input is the user's voice, and the output is the captured voice data.

[0476] Step 3:

[0477] The server converts the voice data into text data.

[0478] Specifically, the server uses voice recognition software (e.g., Google Cloud Speech-to-Text API) to convert the captured voice data into text data. The input is voice data, and the output is text data.

[0479] Step 4:

[0480] The server inputs a prompt sentence into the generative AI model and generates a response.

[0481] Specifically, the server inputs the converted text data into a generative AI model (e.g., OpenAI's GPT-4) as a prompt. The generative AI model generates a response such as, "This product is made using the latest technology and is lightweight and durable. It also features a long battery life and an easy-to-use design." The input is the text data, and the output is the generated response.

[0482] Step 5:

[0483] The terminal provides the generated response audibly to the user.

[0484] Specifically, the device converts the generated text response into voice data using speech synthesis software (e.g., Google Text-to-Speech API) and provides it to the user. The input is the generated response, and the output is the voice data. The user hears the response through the speakers of the VR headset.

[0485] Step 6:

[0486] The device captures the user's emotional state and transmits it to the server.

[0487] Specifically, the device captures the user's facial expressions and tone of voice using a camera and microphone. The captured data is sent to an emotion engine (e.g., Affectiva SDK). The input is the user's facial expressions and tone of voice, and the output is emotional state data.

[0488] Step 7:

[0489] The server inputs additional prompts to the generative AI model based on the emotion information and updates the response.

[0490] Specifically, the server inputs additional prompts to the generative AI model based on the emotion information received from the emotion engine. For example, if the user is angry, the server adds the information "The user is angry" and inputs another prompt to the generative AI model. The generative AI model updates the response to say, "We are sorry, customer. We sincerely apologize for the inconvenience. This product is made with the latest technology and is lightweight and durable. It also features a long battery life and an easy-to-use design." The input is the emotion state data, and the output is the updated response.

[0491] Step 8:

[0492] The terminal provides the updated response to the user.

[0493] Specifically, the device converts the updated text response into voice data using speech synthesis software again and provides it to the user. The input is the updated response and the output is the voice data. The user hears the updated response through the speakers of the VR headset.

[0494] (Application example 1)

[0495] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0496] Conventional customer service simulation systems have difficulty recognizing the emotional state of customers in real time and generating appropriate responses accordingly. In addition, there is a lack of real-time support when store clerks are dealing with customers, making it difficult to improve customer satisfaction. Furthermore, there is also the problem that the evaluation of the clerk's response skills is subjective, making it difficult to make an objective evaluation.

[0497] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0498] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for analyzing the customer's facial expression and tone of voice in real time using smart glasses, a means for the generative AI to generate an appropriate response based on the customer's remarks and emotional information, and a means for displaying the generated response. This makes it possible to recognize the customer's emotional state in real time and generate an appropriate response accordingly. Furthermore, it is possible to objectively evaluate the response skills of store clerks and improve customer satisfaction.

[0499] "Generative AI" is an artificial intelligence system that interactively generates responses to what a subject says.

[0500] The "complainer model" is a simulation model for responding appropriately to customer complaints and questions.

[0501] A "subject" is a person who uses the system to simulate customer interactions.

[0502] The "means for evaluating" refers to a method or device for objectively evaluating the subject's response skills.

[0503] "Biometric information" is information obtained from the subject's body, such as their gaze or tone of voice.

[0504] "Smart glasses" are glasses-type devices that have built-in cameras and microphones and have the ability to analyze and display information in real time.

[0505] "Facial expression" refers to the facial movements and expressions of a customer and is an important factor in indicating their emotional state.

[0506] "Tone of voice" refers to vocal characteristics such as pitch, strength, and rhythm that are used to convey emotions and intentions.

[0507] An "analysis means" is a method or device that processes data obtained from a camera or microphone and converts it into meaningful information.

[0508] An "appropriate response" is an optimal reply or response that is generated based on the customer's statements and emotional state.

[0509] A "means for displaying" is a method or device for visually presenting the generated response.

[0510] A system for implementing the present invention includes smart glasses, a generative AI, an emotion recognition engine, a voice recognition engine, and a display device. The details of each component and their operation are described below.

[0511] Hardware Configuration

[0512] 1. Smart glasses: Equipped with a built-in camera and microphone, they capture customers' facial expressions and tone of voice in real time.

[0513] 2. Display device: Built into the smart glasses, it visually presents the generated response to the store associate.

[0514] Software Configuration

[0515] 1. Generative AI: An artificial intelligence system that interactively generates responses to what the subject says. Specifically, it uses large-scale language models such as GPT-3.

[0516] 2. Emotion recognition engine: Analyzes video data captured by the camera and estimates the customer's emotional state from their facial expressions. For example, use a library such as EmotionRecognizer.

[0517] 3. Speech recognition engine: Converts voice data obtained from a microphone into text, for example using the SpeechRecognition library.

[0518] Data processing and calculation

[0519] 1. Data acquisition: Using the camera and microphone in the smart glasses, the customer's facial expressions and tone of voice are captured in real time.

[0520] 2. Emotion recognition: The captured video data is analyzed using an emotion recognition engine to estimate the customer's emotional state.

[0521] 3. Speech recognition: The captured voice data is converted into text using a speech recognition engine.

[0522] 4. Prompt generation: Generate prompts based on customer statements and sentiment information.

[0523] 5. Response generation: The generative AI generates an appropriate response based on the prompt.

[0524] 6. Display Response: Display the generated response on the display device of the smart glasses.

[0525] Examples

[0526] For example, if a customer asks about a product and looks a little dissatisfied, the following prompt might be generated:

[0527] Example prompt:

[0528] Customers say: Can you tell me more about this product?

[0529] Customer Sentiment: Unhappy

[0530] response:

[0531] By inputting this prompt into the generative AI, the AI ​​can generate a response such as, "This product is made from high-quality materials and will last a long time. If you have any questions, please let us know."

[0532] In this way, store associates can provide appropriate responses in real time depending on the customer's emotional state, which improves customer satisfaction and allows for an objective evaluation of the associates' response skills.

[0533] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0534] Step 1:

[0535] The smart glasses' camera and microphone are used to capture the customer's facial expressions and tone of voice in real time.

[0536] Input: Customer's facial video and audio data.

[0537] Output: Video and audio data.

[0538] How it works: The camera in the smart glasses takes a picture of the customer's face, and the microphone records what the customer says.

[0539] Step 2:

[0540] The captured video data is analyzed using an emotion recognition engine to estimate the customer's emotional state.

[0541] Input: Video data.

[0542] Output: The emotional state of the customer (e.g. dissatisfied, happy, angry, etc.).

[0543] Specific operation: The server inputs the video data into the EmotionRecognizer library and estimates the customer's emotional state from their facial expressions.

[0544] Step 3:

[0545] The acquired voice data is converted into text using a voice recognition engine.

[0546] Input: Audio data.

[0547] Output: Text data (what the customer said).

[0548] Specific operation: The server inputs the audio data into the SpeechRecognition library and converts the audio to text.

[0549] Step 4:

[0550] Generate prompts based on customer statements and sentiment information.

[0551] Input: Text data (what the customer said) and the customer's emotional state.

[0552] Output: The prompt statement.

[0553] Specific operation: The server combines text data and emotion information to generate a prompt sentence to be input to the generative AI.

[0554] Step 5:

[0555] Generative AI generates appropriate responses based on the prompts.

[0556] Input: prompt statement.

[0557] Output: The response text.

[0558] How it works: The server inputs the prompt sentence into the GPT-3 model and generates an appropriate response.

[0559] Step 6:

[0560] The generated response is displayed on the display of the smart glasses.

[0561] Input: Response text.

[0562] Output: The response shown on the smart glasses display.

[0563] Specific operation: The server sends the generated response sentence to the smart glasses and displays the response on the display device.

[0564] Example 2

[0565] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0566] In the conventional complainer model, it was difficult to properly evaluate the subject's response, especially since the subject's emotional state was not taken into account. In addition, there was a lack of a way to evaluate the appropriateness of the generated response in real time, making it difficult to increase subject satisfaction.

[0567] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0568] In this invention, the server includes a means for incorporating generative artificial intelligence and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for analyzing the subject's emotional state in real time and adjusting the evaluation, and a means for evaluating the appropriateness of the generated response. This enables an appropriate evaluation that takes into account the subject's emotional state, thereby increasing the subject's satisfaction.

[0569] "Generative artificial intelligence" is an artificial intelligence technology that generates appropriate responses to what a subject says.

[0570] The "Complainer model" is a simulation model for interactively responding to comments made by subjects.

[0571] "Evaluation of subject's response" is the process of analyzing the subject's reaction to the generated response and evaluating its appropriateness and satisfaction.

[0572] "Biometric information" refers to physical data such as a subject's gaze, tone of voice, and facial expressions, and is information used to estimate the subject's emotional state.

[0573] "Real-time emotional state analysis" refers to the process of collecting a subject's biometric information in real time and instantly determining the subject's emotions based on that data.

[0574] "Adjusting the rating" refers to dynamically changing the rating of a generated response based on the subject's emotional state.

[0575] "Evaluating the appropriateness of a response" is the process of determining how appropriate the response generated is to what the subject said.

[0576] This invention is a system that uses generative artificial intelligence to generate responses to utterances made by subjects and evaluates the subjects' reactions to the responses. The system is composed of three entities: a server, a terminal, and a user.

[0577] The server uses generative artificial intelligence (e.g., a natural language processing model) to generate a response to the prompt input by the user. Specifically, the server uses a generative AI model such as OpenAI's GPT-4. This generative AI model receives input from the user and generates an appropriate response.

[0578] The terminal serves to display responses sent from the server to the user. The user inputs prompts to the server through the terminal and checks the generated responses. The terminal also serves as an interface for sending the user's responses to the server.

[0579] A user inputs a question or request to the system using a terminal. For example, a prompt sentence such as "Please tell me how to return a product." The server receives this prompt sentence and generates an appropriate response using a generative AI model. The generated response is displayed to the user through the terminal.

[0580] The server analyzes the user's reaction and evaluates the appropriateness of the generated response. For example, if the user responds with "This information was very helpful, thank you," the server rates this reaction highly. The server also uses an emotion engine (e.g., emotion recognition software) to analyze the user's emotional state in real time. If the user shows a happy emotion, the response is rated highly. Conversely, if the user shows an unpleasant emotion, the response is rated low.

[0581] As a concrete example, consider the following scenario:

[0582] A user enters the following into the customer support system: "How do I return a product?" The server uses a generative AI model to generate a response: "Here is how to return a product. First, prepare your proof of purchase. Next, fill out the return form and send it together with the product." This response is displayed to the user through the terminal. If the user responds with "This information was very helpful, thank you," the server rates this response highly and uses an emotion engine to analyze the user's emotions of happiness. This adjusts the rating even higher.

[0583] In this way, the system can evaluate the subject's response and adjust the evaluation based on the subject's emotional state.

[0584] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0585] Step 1:

[0586] The user enters a prompt sentence.

[0587] A user inputs a question or request to the system using a terminal. For example, the user inputs a prompt such as "Please tell me how to return a product." The input prompt is sent from the terminal to the server.

[0588] Step 2:

[0589] The server generates a response using a generative AI model.

[0590] The server inputs the prompt sentence received from the user into the generative AI model and generates an appropriate response. Specifically, the server uses the generative AI model (e.g., a natural language processing model) to generate a response to the prompt sentence. The generated response is sent from the server to the terminal.

[0591] Step 3:

[0592] The server generates a response and sends it to the terminal.

[0593] The server sends the response generated by the generative AI model to the terminal. For example, it sends a response such as, "Here is how to return the product. First, please prepare your proof of purchase. Next, please fill out the return form and send it together with the product."

[0594] Step 4:

[0595] The terminal displays the responses to the subject.

[0596] The terminal displays the response received from the server to the user, who then checks the response on the terminal screen. The displayed response becomes the information that the user needs to determine the next action.

[0597] Step 5:

[0598] The subject responds to the response.

[0599] The user responds to the displayed response, for example by saying, "This information was very helpful, thank you." The user's response is sent from the terminal to the server.

[0600] Step 6:

[0601] The terminal transmits the subject's responses to a server.

[0602] The terminal transmits the user's response to the server, which receives the response and proceeds to the next step.

[0603] Step 7:

[0604] The server analyzes the subjects' responses and provides an evaluation.

[0605] The server analyzes the user's reaction and evaluates the appropriateness of the generated response (for example, a response like "Thanks, that's enough" is highly evaluated). The server records the evaluation result.

[0606] Step 8:

[0607] The server uses an emotion engine to analyze the subject's emotional state.

[0608] The server uses an emotion engine (e.g., emotion recognition software) to analyze the user's emotional state in real time. It determines whether the user is showing a happy or unhappy emotion. The analysis result is used to adjust the rating.

[0609] Step 9:

[0610] The server adjusts the rating based on the emotional state.

[0611] The server adjusts the evaluation based on the analysis result of the emotion engine. If the user shows a happy emotion, the response is rated high. Conversely, if the user shows an unpleasant emotion, the response is rated low. The adjusted evaluation result is recorded as the final evaluation.

[0612] (Application example 2)

[0613] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0614] In conventional customer service systems, it was difficult to grasp the emotional state of the customer in real time and adjust responses based on that. In addition, there was no system that evaluated the appropriateness of responses generated by generative AI and evaluated customer service based on the customer's reaction. As a result, customer satisfaction and customer service quality were not sufficiently improved.

[0615] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0616] In this invention, the server includes a means for incorporating a generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for grasping the subject's emotional state in real time using an emotion engine and adjusting the evaluation, and a means for evaluating the appropriateness of the response generated by the generative AI and evaluating the customer service based on the subject's reaction. This makes it possible to grasp the customer's emotional state in real time and adjust the response based on that, thereby achieving increased customer satisfaction and improved customer service quality.

[0617] "Generative AI" refers to artificial intelligence that interactively generates responses to what a subject says.

[0618] The "complainer model" is a simulation model that provides an appropriate response to a subject's comments.

[0619] "Means of evaluation" refers to a function for evaluating the appropriateness based on the subject's response and reaction.

[0620] "Biometric information" refers to information obtained from the subject's body, such as their gaze or tone of voice.

[0621] The "emotion engine" is a system that analyzes the subject's emotional state in real time and adjusts the evaluation based on that information.

[0622] "Response appropriateness" is an indicator of how appropriate the response generated by the generative AI is for the subject.

[0623] "Customer service evaluation" refers to the process of evaluating the quality of customer service based on the responses generated by the generative AI and the subjects' reactions.

[0624] The system for implementing this invention includes a generative AI, an emotion engine, a voice recognition system, and a database. Specific hardware and software configurations and their processing will be described below.

[0625] Hardware and Software Used

[0626] Hardware:

[0627] Smartphones (e.g. iPhone(R), ANDROID(R) devices)

[0628] Smart glasses (e.g. Google Glass(R), Vuzix Blade)

[0629] software:

[0630] Generative AI models (e.g. OpenAI GPT-4)

[0631] Emotion engine (e.g. Affectiva SDK)

[0632] Speech recognition systems (e.g. Google Speech-to-Text API)

[0633] Databases (e.g. Firebase, MongoDB)

[0634] System processing overview

[0635] 1. Getting voice input:

[0636] When a user speaks to their smartphone or smart glasses, a voice recognition system converts the speech into text.

[0637] Example: A user says, "Tell me about this product."

[0638] 2. Response generation using generative AI models:

[0639] The converted text is then fed into a generative AI model to generate an appropriate response.

[0640] Example: A generative AI model generates a response like, "This product is the latest model, and its particular feature is its long battery life."

[0641] 3. Sentiment analysis using emotion engine:

[0642] The emotion engine analyzes the user's facial expressions and voice tone to grasp their emotional state in real time.

[0643] Example: If a user smiles and says "thank you", the emotion engine detects the emotion of happiness.

[0644] 4. Assessment adjustments and feedback:

[0645] The customer service is evaluated based on the generated responses and the user's reaction.

[0646] If the rating is high, feedback will be provided to the staff, such as "the service was good."

[0647] In case of low evaluation, suggestions for improvement will be provided.

[0648] Examples

[0649] scenario:

[0650] A user enters a store and talks to a staff member wearing smart glasses.

[0651] User: "Tell me about this product."

[0652] Staff members' smart glasses recognise the voice and a generative AI model generates a response.

[0653] Staff member: "This product is the latest model, and its particular feature is its long battery life."

[0654] Emotion analysis:

[0655] If the user says with satisfaction, "Thanks, that's good enough," the emotion engine will detect the emotion of joy and give it a thumbs up.

[0656] Examples of prompt statements

[0657] Prompt for generative AI model:

[0658] "When a customer says, 'Tell me about this product,' give them a brief description of the product's features."

[0659] In this way, the system can understand the customer's emotional state in real time and tailor its responses accordingly, resulting in increased customer satisfaction and improved customer experience.

[0660] The flow of the identification process in the application example 2 will be described with reference to FIG.

[0661] Step 1:

[0662] When a user speaks to a smartphone or smart glasses, the device receives voice input. Using a voice recognition system (e.g., Google Speech-to-Text API), the device converts the voice data into text data. The input is the user's voice, and the output is text data.

[0663] Step 2:

[0664] The device sends the converted text data to a generative AI model (e.g., OpenAI GPT-4). The generative AI model generates an appropriate response based on the input text data. The input is the text data, and the output is the generated response text.

[0665] Step 3:

[0666] The device converts the generated response text into speech so that the generated response text can be spoken back to the user. It uses a speech synthesis system (e.g., Google Text-to-Speech API) to convert the text data into speech data. The input is the generated response text, and the output is the speech data.

[0667] Step 4:

[0668] When the user reacts to the response, the device analyzes the user's biometric information such as facial expressions and tone of voice using an emotion engine (e.g., Affectiva SDK). The input is the user's biometric information, and the output is the user's emotional state.

[0669] Step 5:

[0670] The server evaluates the appropriateness of the generated response based on the user's emotional state obtained from the emotion engine. The inputs are the user's emotional state and the generated response text, and the output is the appropriateness rating of the response.

[0671] Step 6:

[0672] The server evaluates the service based on the evaluation results and provides feedback to the staff. If the evaluation is high, the server gives the staff the feedback "good service." If the evaluation is low, the server suggests areas for improvement. The input is the appropriateness of the response, and the output is a feedback message.

[0673] In this way, the system can understand the user's emotional state in real time and adjust its responses accordingly, resulting in increased customer satisfaction and improved customer service quality.

[0674] Example 3

[0675] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0676] Conventional response systems have difficulty accurately grasping the subject's emotional state and intentions, making it difficult to generate appropriate responses. In addition, there is a lack of technology to utilize biological information such as the subject's gaze and tone of voice in response generation, which leads to a problem of reduced quality of interactive responses. This leads to issues such as reduced subject satisfaction and a loss of system reliability.

[0677] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0678] In this invention, the server includes a means for incorporating generative AI and creating a response model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biometric information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for collecting the subject's biometric information, a means for analyzing the collected biometric information and identifying the subject's emotional state, a means for generating an appropriate response based on the analysis results, and a means for providing the generated response to the subject. This makes it possible to accurately grasp the subject's emotional state and intentions, and generate and provide an appropriate response in real time accordingly.

[0679] "Generative AI" refers to artificial intelligence that generates appropriate responses based on the subject's statements and biometric information.

[0680] A "response model" is a model that uses generative AI to interactively respond to the subject's statements.

[0681] An "assessment tool" is a tool for evaluating the quality of a subject's response or responses.

[0682] "Biometric information" refers to information obtained from the subject's body, such as their gaze or tone of voice.

[0683] "Collection means" refers to means for collecting biometric information from subjects using a camera, microphone, etc.

[0684] "Analysis means" refers to means for analyzing collected biometric information and identifying the emotional state of the subject.

[0685] A "response generation means" refers to a means for generating an appropriate response based on the analysis result.

[0686] "Providing means" refers to a means for providing the generated response to a subject.

[0687] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0688] This invention is a system that collects biological information such as the subject's gaze and tone of voice, and generates and evaluates responses based on that information. The purpose of this system is to more accurately grasp the subject's emotional state and intentions, and provide appropriate responses accordingly.

[0689] Hardware and Software Used

[0690] Hardware: Cameras, microphones, servers

[0691] Software: Emotion engine, generative AI model

[0692] System Overview

[0693] 1. Data collection:

[0694] The device uses a camera and microphone to collect biometric information such as the user's gaze and tone of voice.

[0695] Specifically, the camera captures the user's facial expressions and eye movements, and the microphone records the tone and volume of the user's voice.

[0696] 2. Data transmission:

[0697] The biometric information collected by the device is transmitted to the server in real time.

[0698] Specifically, the terminal converts data acquired from the camera and microphone into packets and transmits them to a server via the Internet.

[0699] 3. Data Analysis:

[0700] The server analyzes the received biometric information and identifies the user's emotional state.

[0701] Specifically, the server uses an emotion engine to analyze eye movements and changes in tone of voice to determine whether the user is surprised or excited.

[0702] 4. Response Generation:

[0703] Based on the analysis results, the server uses a generative AI model to generate an appropriate response.

[0704] Specifically, the server inputs the prompt sentence into a generative AI model, which generates a response according to the user's emotional state.

[0705] 5. Response sending:

[0706] The server generates a response and sends it to the terminal.

[0707] Specifically, the text data generated by the server is converted into packets and transmitted to the terminal via the Internet.

[0708] 6. Response provision:

[0709] The terminal provides the response received from the server to the user.

[0710] Specifically, the text data received by the terminal is input to a voice synthesis engine and played back as voice to the user.

[0711] Examples

[0712] Example 1:

[0713] If the user's tone of voice becomes higher when asking a question, the system will determine that the user is excited.

[0714] The server uses a generative AI model to generate a response: "Please stay calm. We'll answer your questions."

[0715] The terminal provides this response to the user.

[0716] Example 2:

[0717] If the user shows a look of surprise, the system determines that the user is surprised.

[0718] The server uses a generative AI model to generate a response that says, "Sorry for the surprise. Let me explain further."

[0719] The terminal provides this response to the user.

[0720] Examples of prompt statements

[0721] "If the user looks surprised, generate a reassuring response."

[0722] "If the user's voice tone rises, generate a soothing response."

[0723] In this way, the system can grasp the user's emotional state in real time and provide an appropriate response accordingly. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0724] Step 1:

[0725] Data collection

[0726] The device uses a camera and microphone to collect biometric information such as the user's gaze and tone of voice.

[0727] Input: User's facial expressions, eye movements, tone of voice, volume

[0728] What it does: The camera captures your facial expressions and eye movements, and the microphone records the tone and volume of your voice.

[0729] Output: Collected biometric data (image data, audio data)

[0730] Step 2:

[0731] Data transmission

[0732] The biometric information collected by the device is transmitted to the server in real time.

[0733] Input: Collected biometric data (image data, audio data)

[0734] Specific operation: The device converts data acquired from the camera and microphone into packets and sends them to a server via the Internet.

[0735] Output: Biometric data sent to the server

[0736] Step 3:

[0737] 4. Data Analysis

[0738] The server analyzes the received biometric information and identifies the user's emotional state.

[0739] Input: Biometric data sent to the server

[0740] What it does: The server uses an emotion engine to analyze eye movements and changes in tone of voice to determine whether the user is surprised or excited.

[0741] Output: User's emotional state (e.g. surprise, excitement)

[0742] Step 4:

[0743] Response Generation

[0744] Based on the analysis results, the server uses a generative AI model to generate an appropriate response.

[0745] Input: User's emotional state, prompt

[0746] Specific operation: The server inputs the prompt sentence into the generative AI model, which generates a response according to the user's emotional state.

[0747] Output: Generated response (text data)

[0748] Step 5:

[0749] Response Sending

[0750] The server generates a response and sends it to the terminal.

[0751] Input: Generated response (text data)

[0752] Specific operation: The text data generated by the server is converted into packets and sent to the terminal via the Internet.

[0753] Output: Response data sent to the terminal

[0754] Step 6:

[0755] Response provision

[0756] The terminal provides the response received from the server to the user.

[0757] Input: Response data sent to the terminal

[0758] Specific operation: The text data received by the terminal is input into a speech synthesis engine and played back as speech to the user.

[0759] Output: The audio response provided to the user.

[0760] (Application example 3)

[0761] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0762] Conventional interactive response systems have difficulty in accurately grasping the subject's emotional state and intentions, making it difficult to generate appropriate responses. In addition, there was a lack of technology to analyze the subject's biological information, such as gaze and tone of voice, in real time and reflect it in the response, which led to a problem of poor response quality. This posed a risk of insufficient response in situations that required a quick and appropriate response, especially in security services.

[0763] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0764] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for analyzing the subject's gaze and tone of voice in real time using a camera and microphone, a means for generating an appropriate response based on the subject's emotional state, and a means for displaying or notifying the generated response. This makes it possible to accurately grasp the subject's emotional state and intentions and generate an appropriate response in real time.

[0765] "Generative AI" is an artificial intelligence system that interactively generates responses based on the subject's statements and biometric information.

[0766] The "complainer model" is a simulation model for providing an appropriate response to a subject's statement.

[0767] A "subject" is a person who provides speech and biometric information to the system.

[0768] "Gaze" is biometric information that refers to the subject's eye movement and point of gaze.

[0769] "Tone of voice" refers to the vocal characteristics of a subject's speech and is an element that indicates emotional state and intention.

[0770] "Biometric information" is information obtained from the subject's body, such as their gaze or tone of voice.

[0771] "Real-time" refers to immediate processing or response without delay.

[0772] A "camera" is a device for capturing images of the subject's gaze and facial expression.

[0773] A "microphone" is a device for picking up the subject's voice.

[0774] "Emotional state" refers to the subject's psychological state and includes emotions such as happiness, anger, sadness, etc.

[0775] A "response" is a reply or reaction generated based on the subject's words or biometric information.

[0776] "Display" refers to a visual indication of the generated response.

[0777] "Notification" refers to notifying the user of the generated response by sound, vibration, etc.

[0778] A system for implementing this invention includes a means for incorporating a generative AI and creating a complainer model that can interactively respond to comments made by a subject, a means for evaluating the subject's response, a means for including biometric information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for analyzing the subject's gaze and tone of voice in real time using a camera and microphone, a means for generating an appropriate response based on the subject's emotional state, and a means for displaying or notifying the generated response.

[0779] The server uses a camera and microphone to capture the subject's gaze and voice tone in real time. The camera captures the subject's eye movements and gaze point, and the microphone captures the subject's voice. This biometric information is analyzed by the server's emotion recognition engine (EmotionRecognizer) and voice tone analysis engine (VoiceToneAnalyzer).

[0780] The analyzed gaze information and tone of voice information are input to a generative AI (Response Generator) to estimate the subject's emotional state and intention. The generative AI generates an appropriate response based on the estimated emotional state. This response is generated in text format and displayed on the smart glasses' display or notified by voice.

[0781] As a concrete example, consider a case where a security guard detects an individual in an agitated state while patrolling a facility. In this case, the server grasps the individual's emotional state in real time and generates a response to the security guard such as "This individual is in an agitated state. Please remain calm."

[0782] Examples of prompts to input to a generative AI model include:

[0783] "This person is agitated. Please remain calm."

[0784] In this way, the server can accurately grasp the subject's emotional state and intentions and generate an appropriate response in real time, improving response times, especially in security services, where a fast and appropriate response is required.

[0785] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0786] Step 1:

[0787] The server uses a camera to obtain the subject's gaze information. The camera captures the subject's eye movements and gaze point, and sends the video data to the server. The server receives this video data as input and analyzes the subject's gaze using a facial recognition algorithm. The output is the coordinate data of the subject's gaze direction and gaze point.

[0788] Step 2:

[0789] The server acquires the subject's voice tone information using a microphone. The microphone picks up the subject's speech and sends the voice data to the server. The server receives this voice data as input and analyzes the voice tone using a voice tone analysis engine (VoiceToneAnalyzer). As output, data on the subject's voice tone characteristics (e.g., pitch, volume, tempo, etc.) are obtained.

[0790] Step 3:

[0791] The server combines the gaze information obtained in step 1 and the tone of voice information obtained in step 2. Using these data as input, the server estimates the subject's emotional state using an emotion recognition engine (EmotionRecognizer). As output, data on the subject's emotional state (e.g., excitement, surprise, relief, etc.) is obtained.

[0792] Step 4:

[0793] The server uses the emotional state data obtained in step 3 as input and generates an appropriate response using a generative AI (Response Generator). The generative AI generates a prompt sentence based on the emotional state and creates a response to the subject. The output is the text data of the generated response.

[0794] Step 5:

[0795] The server displays the text data of the response generated in step 4 on the display of the smart glasses or notifies the user by voice. The smart glasses receive the response data sent from the server and present it to the user visually or audibly, allowing the user to confirm the appropriate response in real time.

[0796] In this way, the server can analyze the subject's gaze and tone of voice in real time and generate appropriate responses based on their emotional state to provide to the user.

[0797] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0798] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0799] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0800] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0801] [Second embodiment]

[0802] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0803] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0804] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0805] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0806] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0807] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0808] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0809] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0810] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0811] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0812] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0813] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0814] "Example 1"

[0815] In one embodiment of the present invention, a generative AI is incorporated into a VR simulation system for the customer service industry. This generative AI creates a complaint model that can interactively respond to the test subject's comments. Specifically, when the test subject says, "Please tell me the details of this product," the generative AI generates a response such as, "This product is ~~~ and has ~~~ characteristics."

[0816] "Example 2"

[0817] Furthermore, in one embodiment of the present invention, the subject's response is evaluated. This evaluation is based on the appropriateness of the response generated by the generative AI and the subject's reaction. For example, if the subject responds with "Thank you, that's enough," the response is highly evaluated.

[0818] "Example 3"

[0819] In one embodiment of the present invention, biological information such as the subject's gaze and tone of voice is included in the response generation / evaluation. This makes it possible to more accurately grasp the subject's emotional state and intentions and generate a response accordingly. For example, if the subject's tone of voice becomes higher, it is determined that the subject is in an excited state and an appropriate response is generated.

[0820] The process flow of each embodiment will be described below.

[0821] "Example 1"

[0822] Step 1: Launch the VR simulation system for hospitality.

[0823] Step 2: Incorporate generative AI. This AI creates a complaint model that can respond interactively to the subject's comments.

[0824] Step 3: If the subject says, "Please tell me more about this product," the generative AI generates a response such as, "This product is ~~ and has ~~ characteristics."

[0825] "Example 2"

[0826] Step 1: Evaluate the appropriateness of the response generated by the generative AI.

[0827] Step 2: Evaluate the subject's response. For example, if the subject responds, "Thank you, that's enough," the response is rated high.

[0828] "Example 3"

[0829] Step 1: Obtain biometric information such as the subject's gaze and tone of voice.

[0830] Step 2: Include the acquired biometric information in the response generation / evaluation, which allows for a more accurate understanding of the subject's emotional state and intentions.

[0831] Step 3: If the subject's voice tone becomes higher, determine that the subject is excited and generate an appropriate response.

[0832] Example 1

[0833] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0834] Conventional simulation systems for the customer service industry have a problem in that responses to the subjects' comments are fixed, making interactive responses difficult. In addition, evaluations do not take into account the subjects' biometric information, making it difficult to simulate a real customer service situation. Furthermore, there is an issue that practical training is difficult because responses cannot be generated in real time.

[0835] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for incorporating a generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for converting the subject's remarks into text data using a voice recognition technology, a means for generating a prompt sentence in the generative AI model and generating an appropriate response, a means for converting the generated response text into voice data using a voice synthesis technology, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, and a means for providing the generated voice data to the subject. This enables interactive response in real time to the subject's remarks, and realizes a simulation close to an actual customer service situation. Furthermore, since evaluation can be performed taking biological information into account, more practical training is possible.

[0836] "Generative AI" is an artificial intelligence technology that interactively generates responses to what a subject says.

[0837] The "complainer model" is a model for simulating a complaint in response to a statement made by a subject.

[0838] "Voice recognition technology" is a technology that converts voice data into text data.

[0839] A "prompt sentence" is an instruction sentence that causes a generative AI model to generate an appropriate response.

[0840] "Speech synthesis technology" is a technology that converts text data into voice data.

[0841] "Biometric information" is information obtained from the subject's body, such as their gaze or tone of voice.

[0842] An "evaluation means" is a means for evaluating the subject's response.

[0843] "Interactive" refers to a state in which two-way communication is possible.

[0844] "Real time" refers to a state in which processing is carried out immediately without delay.

[0845] This invention provides a customer service industry VR simulation system that incorporates generative AI to provide a customer complaint model that responds interactively to the comments of test subjects. A specific embodiment of this system will be described below.

[0846] First, the device collects the subject's speech through a microphone. The collected voice data is converted into text data using voice recognition technology. Specifically, the voice data is converted into text data using the Google Speech-to-Text API. In this conversion process, the voice signal is analyzed and the corresponding text is generated.

[0847] Next, the server generates a prompt sentence for the generative AI model based on the text data received from the terminal. This prompt sentence includes instructions for the generative AI model to generate an appropriate response. For example, the server generates a prompt sentence such as "The user said, 'Please tell me the details of this product.' Please generate a response."

[0848] The server then inputs the prompt sentence into a generative AI model (e.g., GPT-4) to generate an appropriate response. The generative AI model analyzes the prompt sentence and generates a natural response according to the context. For example, it generates a response such as, "This product is made using the latest technology, and has features such as high durability and ease of use."

[0849] The generated response text is converted to voice data by the server using voice synthesis technology. Specifically, Amazon Polly is used to convert text data into voice data. In this conversion process, text data is converted into voice signals to generate natural voice.

[0850] Finally, the device plays back the voice data received from the server and provides the response to the subject, allowing the subject to hear the response generated by the generative AI model.

[0851] As a concrete example, the process is as follows when a subject says "Tell me more about this product" in a VR simulation. The device converts the subject's speech into text using the Google Speech-to-Text API, and the server generates a prompt saying, "The user said, 'Tell me more about this product.' Please generate a response." The generative AI model (GPT-4) generates a response saying, "This product is made using the latest technology, and is durable and easy to use," and the server converts this into voice data using Amazon Polly. Finally, the device plays back the voice data and provides the response to the subject.

[0852] In this way, the server, the terminal, and the user can cooperate to realize an interactive response.

[0853] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0854] Step 1: Audio collection and recognition

[0855] The device collects the subject's speech through a microphone and converts the collected voice data into text data using the Google Speech-to-Text API. This conversion process involves analyzing the voice signal and generating the corresponding text.

[0856] Specific behavior:

[0857] Input: Subject's speech (audio data)

[0858] Data processing: Send the voice data to the Google Speech-to-Text API and analyze the voice signal.

[0859] Output: Text data (e.g. "Please tell me the details of this product")

[0860] Step 2: Generate a prompt

[0861] The server generates a prompt sentence for the generative AI model based on the text data received from the terminal. The prompt sentence includes instructions for generating an appropriate response for the generative AI model.

[0862] Specific behavior:

[0863] Input: Text data (e.g. "Please tell me the details of this product")

[0864] Data processing: Analyze text data and generate prompts suitable for generative AI models

[0865] Output: Prompt statement (e.g. "The user says 'Tell me more about this product.' Generate a response.")

[0866] Step 3: Generate a response

[0867] The server inputs a prompt sentence into a generative AI model (e.g., GPT-4) and generates an appropriate response. The generative AI model analyzes the prompt sentence and generates a natural response according to the context.

[0868] Specific behavior:

[0869] Input: Prompt statement (e.g. "The user says 'Tell me more about this product.' Generate a response.")

[0870] Data computation: A generative AI model analyzes the prompt and generates an appropriate response

[0871] Output: Response text (e.g. "This product is made using the latest technology, is durable, and has easy-to-use features.")

[0872] Step 4: Text-to-speech response

[0873] The server converts the generated response text into voice data using speech synthesis technology. Specifically, it uses Amazon Polly to convert text data into voice data.

[0874] Specific behavior:

[0875] Input: Response text (e.g. "This product is made using the latest technology, is durable, and has easy-to-use features.")

[0876] Data processing: Send text data to Amazon Polly and convert it into voice data

[0877] Output: Audio data

[0878] Step 5: Providing a response

[0879] The device plays the voice data received from the server and provides the response to the subject, allowing the subject to hear the response generated by the generative AI model.

[0880] Specific behavior:

[0881] Input: Audio data

[0882] Data processing: Playing back audio data

[0883] Output: Audio response to the subject (e.g., "This product is made using the latest technology, is durable, and has easy-to-use features.")

[0884] In this way, the server, the terminal, and the user work together to realize interactive responses.

[0885] (Application example 1)

[0886] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0887] In conventional customer service work, it is difficult for store clerks to respond quickly and appropriately to questions and complaints from customers. It is especially difficult for new or inexperienced store clerks to maintain the quality of customer service, which can lead to a decrease in customer satisfaction and an increase in complaints. It is also difficult for store clerks to accurately grasp the emotional state of customers and respond accordingly. To solve these problems, a system is needed that allows store clerks to receive appropriate responses in real time.

[0888] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0889] In this invention, the server includes a means for incorporating a generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for converting the customer's remarks into text using voice recognition, and a means for the generative AI to generate an appropriate response based on the customer's remarks and display it on a display device. This enables store clerks to respond quickly and appropriately to questions and complaints from customers, which is expected to improve customer satisfaction and reduce complaints.

[0890] "Generative AI" refers to artificial intelligence that automatically generates responses and content based on input data.

[0891] The "complainer model" is a simulation model for responding appropriately to complaints and questions from customers.

[0892] "Subject" means a person who is trained and evaluated using the system.

[0893] "Biometric information" refers to physical data such as a subject's gaze, tone of voice, heart rate, etc.

[0894] "Speech recognition" is a technology that converts speech into text data.

[0895] "Display device" means a device for visually displaying a generated response.

[0896] "Real-time" refers to immediate processing or response without delay.

[0897] "Customer satisfaction" refers to the degree of satisfaction a customer feels toward a service or product.

[0898] "Response generation" refers to the process of generating an appropriate response to a customer's utterance.

[0899] "Evaluation" refers to the process of measuring the quality of a subject's actions or responses and providing feedback.

[0900] A system for carrying out this invention incorporates a generative AI and creates a complainer model that can interactively respond to comments made by subjects. The system includes a means for evaluating the response of the subject, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for converting the customer's comments into text using voice recognition, and a means for the generative AI to generate an appropriate response based on the customer's comments and display it on a display device.

[0901] Hardware and Software Used

[0902] Hardware: Smart glasses, microphone

[0903] Software: OpenAI API, SpeechRecognition library, Smart Glasses SDK

[0904] System processing overview

[0905] The server first uses a microphone to capture the customer's speech as audio. It then uses the SpeechRecognition library to convert the speech into text data. This text data is input into a generative AI (OpenAI API) that generates an appropriate response based on the customer's speech. The generated response is displayed on the smart glasses' display device, allowing the store clerk to check it in real time.

[0906] Examples

[0907] For example, if a customer asks, "Please tell me more about this product," the generative AI will generate a response such as, "This product uses the latest technology, is lightweight, and is easy to carry." This response is displayed on the smart glasses' display, and the store clerk can explain it to the customer while checking it.

[0908] Examples of prompt statements

[0909] Customer Question: What are the details of this product?

[0910] Salesperson's response: This product uses the latest technology, is lightweight and easy to carry.

[0911] In this way, the system helps store clerks to respond quickly and appropriately to customer inquiries and complaints, improving customer satisfaction and reducing complaints.

[0912] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0913] Step 1:

[0914] The server uses a microphone to capture the customer's speech as audio: the input is the customer's voice, and the output is audio data, which is converted to text in the next step.

[0915] Step 2:

[0916] The server converts the voice data into text data using the SpeechRecognition library. The input is the voice data obtained in step 1, and the output is text data. This text data is input to the generative AI.

[0917] Step 3:

[0918] The server inputs text data into the generative AI (OpenAI API) and generates an appropriate response based on the customer's utterance. The input is the text data obtained in step 2, and the output is the generated response text. This response text is displayed on a display device in the next step.

[0919] Step 4:

[0920] The server sends the generated response text to the display device of the smart glasses for display. The input is the response text obtained in step 3, and the output is the text displayed on the display of the smart glasses. This allows the user (store clerk) to provide an appropriate response to the customer in real time.

[0921] Step 5:

[0922] The user (store clerk) checks the response displayed on the smart glasses display and provides explanations and responses to the customer based on that. The input is the response text displayed in step 4, and the output is a verbal explanation or response to the customer. This is expected to improve customer satisfaction.

[0923] Example 2

[0924] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0925] Conventional systems using generative AI had insufficient means to evaluate the appropriateness of responses to the subject's comments and the subject's reactions, making it difficult to effectively generate and evaluate interactive responses. In addition, it was difficult to include biological information such as the subject's gaze and tone of voice in response generation and evaluation, making it difficult to accurately estimate the subject's emotional state. Furthermore, the process of the user inputting a reaction to a generated response and evaluating that reaction was complicated, which led to problems with not improving the user experience.

[0926] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0927] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for the user to input a prompt sentence through the terminal, a means for the terminal to transmit the input prompt sentence to the server, a means for the server to generate a response to the prompt sentence using the generative AI model, a means for the server to transmit the generated response to the terminal, a means for the terminal to display the generated response to the user, a means for the user to input a reaction to the generated response, a means for the terminal to transmit the user's reaction to the server, a means for the server to analyze and evaluate the user's reaction, a means for the server to transmit the evaluation result to the terminal, and a means for the terminal to display the evaluation result to the user. This enables interactive response generation and evaluation using generative AI, making it possible to accurately estimate the subject's emotional state and improve the user experience.

[0928] "Generative AI" is a system that uses artificial intelligence technology to automatically generate responses to user input.

[0929] The "complainer model" is a model that responds interactively to comments made by subjects and incorporates generative AI.

[0930] A "subject" refers to a user who makes statements and responds to the system.

[0931] "Evaluation" is the process of analyzing the appropriateness of the responses generated by the generative AI and the subjects' reactions to determine their quality and usefulness.

[0932] "Biometric information" refers to physical data such as the subject's gaze, tone of voice, and heart rate, which are used to generate and evaluate responses.

[0933] A "prompt sentence" refers to a question or instruction that a user inputs to a generative AI.

[0934] "Terminal" refers to a device through which a user can input prompts and display generated responses and evaluation results.

[0935] "Server" refers to a computer system that executes a generative AI model, generates responses to prompts, and evaluates them.

[0936] "Response" refers to the reply or reaction that a generative AI generates based on a prompt.

[0937] "Analysis" refers to the process of processing the user's reactions and biometric information as data and understanding and evaluating its content.

[0938] "Evaluation Results" refers to the conclusions or feedback obtained after the Server analyzes the User's reactions and generated responses.

[0939] This invention is a system that uses generative AI to generate responses to utterances by subjects and evaluates the appropriateness of the responses and the reactions of the subjects. A specific embodiment of this system will be described below.

[0940] Hardware and Software Used

[0941] The server uses a high-performance computing system equipped with an NVIDIA GPU. For software, it uses deep learning frameworks such as TensorFlow and PyTorch to execute generative AI models, and natural language processing (NLP) libraries such as spaCy and NLTK to analyze the data.

[0942] The terminal is a device for a user to input a prompt sentence and display the generated response and the evaluation result. The terminal can be a general personal computer or a smartphone.

[0943] System Operation

[0944] 1. The user enters a prompt through the terminal.

[0945] The user enters a prompt sentence into the input field of the terminal, for example, "If the generating AI responded, 'Thank you, that's enough,' how would you rate that response?"

[0946] 2. The terminal sends the prompt text entered to the server

[0947] The terminal sends the entered prompt text to the server, using HTTP or WebSocket as the communication protocol.

[0948] 3. The server uses the generative AI model to generate a response to the prompt.

[0949] The server runs a generative AI model built using TensorFlow and PyTorch to generate a response to the prompt, such as "That's a very appropriate response."

[0950] 4. The server generates a response and sends it to the device.

[0951] The server then sends the generated response to the terminal, again using HTTP or WebSocket as the communication protocol.

[0952] 5. The terminal displays the generated response to the user.

[0953] The terminal displays the response received from the server to the user. For example, it displays "That response is very appropriate" on the screen.

[0954] 6. The user enters a response to the generated response

[0955] The user enters their reaction to the generated response, for example, "Thanks, that's good enough."

[0956] 7. The device sends the user's response to the server

[0957] The terminal sends the user's response to the server, again using HTTP or WebSocket as the communication protocol.

[0958] 8. The server analyzes the user's response and gives an evaluation.

[0959] The server uses natural language processing libraries (e.g., spaCy or NLTK) to analyze the user's response and rate its appropriateness. For example, a response like "Thanks, that's enough" is highly rated.

[0960] 9. The server sends the evaluation results to the device.

[0961] The server then transmits the evaluation results to the terminal, again using HTTP or WebSocket as the communication protocol.

[0962] 10. The terminal displays the evaluation results to the user.

[0963] The terminal displays the evaluation result received from the server to the user. For example, it displays "Your response is very appropriate" on the screen.

[0964] In this way, a system is realized in which the server, terminal, and user work together to generate and evaluate responses using a generative AI model. This system enables interactive response generation and evaluation using generative AI, accurately estimating the subject's emotional state and improving the user experience.

[0965] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0966] Step 1:

[0967] The user enters a prompt through the terminal.

[0968] Specifically, the user inputs a prompt sentence into the input field of the terminal. For example, the user inputs, "If the generating AI responded, 'Thank you, that's enough,' how would you evaluate that response?" The input data is the prompt sentence in text format.

[0969] Step 2:

[0970] The terminal sends the prompt text entered to the server.

[0971] Specifically, the terminal sends the input prompt text to the server using HTTP or WebSocket. The input data is the prompt text entered by the user, and the output data is the prompt text sent to the server.

[0972] Step 3:

[0973] The server uses a generative AI model to generate a response to the prompt.

[0974] Specifically, the server runs a generative AI model built using TensorFlow and PyTorch to generate a response to the prompt. The input data is the prompt, and the output data is the generated response. For example, it generates a response such as "That response is very appropriate."

[0975] Step 4:

[0976] The server sends the generated response to the terminal.

[0977] Specifically, the server sends the generated response to the terminal using HTTP or WebSocket. The input data is the generated response, and the output data is the response sent to the terminal.

[0978] Step 5:

[0979] The terminal displays the generated response to the user.

[0980] Specifically, the terminal displays the response received from the server to the user. The input data is the response sent from the server, and the output data is the response displayed to the user. For example, "That response is very appropriate" is displayed on the screen.

[0981] Step 6:

[0982] The user enters a reaction to the generated response.

[0983] As a specific operation, the user inputs his / her reaction to the generated response. The input data is the user's reaction, and the output data is the reaction input to the terminal. For example, the user inputs "Thank you, that's enough."

[0984] Step 7:

[0985] The device transmits the user's response to the server.

[0986] Specifically, the terminal transmits the user's response to the server using HTTP or WebSocket. The input data is the user's response, and the output data is the response transmitted to the server.

[0987] Step 8:

[0988] The server analyzes the user's response and provides an evaluation.

[0989] Specifically, the server uses a natural language processing library (e.g., spaCy or NLTK) to analyze the user's response and evaluate its appropriateness. The input data is the user's response, and the output data is the evaluation result. For example, a response like "Thanks, that's enough" is highly evaluated.

[0990] Step 9:

[0991] The server transmits the evaluation results to the terminal.

[0992] Specifically, the server sends the evaluation results to the terminal using HTTP or WebSocket. The input data is the evaluation results, and the output data is the evaluation results sent to the terminal.

[0993] Step 10:

[0994] The terminal displays the evaluation results to the user.

[0995] As a specific operation, the terminal displays the evaluation result received from the server to the user. The input data is the evaluation result sent from the server, and the output data is the evaluation result displayed to the user. For example, "Your response is very appropriate" is displayed on the screen.

[0996] (Application example 2)

[0997] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0998] Conventional customer service systems did not provide sufficient support for store clerks to respond appropriately to customers, making it difficult to improve customer satisfaction. In addition, there was a lack of means to evaluate the appropriateness of responses based on customer reactions, making it difficult to improve the response skills of store clerks. Furthermore, there was a need for a system that could efficiently handle customer service in physical stores.

[0999] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1000] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for generating responses to support customer service in a physical store, a means for evaluating the appropriateness of the response based on the customer's reaction, and a means including an application to be installed on a smartphone. This enables support for store clerks to respond appropriately to customers, thereby improving customer satisfaction and the response skills of store clerks.

[1001] "Generative AI" is a system that uses artificial intelligence technology to generate appropriate responses to user input.

[1002] The "complainer model" is a model for appropriately responding to complaints and inquiries that arise in customer service.

[1003] "Subject" refers to a user who participates in the evaluation and testing of the system.

[1004] "Biometrics" refers to physical data such as a subject's gaze, tone of voice, heart rate, etc.

[1005] "Brick and mortar store" refers to a sales or service location that is located in a physical location.

[1006] "Customer service" refers to the way store clerks respond to and provide services to customers.

[1007] "Response appropriateness" is the measure of how well the generated response meets the customer's requirements and expectations.

[1008] A "smartphone" is a multi-function mobile device that, in addition to the functions of a mobile phone, can connect to the Internet and run applications.

[1009] An "Application" is a software program designed to provide a particular function or service.

[1010] A system for implementing this invention includes a means for incorporating a generative AI and creating a complainer model that can interactively respond to comments made by a subject, a means for evaluating the subject's response, a means for including biometric information such as the subject's gaze and tone of voice in response generation / evaluation, a means for generating responses to assist with customer service in physical stores, a means for evaluating the appropriateness of the response based on the customer's reaction, and a means for including an application to be installed on a smartphone.

[1011] System Programming

[1012] The system is implemented using Python. OpenAI's GPT-3 is used as the generative AI model. The server receives the prompt entered by the store clerk and sends it to the generative AI model to generate a response. The generated response is displayed on the smartphone, providing assistance to the store clerk in providing an appropriate response to the customer.

[1013] Processing Description

[1014] The server receives the prompt entered by the store clerk and sends it to the generative AI model using the OpenAI API. The generative AI model generates an appropriate response based on the prompt and returns the response to the server. The server sends the generated response to the smartphone, and the store clerk responds to the customer. The customer's response is collected through the smartphone's microphone and camera and sent to the server. The server evaluates the appropriateness of the response based on the customer's response and provides feedback to the store clerk.

[1015] Hardware and Software Used

[1016] Hardware: Smartphones, servers

[1017] Software: Python, OpenAI API

[1018] Examples

[1019] For example, if a sales associate types, "Tell me what products you're looking for," the generative AI model responds with, "What kind of products are you looking for? For example, clothing or electronics?" If the customer responds, "Thanks, that's fine," this response is scored as "highly rated."

[1020] Example prompt:

[1021] "Tell us what product you're looking for."

[1022] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1023] Step 1:

[1024] The store clerk enters the prompt text into his / her smartphone.

[1025] Input: Prompt text to be entered by the salesperson (e.g. "Tell me about the product you are looking for.")

[1026] Output: The prompt text is sent to the server.

[1027] Specific actions: The store clerk opens the application on his / her smartphone, enters a prompt in the text box, and presses the send button.

[1028] Step 2:

[1029] The server receives the prompt text and sends it to the generative AI model.

[1030] Input: Prompt text sent from smartphone

[1031] Output: The prompt sent to the generative AI model

[1032] Specific operation: The server creates a request to send the prompt text received from the smartphone to the OpenAI API and sends it to the API endpoint.

[1033] Step 3:

[1034] A generative AI model generates a response based on the prompt.

[1035] Input: Prompt text sent from the server

[1036] Output: The generated response text

[1037] Specific operation: The generative AI model (e.g. GPT-3) analyzes the received prompt and generates an appropriate response, which is returned to the server as an API response.

[1038] Step 4:

[1039] The server receives the generated response and sends it to the smartphone.

[1040] Input: Response text returned by the generative AI model

[1041] Output: Response text sent to smartphone

[1042] Specific operation: The server analyzes the response text received from the generative AI model and creates a message to send to the smartphone. The message is sent to the smartphone application.

[1043] Step 5:

[1044] The smartphone displays the response text, and the store clerk responds to the customer.

[1045] Input: The response text sent by the server

[1046] Output: The response the salesperson gives to the customer

[1047] Specific operation: The smartphone application displays the received response text on the screen. The store clerk uses the displayed response as a reference to respond appropriately to the customer.

[1048] Step 6:

[1049] The smartphone collects customer responses and sends them to the server.

[1050] Input: Customer response (voice, gaze, facial expression, etc.)

[1051] Output: Customer response data sent to the server

[1052] How it works: The smartphone's microphone and camera are used to collect customer responses, and the collected data is sent to a server.

[1053] Step 7:

[1054] The server evaluates the appropriateness of the response based on the customer's reaction and provides feedback to the store attendant.

[1055] Input: Customer response data

[1056] Output: Response appropriateness assessment and feedback

[1057] Specific operation: The server analyzes the received customer response data and evaluates the appropriateness of the response. The evaluation result is sent to the store clerk's smartphone as feedback.

[1058] Example 3

[1059] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1060] In conventional interactive systems, responses to the subject's utterances were uniform, making it difficult to accurately grasp the subject's emotional state and intentions. As a result, it was not possible to generate appropriate responses according to the subject's emotions, resulting in a poor user experience. Furthermore, it was not possible to utilize biological information such as the subject's gaze or tone of voice, resulting in low response accuracy.

[1061] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1062] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for collecting gaze data and tone of voice data of the subject using a high-resolution camera and microphone, an image processing means for analyzing the collected gaze data, a voice analysis means for analyzing the collected tone of voice data, and a means for grasping the emotional state and intention of the subject based on the analyzed gaze data and tone of voice data and generating an appropriate response. This makes it possible to accurately grasp the emotional state and intention of the subject and generate an appropriate response accordingly.

[1063] "Generative AI" is an artificial intelligence system that generates responses in real time based on the subject's statements and biometric information.

[1064] The "complainer model" is a model that responds interactively to comments made by subjects, and is constructed using generative AI.

[1065] "Biometric information" is information obtained from a subject's body, such as their gaze or tone of voice, and is used to understand their emotional state and intentions.

[1066] A "high-resolution camera" is a camera for collecting gaze data from subjects, and is a device that has high resolution and can obtain detailed gaze information.

[1067] A "microphone" is a voice input device for collecting tone data of the subject's voice.

[1068] "Image processing means" refers to technology for analyzing collected gaze data and identifying the subject's gaze direction and movement.

[1069] The "voice analysis means" is a technology for analyzing collected voice tone data and detecting changes in the subject's voice tone.

[1070] The "response generation means" is a technology for understanding the subject's emotional state and intentions based on analyzed gaze data and tone of voice data, and generating an appropriate response.

[1071] This invention is a system that incorporates generative AI and can interactively respond to the subject's comments. Specifically, it is a system that collects biological information such as the subject's gaze and tone of voice, and generates / evaluates a response based on that information. A specific embodiment of this system is described below.

[1072] Hardware and Software Used

[1073] Hardware:

[1074] High-resolution camera: Used to collect subject's gaze data.

[1075] Microphone: Used to collect subject's vocal tone data.

[1076] Server: Equipped with high-performance CPU and GPU, it performs data analysis and response generation.

[1077] software:

[1078] OpenCV: An image processing library for analyzing gaze data.

[1079] PyAudio: An audio processing library for analyzing vocal tone data.

[1080] TensorFlow: A machine learning framework for running generative AI models.

[1081] System processing flow

[1082] Data collection:

[1083] The server activates a high-resolution camera to capture the user's face. The camera tracks the user's eye movements in real time to collect gaze data. The server also activates a microphone to record the user's voice. The microphone collects data for analyzing the tone of the user's voice in real time. The user speaks naturally into the camera and microphone while interacting with the system.

[1084] Data Analysis:

[1085] The server uses OpenCV to analyze the video data obtained from the camera. Specifically, it detects the position of the user's pupils and identifies the direction of gaze. For example, if the user is looking at the top right of the screen, that direction is identified. The server uses PyAudio to analyze the recorded audio data. Specifically, it analyzes the frequency components of the audio and detects changes in the tone of the voice. For example, if the tone of the user's voice becomes higher, it is determined that the user is excited.

[1086] Response generation:

[1087] The server uses TensorFlow to input gaze data and tone of voice data into the generative AI model. The generative AI model understands the user's emotional state and intentions based on this data. For example, if it determines that the user is excited, it generates a response that corresponds to that state. The server uses the generative AI model to generate a response appropriate to the user's state. For example, if the user is excited, it generates a response to respond calmly and chooses words to calm the user.

[1088] Examples of concrete examples and prompts

[1089] Examples:

[1090] Suppose a user is using an online customer support system. When the user asks a question, the system analyzes the user's gaze and tone of voice to determine whether the user is excited. For example, if the user's tone of voice becomes higher, the system determines that the user is excited and generates a response to help the user stay calm.

[1091] Example prompt:

[1092] "When a user is using your online customer support system, analyze their gaze and tone of voice to understand their emotional state and generate a response accordingly."

[1093] In this way, the server can generate a response based on the biometric information of the user, and can take a more accurate and appropriate action. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1094] Step 1:

[1095] Data collection

[1096] The server activates a high-resolution camera and microphone to collect the user's gaze data and tone of voice data. Specifically, the camera captures the user's face and tracks eye movements in real time, and the microphone records the user's voice and collects data for real-time tone of voice analysis.

[1097] Input: User's face video and audio

[1098] Output: Gaze data and tone of voice data

[1099] Step 2:

[1100] Analysis of gaze data

[1101] The server uses OpenCV to analyze the video data obtained from the camera. Specifically, it detects the position of the user's pupils and identifies the direction of the user's gaze. For example, if the user is looking at the upper right corner of the screen, that direction is identified.

[1102] Input: Gaze data

[1103] Output: Gaze direction information

[1104] Step 3:

[1105] Analysis of voice tone data

[1106] The server uses PyAudio to analyze the recorded voice data. Specifically, it analyzes the frequency components of the voice and detects changes in the tone of the voice. For example, if the user's voice tone becomes higher, it is determined that the user is excited.

[1107] Input: Voice tone data

[1108] Output: Voice tone change information

[1109] Step 4:

[1110] Emotional state estimation

[1111] The server understands the user's emotional state and intentions based on the analyzed gaze data and tone of voice data. Specifically, it determines whether the user is excited by combining gaze direction information and tone of voice change information.

[1112] Input: Gaze direction information, voice tone change information

[1113] Output: Emotional state information

[1114] Step 5:

[1115] Response Generation

[1116] The server uses TensorFlow to input emotional state information into a generative AI model to generate an appropriate response. For example, if the server determines that the user is excited, it generates a response according to that state.

[1117] Input: Emotional state information

[1118] Output: Response message

[1119] Step 6:

[1120] Providing a response

[1121] The server provides the generated response message to the user, either by displaying it in text form or by responding audibly using speech synthesis.

[1122] Input: Response message

[1123] Output: Response to the user

[1124] In this way, the server can generate a response based on the user's biometric information, enabling more accurate and appropriate action to be taken.

[1125] (Application example 3)

[1126] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1127] Conventional customer service systems have the problem that it is difficult to accurately grasp the emotional state and intentions of customers, and they are unable to generate appropriate responses. In addition, there is a lack of technology that utilizes biometric information such as gaze and tone of voice, and effective means to improve customer satisfaction are required. Furthermore, when introducing customer service robots into brick-and-mortar stores, flexible responses according to the customer's emotions and intentions are required.

[1128] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1129] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for grasping the subject's emotional state and intention and generating a response accordingly, and a means for being installed in a customer service robot in a brick-and-mortar store. This makes it possible to accurately grasp the customer's emotional state and intention and generate the optimal response accordingly.

[1130] "Generative AI" is an artificial intelligence system that interactively generates responses based on the subjects' statements and actions.

[1131] The "complainer model" is a model for generating appropriate responses to the statements or actions of customers or subjects.

[1132] An "evaluation means" is a means for evaluating the subject's response or behavior.

[1133] "Biometric information" is information obtained from the subject's body, such as their gaze or tone of voice.

[1134] The "response generating means" is a means for generating an appropriate response based on the subject's utterances and biometric information.

[1135] "Emotional state" refers to the subject's current emotional state.

[1136] "Intention" refers to what the subject is thinking and intending to do.

[1137] A "brick and mortar store" is a store that exists in a physical location.

[1138] A "customer service robot" is a robot that provides service to customers in a physical store.

[1139] A system for implementing this invention includes a means for incorporating generative AI and creating a complainer model that can interactively respond to comments made by a subject, a means for evaluating the subject's response, a means for including biometric information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for grasping the subject's emotional state and intentions and generating a response accordingly, and a means for being installed in a customer service robot in a physical store.

[1140] System Programming

[1141] The program of this system uses the following hardware and software:

[1142] Hardware: Camera (for gaze detection), microphone (for voice recognition)

[1143] Software: OpenCV (for face recognition and gaze detection), SpeechRecognition (for voice recognition), Transformers (for sentiment analysis)

[1144] Processing flow

[1145] The server captures video and audio in real time using a camera and microphone. It uses OpenCV to detect faces and eyes from the camera footage, and SpeechRecognition to convert the audio captured from the microphone into text. The converted text is analyzed for sentiment using a generative AI model from Transformers. Based on the analysis results, an appropriate response is generated.

[1146] Examples

[1147] For example, if a customer is excited, it is determined that the customer's voice tone is high and their gaze is focused on the product. In this case, the customer service robot will respond, "Sir, what product are you looking for today?" On the other hand, if a customer is in trouble, it is determined that the customer's voice tone is low and their gaze is unsteady. In this case, the customer service robot will respond, "If there's anything I can help you with, I'd like to help you."

[1148] Examples of prompt statements

[1149] "Generate responses where the customer's voice tone is high and their gaze is focused on the product."

[1150] In this way, customer service robots in physical stores will be able to accurately grasp the emotional state and intentions of customers and provide the most appropriate response accordingly.

[1151] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1152] Step 1:

[1153] The server uses a camera and microphone to capture video and audio in real time.

[1154] Input: Camera video, microphone audio

[1155] Output: Video data, audio data

[1156] Specific operation: Captures the customer's face and gaze through a camera, and records what the customer says through a microphone.

[1157] Step 2:

[1158] The server uses OpenCV to detect faces and eyes from camera footage.

[1159] Input: Video data

[1160] Output: Face and eye position information

[1161] Specific operation: Apply OpenCV's facial recognition algorithm to identify the face and eye positions from video data.

[1162] Step 3:

[1163] The server uses SpeechRecognition to convert the audio data into text.

[1164] Input: Audio data

[1165] Output: Text data

[1166] What it does: Apply a speech recognition algorithm to convert the audio data into text.

[1167] Step 4:

[1168] The server uses Transformers' generative AI models to parse sentiment from text data.

[1169] Input: Text data

[1170] Output: Emotion analysis results

[1171] How it works: Input text data into a generative AI model, and output an emotional state (e.g. positive, negative, neutral).

[1172] Step 5:

[1173] The server generates an appropriate response based on the face and eye position information and emotion analysis results.

[1174] Input: Face and eye position information, emotion analysis results

[1175] Output: Response text

[1176] Specific operation: Determine the direction of gaze from face and eye position information, combine it with emotion analysis results, and generate an appropriate response using a generative AI model.

[1177] Step 6:

[1178] The server transmits the generated response text to a customer service robot, which provides a response to the customer.

[1179] Input: Response text

[1180] Output: Audio response

[1181] Specific operation: The response text is input to the speech synthesis system and output as a voice. The customer service robot plays this voice and provides the response to the customer.

[1182] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1183] "Example 1"

[1184] One embodiment of the present invention is to incorporate an emotion engine into the generative AI. This emotion engine recognizes the emotional state of the subject from the tone of voice, facial expression, eye movement, etc., and feeds this information back to the generative AI. The generative AI generates a response to the subject based on this emotion information. For example, if the subject shows an emotion of anger, the generative AI receives that information and generates a response that soothes the subject.

[1185] "Example 2"

[1186] The emotion engine also has the ability to adjust the evaluation based on the subject's emotional state. Specifically, if the subject is happy, the response is rated high. Conversely, if the subject is unhappy, the response is rated low. In this way, the emotion engine grasps the subject's emotional state in real time and adjusts the evaluation accordingly.

[1187] "Example 3"

[1188] In addition, the emotion engine also has the ability to adjust responses based on the subject's emotional state. For example, if the subject shows signs of surprise, the generative AI will receive that information and generate a response that will reassure the subject. In this way, the emotion engine understands the subject's emotional state in real time and adjusts responses accordingly.

[1189] The process flow of each embodiment will be described below.

[1190] "Example 1"

[1191] Step 1: The generative AI collects information from the subject's tone of voice, facial expressions, eye movements, etc.

[1192] Step 2: Send the collected information to the emotion engine.

[1193] Step 3: The emotion engine recognizes the subject's emotional state and feeds that information back to the generative AI.

[1194] Step 4: The generative AI generates a response to the subject based on the feedback information. "Example 2"

[1195] Step 1: The generative AI sends the subject’s emotional state to the emotion engine.

[1196] Step 2: The emotion engine adjusts the rating based on the subject's emotional state.

[1197] Step 3: The adjusted evaluation is fed back to the generative AI.

[1198] Step 4: The generative AI generates a response to the subject based on the feedback information. "Example 3"

[1199] Step 1: The generative AI sends the subject’s emotional state to the emotion engine.

[1200] Step 2: The emotion engine adjusts the response based on the subject's emotional state.

[1201] Step 3: The adjusted response is fed back to the generative AI.

[1202] Step 4: The generative AI generates a response for the subject based on the feedback information.

[1203] Example 1

[1204] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1205] In conventional customer service simulation systems, responses to comments made by subjects were fixed, making interactive responses difficult. In addition, responses could not be generated that took into account the subject's emotional state, making it impossible to realize a realistic customer service simulation. Furthermore, evaluations of the subjects' responses were subjective, making it difficult to make objective evaluations.

[1206] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1207] In this invention, the server includes a means for incorporating a generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for capturing the subject's remarks as voice data, a means for converting the voice data into text data, a means for inputting a prompt sentence into the generative AI model and generating a response, a means for providing the generated response by voice, a means for capturing the subject's emotional state and feeding it back to the generative AI model, a means for updating the response based on the emotional information, and a means for providing the updated response by voice. This enables interactive response in real time according to the subject's remarks and emotional state, and realizes a more realistic customer service simulation.

[1208] "Generative AI" is an artificial intelligence technology that interactively generates responses to what a subject says.

[1209] The "complainer model" is a model that generates responses to statements made by subjects and simulates how to handle complaints.

[1210] "Subject" refers to a user of the simulation system.

[1211] "Biometric information" refers to physical information such as the subject's gaze, tone of voice, and facial expressions.

[1212] "Audio data" refers to audio signals that capture the subject's speech.

[1213] "Text data" refers to character information obtained by analyzing voice data.

[1214] A "prompt sentence" refers to the text data input into a generative AI model.

[1215] "Response" refers to the reply that a generative AI model generates based on a prompt.

[1216] "Emotional state" is information that indicates the subject's emotions and is estimated from gaze, tone of voice, facial expression, etc.

[1217] "Feedback" refers to feeding information such as emotional state back to the generative AI model.

[1218] "Updating" refers to the generative AI model regenerating a response based on fed-back information.

[1219] This invention incorporates generative AI into a VR simulation system for the customer service industry to interactively generate responses to the subjects' comments, realizing a realistic customer service simulation. This system captures the subjects' comments as voice data, converts the voice data into text data, and inputs prompts into the generative AI model to generate responses. It also captures the subject's emotional state and feeds that information back to the generative AI model to generate responses according to the emotions.

[1220] Hardware and Software Used

[1221] Hardware

[1222] VR headset (e.g. Oculus Rift)

[1223] Built-in microphone

[1224] Built-in camera

[1225] software

[1226] Speech recognition software (e.g. Google Cloud Speech-to-Text API)

[1227] Generative AI models (e.g. OpenAI's GPT-4)

[1228] Text-to-speech software (e.g. Google Text-to-Speech API)

[1229] Emotion engine (e.g. Affectiva SDK)

[1230] System Operation

[1231] 1. Capturing what users say

[1232] The user puts on the VR headset and speaks within the simulation. For example, the user might say, "Please tell me more about this product." The device uses a built-in microphone to capture the user's speech as voice data.

[1233] 2. Converting voice data to text

[1234] The server uses speech recognition software to convert the captured voice data into text, such as "Please tell me more about this product."

[1235] 3. Generating the Response

[1236] The server inputs the converted text data as a prompt to the generative AI model, which then generates a response such as, "This product is made using the latest technology and is lightweight and durable. It also features a long battery life and an easy-to-use design."

[1237] 4. Providing a Response

[1238] The device uses speech synthesis software to convert the generated text response into audio data and provides it to the user, who hears the response through the VR headset's speakers.

[1239] 5. Emotional state capture and feedback

[1240] The device uses a camera and microphone to capture the user's facial expressions and tone of voice. The captured data is sent to the emotion engine, which analyzes the user's emotional state. The server inputs additional prompts into the generative AI model based on the emotion information received from the emotion engine.

[1241] 6. Response Updates

[1242] For example, if the user is angry, the server adds the information "The user is angry" and re-prompts the generative AI model. The generative AI model updates the response to say, "We are sorry, customer. We sincerely apologize for the inconvenience. This product is made with the latest technology, making it lightweight and durable. It also features a long battery life and an easy-to-use design."

[1243] 7. Providing an Updated Response

[1244] The device then converts the updated text response into voice data using speech synthesis software again and provides it to the user, who hears the updated response through the VR headset's speakers.

[1245] Examples

[1246] For example, if a user says "Tell me more about this product" and indicates anger, the following specific actions will be taken:

[1247] 1. The user says, "Tell me more about this product."

[1248] 2. The device captures the audio data and sends it to the server.

[1249] 3. The server converts the voice data into text data.

[1250] 4. The server inputs the prompt sentence into the generative AI model and generates an initial response.

[1251] 5. The device provides an initial voice response to the user.

[1252] 6. The device captures the user's anger emotion and sends it to the server.

[1253] 7. The server inputs additional prompts to the generative AI model based on the emotion information and updates the response.

[1254] 8. The terminal provides the updated response audibly to the user.

[1255] In this way, the system generates and delivers appropriate responses depending on the user's utterances and emotional state.

[1256] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1257] Step 1:

[1258] The user speaks within the VR simulation.

[1259] Specifically, the user puts on the VR headset and says, "Tell me more about this product." The input is the user's voice, and the output is voice data.

[1260] Step 2:

[1261] The device captures the user's speech as audio data.

[1262] Specifically, the device uses a built-in microphone to capture the user's speech as voice data. The input is the user's voice, and the output is the captured voice data.

[1263] Step 3:

[1264] The server converts the voice data into text data.

[1265] Specifically, the server uses voice recognition software (e.g., Google Cloud Speech-to-Text API) to convert the captured voice data into text data. The input is voice data, and the output is text data.

[1266] Step 4:

[1267] The server inputs a prompt sentence into the generative AI model and generates a response.

[1268] Specifically, the server inputs the converted text data into a generative AI model (e.g., OpenAI's GPT-4) as a prompt. The generative AI model generates a response such as, "This product is made using the latest technology and is lightweight and durable. It also features a long battery life and an easy-to-use design." The input is the text data, and the output is the generated response.

[1269] Step 5:

[1270] The terminal provides the generated response audibly to the user.

[1271] Specifically, the device converts the generated text response into voice data using speech synthesis software (e.g., Google Text-to-Speech API) and provides it to the user. The input is the generated response, and the output is the voice data. The user hears the response through the speakers of the VR headset.

[1272] Step 6:

[1273] The device captures the user's emotional state and transmits it to the server.

[1274] Specifically, the device captures the user's facial expressions and tone of voice using a camera and microphone. The captured data is sent to an emotion engine (e.g., Affectiva SDK). The input is the user's facial expressions and tone of voice, and the output is emotional state data.

[1275] Step 7:

[1276] The server inputs additional prompts to the generative AI model based on the emotion information and updates the response.

[1277] Specifically, the server inputs additional prompts to the generative AI model based on the emotion information received from the emotion engine. For example, if the user is angry, the server adds the information "The user is angry" and inputs another prompt to the generative AI model. The generative AI model updates the response to say, "We are sorry, customer. We sincerely apologize for the inconvenience. This product is made with the latest technology and is lightweight and durable. It also features a long battery life and an easy-to-use design." The input is the emotion state data, and the output is the updated response.

[1278] Step 8:

[1279] The terminal provides the updated response to the user.

[1280] Specifically, the device converts the updated text response into voice data using speech synthesis software again and provides it to the user. The input is the updated response and the output is the voice data. The user hears the updated response through the speakers of the VR headset.

[1281] (Application example 1)

[1282] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1283] Conventional customer service simulation systems have difficulty recognizing the emotional state of customers in real time and generating appropriate responses accordingly. In addition, there is a lack of real-time support when store clerks are dealing with customers, making it difficult to improve customer satisfaction. Furthermore, there is also the problem that the evaluation of the clerk's response skills is subjective, making it difficult to make an objective evaluation.

[1284] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1285] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for analyzing the customer's facial expression and tone of voice in real time using smart glasses, a means for the generative AI to generate an appropriate response based on the customer's remarks and emotional information, and a means for displaying the generated response. This makes it possible to recognize the customer's emotional state in real time and generate an appropriate response accordingly. Furthermore, it is possible to objectively evaluate the response skills of store clerks and improve customer satisfaction.

[1286] "Generative AI" is an artificial intelligence system that interactively generates responses to what a subject says.

[1287] The "complainer model" is a simulation model for responding appropriately to customer complaints and questions.

[1288] A "subject" is a person who uses the system to simulate customer interactions.

[1289] The "means for evaluating" refers to a method or device for objectively evaluating the subject's response skills.

[1290] "Biometric information" is information obtained from the subject's body, such as their gaze or tone of voice.

[1291] "Smart glasses" are glasses-type devices that have built-in cameras and microphones and have the ability to analyze and display information in real time.

[1292] "Facial expression" refers to the facial movements and expressions of a customer and is an important factor in indicating their emotional state.

[1293] "Tone of voice" refers to vocal characteristics such as pitch, strength, and rhythm that are used to convey emotions and intentions.

[1294] An "analysis means" is a method or device that processes data obtained from a camera or microphone and converts it into meaningful information.

[1295] An "appropriate response" is an optimal reply or response that is generated based on the customer's statements and emotional state.

[1296] A "means for displaying" is a method or device for visually presenting the generated response.

[1297] A system for implementing the present invention includes smart glasses, a generative AI, an emotion recognition engine, a voice recognition engine, and a display device. The details of each component and their operation are described below.

[1298] Hardware Configuration

[1299] 1. Smart glasses: Equipped with a built-in camera and microphone, they capture customers' facial expressions and tone of voice in real time.

[1300] 2. Display device: Built into the smart glasses, it visually presents the generated response to the store associate.

[1301] Software Configuration

[1302] 1. Generative AI: An artificial intelligence system that interactively generates responses to what the subject says. Specifically, it uses large-scale language models such as GPT-3.

[1303] 2. Emotion recognition engine: Analyzes video data captured by the camera and estimates the customer's emotional state from their facial expressions. For example, use a library such as EmotionRecognizer.

[1304] 3. Speech recognition engine: Converts voice data obtained from a microphone into text, for example using the SpeechRecognition library.

[1305] Data processing and calculation

[1306] 1. Data acquisition: Using the camera and microphone in the smart glasses, the customer's facial expressions and tone of voice are captured in real time.

[1307] 2. Emotion recognition: The captured video data is analyzed using an emotion recognition engine to estimate the customer's emotional state.

[1308] 3. Speech recognition: The captured voice data is converted into text using a speech recognition engine.

[1309] 4. Prompt generation: Generate prompts based on customer statements and sentiment information.

[1310] 5. Response generation: The generative AI generates an appropriate response based on the prompt.

[1311] 6. Display Response: Display the generated response on the display device of the smart glasses.

[1312] Examples

[1313] For example, if a customer asks about a product and looks a little dissatisfied, the following prompt might be generated:

[1314] Example prompt:

[1315] Customers say: Can you tell me more about this product?

[1316] Customer Sentiment: Unhappy

[1317] response:

[1318] By inputting this prompt into the generative AI, the AI ​​can generate a response such as, "This product is made from high-quality materials and will last a long time. If you have any questions, please let us know."

[1319] In this way, store associates can provide appropriate responses in real time depending on the customer's emotional state, which improves customer satisfaction and allows for an objective evaluation of the associates' response skills.

[1320] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1321] Step 1:

[1322] The smart glasses' camera and microphone are used to capture the customer's facial expressions and tone of voice in real time.

[1323] Input: Customer's facial video and audio data.

[1324] Output: Video and audio data.

[1325] How it works: The camera in the smart glasses takes a picture of the customer's face, and the microphone records what the customer says.

[1326] Step 2:

[1327] The captured video data is analyzed using an emotion recognition engine to estimate the customer's emotional state.

[1328] Input: Video data.

[1329] Output: The emotional state of the customer (e.g. dissatisfied, happy, angry, etc.).

[1330] Specific operation: The server inputs the video data into the EmotionRecognizer library and estimates the customer's emotional state from their facial expressions.

[1331] Step 3:

[1332] The acquired voice data is converted into text using a voice recognition engine.

[1333] Input: Audio data.

[1334] Output: Text data (what the customer said).

[1335] Specific operation: The server inputs the audio data into the SpeechRecognition library and converts the audio to text.

[1336] Step 4:

[1337] Generate prompts based on customer statements and sentiment information.

[1338] Input: Text data (what the customer said) and the customer's emotional state.

[1339] Output: The prompt statement.

[1340] Specific operation: The server combines text data and emotion information to generate a prompt sentence to be input to the generative AI.

[1341] Step 5:

[1342] Generative AI generates appropriate responses based on the prompts.

[1343] Input: prompt statement.

[1344] Output: The response text.

[1345] How it works: The server inputs the prompt sentence into the GPT-3 model and generates an appropriate response.

[1346] Step 6:

[1347] The generated response is displayed on the display of the smart glasses.

[1348] Input: Response text.

[1349] Output: The response shown on the smart glasses display.

[1350] Specific operation: The server sends the generated response sentence to the smart glasses and displays the response on the display device.

[1351] Example 2

[1352] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[1353] In the conventional complainer model, it was difficult to properly evaluate the subject's response, especially since the subject's emotional state was not taken into account. In addition, there was a lack of a way to evaluate the appropriateness of the generated response in real time, making it difficult to increase subject satisfaction.

[1354] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1355] In this invention, the server includes a means for incorporating generative artificial intelligence and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for analyzing the subject's emotional state in real time and adjusting the evaluation, and a means for evaluating the appropriateness of the generated response. This enables an appropriate evaluation that takes into account the subject's emotional state, thereby increasing the subject's satisfaction.

[1356] "Generative artificial intelligence" is an artificial intelligence technology that generates appropriate responses to what a subject says.

[1357] The "Complainer model" is a simulation model for interactively responding to comments made by subjects.

[1358] "Evaluation of subject's response" is the process of analyzing the subject's reaction to the generated response and evaluating its appropriateness and satisfaction.

[1359] "Biometric information" refers to physical data such as a subject's gaze, tone of voice, and facial expressions, and is information used to estimate the subject's emotional state.

[1360] "Real-time emotional state analysis" refers to the process of collecting a subject's biometric information in real time and instantly determining the subject's emotions based on that data.

[1361] "Adjusting the rating" refers to dynamically changing the rating of a generated response based on the subject's emotional state.

[1362] "Evaluating the appropriateness of a response" is the process of determining how appropriate the response generated is to what the subject said.

[1363] This invention is a system that uses generative artificial intelligence to generate responses to utterances made by subjects and evaluates the subjects' reactions to the responses. The system is composed of three entities: a server, a terminal, and a user.

[1364] The server uses generative artificial intelligence (e.g., a natural language processing model) to generate a response to the prompt input by the user. Specifically, the server uses a generative AI model such as OpenAI's GPT-4. This generative AI model receives input from the user and generates an appropriate response.

[1365] The terminal serves to display responses sent from the server to the user. The user inputs prompts to the server through the terminal and checks the generated responses. The terminal also serves as an interface for sending the user's responses to the server.

[1366] A user inputs a question or request to the system using a terminal. For example, a prompt sentence such as "Please tell me how to return a product." The server receives this prompt sentence and generates an appropriate response using a generative AI model. The generated response is displayed to the user through the terminal.

[1367] The server analyzes the user's reaction and evaluates the appropriateness of the generated response. For example, if the user responds with "This information was very helpful, thank you," the server rates this reaction highly. The server also uses an emotion engine (e.g., emotion recognition software) to analyze the user's emotional state in real time. If the user shows a happy emotion, the response is rated highly. Conversely, if the user shows an unpleasant emotion, the response is rated low.

[1368] As a concrete example, consider the following scenario:

[1369] A user enters the following into the customer support system: "How do I return a product?" The server uses a generative AI model to generate a response: "Here is how to return a product. First, prepare your proof of purchase. Next, fill out the return form and send it together with the product." This response is displayed to the user through the terminal. If the user responds with "This information was very helpful, thank you," the server rates this response highly and uses an emotion engine to analyze the user's emotions of happiness. This adjusts the rating even higher.

[1370] In this way, the system can evaluate the subject's response and adjust the evaluation based on the subject's emotional state.

[1371] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1372] Step 1:

[1373] The user enters a prompt sentence.

[1374] A user inputs a question or request to the system using a terminal. For example, the user inputs a prompt such as "Please tell me how to return a product." The input prompt is sent from the terminal to the server.

[1375] Step 2:

[1376] The server generates a response using a generative AI model.

[1377] The server inputs the prompt sentence received from the user into the generative AI model and generates an appropriate response. Specifically, the server uses the generative AI model (e.g., a natural language processing model) to generate a response to the prompt sentence. The generated response is sent from the server to the terminal.

[1378] Step 3:

[1379] The server generates a response and sends it to the terminal.

[1380] The server sends the response generated by the generative AI model to the terminal. For example, it sends a response such as, "Here is how to return the product. First, please prepare your proof of purchase. Next, please fill out the return form and send it together with the product."

[1381] Step 4:

[1382] The terminal displays the responses to the subject.

[1383] The terminal displays the response received from the server to the user, who then checks the response on the terminal screen. The displayed response becomes the information that the user needs to determine the next action.

[1384] Step 5:

[1385] The subject responds to the response.

[1386] The user responds to the displayed response, for example by saying, "This information was very helpful, thank you." The user's response is sent from the terminal to the server.

[1387] Step 6:

[1388] The terminal transmits the subject's responses to a server.

[1389] The terminal transmits the user's response to the server, which receives the response and proceeds to the next step.

[1390] Step 7:

[1391] The server analyzes the subjects' responses and provides an evaluation.

[1392] The server analyzes the user's reaction and evaluates the appropriateness of the generated response (for example, a response like "Thanks, that's enough" is highly evaluated). The server records the evaluation result.

[1393] Step 8:

[1394] The server uses an emotion engine to analyze the subject's emotional state.

[1395] The server uses an emotion engine (e.g., emotion recognition software) to analyze the user's emotional state in real time. It determines whether the user is showing a happy or unhappy emotion. The analysis result is used to adjust the rating.

[1396] Step 9:

[1397] The server adjusts the rating based on the emotional state.

[1398] The server adjusts the evaluation based on the analysis result of the emotion engine. If the user shows a happy emotion, the response is rated high. Conversely, if the user shows an unpleasant emotion, the response is rated low. The adjusted evaluation result is recorded as the final evaluation.

[1399] (Application example 2)

[1400] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1401] In conventional customer service systems, it was difficult to grasp the emotional state of the customer in real time and adjust responses based on that. In addition, there was no system that evaluated the appropriateness of responses generated by generative AI and evaluated customer service based on the customer's reaction. As a result, customer satisfaction and customer service quality were not sufficiently improved.

[1402] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1403] In this invention, the server includes a means for incorporating a generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for grasping the subject's emotional state in real time using an emotion engine and adjusting the evaluation, and a means for evaluating the appropriateness of the response generated by the generative AI and evaluating the customer service based on the subject's reaction. This makes it possible to grasp the customer's emotional state in real time and adjust the response based on that, thereby achieving increased customer satisfaction and improved customer service quality.

[1404] "Generative AI" refers to artificial intelligence that interactively generates responses to what a subject says.

[1405] The "complainer model" is a simulation model that provides an appropriate response to a subject's comments.

[1406] "Means of evaluation" refers to a function for evaluating the appropriateness based on the subject's response and reaction.

[1407] "Biometric information" refers to information obtained from the subject's body, such as their gaze or tone of voice.

[1408] The "emotion engine" is a system that analyzes the subject's emotional state in real time and adjusts the evaluation based on that information.

[1409] "Response appropriateness" is an indicator of how appropriate the response generated by the generative AI is for the subject.

[1410] "Customer service evaluation" refers to the process of evaluating the quality of customer service based on the responses generated by the generative AI and the subjects' reactions.

[1411] The system for implementing this invention includes a generative AI, an emotion engine, a voice recognition system, and a database. Specific hardware and software configurations and their processing will be described below.

[1412] Hardware and Software Used

[1413] Hardware:

[1414] Smartphones (e.g. iPhone, Android devices)

[1415] Smart glasses (e.g. Google Glass, Vuzix Blade)

[1416] software:

[1417] Generative AI models (e.g. OpenAI GPT-4)

[1418] Emotion engine (e.g. Affectiva SDK)

[1419] Speech recognition systems (e.g. Google Speech-to-Text API)

[1420] Databases (e.g. Firebase, MongoDB)

[1421] System processing overview

[1422] 1. Getting voice input:

[1423] When a user speaks to their smartphone or smart glasses, a voice recognition system converts the speech into text.

[1424] Example: A user says, "Tell me about this product."

[1425] 2. Response generation using generative AI models:

[1426] The converted text is then fed into a generative AI model to generate an appropriate response.

[1427] Example: A generative AI model generates a response like, "This product is the latest model, and its particular feature is its long battery life."

[1428] 3. Sentiment analysis using emotion engine:

[1429] The emotion engine analyzes the user's facial expressions and voice tone to grasp their emotional state in real time.

[1430] Example: If a user smiles and says "thank you", the emotion engine detects the emotion of happiness.

[1431] 4. Assessment adjustments and feedback:

[1432] The customer service is evaluated based on the generated responses and the user's reaction.

[1433] If the rating is high, feedback will be provided to the staff, such as "the service was good."

[1434] In case of low evaluation, suggestions for improvement will be provided.

[1435] Examples

[1436] scenario:

[1437] A user enters a store and talks to a staff member wearing smart glasses.

[1438] User: "Tell me about this product."

[1439] Staff members' smart glasses recognise the voice and a generative AI model generates a response.

[1440] Staff member: "This product is the latest model, and its particular feature is its long battery life."

[1441] Emotion analysis:

[1442] If the user says with satisfaction, "Thanks, that's good enough," the emotion engine will detect the emotion of joy and give it a thumbs up.

[1443] Examples of prompt statements

[1444] Prompt for generative AI model:

[1445] "When a customer says, 'Tell me about this product,' give them a brief description of the product's features."

[1446] In this way, the system can understand the customer's emotional state in real time and tailor its responses accordingly, resulting in increased customer satisfaction and improved customer experience.

[1447] The flow of the identification process in the application example 2 will be described with reference to FIG.

[1448] Step 1:

[1449] When a user speaks to a smartphone or smart glasses, the device receives voice input. Using a voice recognition system (e.g., Google Speech-to-Text API), the device converts the voice data into text data. The input is the user's voice, and the output is text data.

[1450] Step 2:

[1451] The device sends the converted text data to a generative AI model (e.g., OpenAI GPT-4). The generative AI model generates an appropriate response based on the input text data. The input is the text data, and the output is the generated response text.

[1452] Step 3:

[1453] The device converts the generated response text into speech so that the generated response text can be spoken back to the user. It uses a speech synthesis system (e.g., Google Text-to-Speech API) to convert the text data into speech data. The input is the generated response text, and the output is the speech data.

[1454] Step 4:

[1455] When the user reacts to the response, the device analyzes the user's biometric information such as facial expressions and tone of voice using an emotion engine (e.g., Affectiva SDK). The input is the user's biometric information, and the output is the user's emotional state.

[1456] Step 5:

[1457] The server evaluates the appropriateness of the generated response based on the user's emotional state obtained from the emotion engine. The inputs are the user's emotional state and the generated response text, and the output is the appropriateness rating of the response.

[1458] Step 6:

[1459] The server evaluates the service based on the evaluation results and provides feedback to the staff. If the evaluation is high, the server gives the staff the feedback "good service." If the evaluation is low, the server suggests areas for improvement. The input is the appropriateness of the response, and the output is a feedback message.

[1460] In this way, the system can understand the user's emotional state in real time and adjust its responses accordingly, resulting in increased customer satisfaction and improved customer service quality.

[1461] Example 3

[1462] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1463] Conventional response systems have difficulty accurately grasping the subject's emotional state and intentions, making it difficult to generate appropriate responses. In addition, there is a lack of technology to utilize biological information such as the subject's gaze and tone of voice in response generation, which leads to a problem of reduced quality of interactive responses. This leads to issues such as reduced subject satisfaction and a loss of system reliability.

[1464] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1465] In this invention, the server includes a means for incorporating generative AI and creating a response model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biometric information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for collecting the subject's biometric information, a means for analyzing the collected biometric information and identifying the subject's emotional state, a means for generating an appropriate response based on the analysis results, and a means for providing the generated response to the subject. This makes it possible to accurately grasp the subject's emotional state and intentions, and generate and provide an appropriate response in real time accordingly.

[1466] "Generative AI" refers to artificial intelligence that generates appropriate responses based on the subject's statements and biometric information.

[1467] A "response model" is a model that uses generative AI to interactively respond to the subject's statements.

[1468] An "assessment tool" is a tool for evaluating the quality of a subject's response or responses.

[1469] "Biometric information" refers to information obtained from the subject's body, such as their gaze or tone of voice.

[1470] "Collection means" refers to means for collecting biometric information from subjects using a camera, microphone, etc.

[1471] "Analysis means" refers to means for analyzing collected biometric information and identifying the emotional state of the subject.

[1472] A "response generation means" refers to a means for generating an appropriate response based on the analysis result.

[1473] "Providing means" refers to a means for providing the generated response to a subject.

[1474] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1475] This invention is a system that collects biological information such as the subject's gaze and tone of voice, and generates and evaluates responses based on that information. The purpose of this system is to more accurately grasp the subject's emotional state and intentions, and provide appropriate responses accordingly.

[1476] Hardware and Software Used

[1477] Hardware: Cameras, microphones, servers

[1478] Software: Emotion engine, generative AI model

[1479] System Overview

[1480] 1. Data collection:

[1481] The device uses a camera and microphone to collect biometric information such as the user's gaze and tone of voice.

[1482] Specifically, the camera captures the user's facial expressions and eye movements, and the microphone records the tone and volume of the user's voice.

[1483] 2. Data transmission:

[1484] The biometric information collected by the device is transmitted to the server in real time.

[1485] Specifically, the terminal converts data acquired from the camera and microphone into packets and transmits them to a server via the Internet.

[1486] 3. Data Analysis:

[1487] The server analyzes the received biometric information and identifies the user's emotional state.

[1488] Specifically, the server uses an emotion engine to analyze eye movements and changes in tone of voice to determine whether the user is surprised or excited.

[1489] 4. Response Generation:

[1490] Based on the analysis results, the server uses a generative AI model to generate an appropriate response.

[1491] Specifically, the server inputs the prompt sentence into a generative AI model, which generates a response according to the user's emotional state.

[1492] 5. Response sending:

[1493] The server generates a response and sends it to the terminal.

[1494] Specifically, the text data generated by the server is converted into packets and transmitted to the terminal via the Internet.

[1495] 6. Response provision:

[1496] The terminal provides the response received from the server to the user.

[1497] Specifically, the text data received by the terminal is input to a voice synthesis engine and played back as voice to the user.

[1498] Examples

[1499] Example 1:

[1500] If the user's tone of voice becomes higher when asking a question, the system will determine that the user is excited.

[1501] The server uses a generative AI model to generate a response: "Please stay calm. We'll answer your questions."

[1502] The terminal provides this response to the user.

[1503] Example 2:

[1504] If the user shows a look of surprise, the system determines that the user is surprised.

[1505] The server uses a generative AI model to generate a response that says, "Sorry for the surprise. Let me explain further."

[1506] The terminal provides this response to the user.

[1507] Examples of prompt statements

[1508] "If the user looks surprised, generate a reassuring response."

[1509] "If the user's voice tone rises, generate a soothing response."

[1510] In this way, the system can grasp the user's emotional state in real time and provide an appropriate response accordingly. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1511] Step 1:

[1512] Data collection

[1513] The device uses a camera and microphone to collect biometric information such as the user's gaze and tone of voice.

[1514] Input: User's facial expressions, eye movements, tone of voice, volume

[1515] What it does: The camera captures your facial expressions and eye movements, and the microphone records the tone and volume of your voice.

[1516] Output: Collected biometric data (image data, audio data)

[1517] Step 2:

[1518] Data transmission

[1519] The biometric information collected by the device is transmitted to the server in real time.

[1520] Input: Collected biometric data (image data, audio data)

[1521] Specific operation: The device converts data acquired from the camera and microphone into packets and sends them to a server via the Internet.

[1522] Output: Biometric data sent to the server

[1523] Step 3:

[1524] 4. Data Analysis

[1525] The server analyzes the received biometric information and identifies the user's emotional state.

[1526] Input: Biometric data sent to the server

[1527] What it does: The server uses an emotion engine to analyze eye movements and changes in voice tone to determine whether the user is surprised or excited.

[1528] Output: User's emotional state (e.g. surprise, excitement)

[1529] Step 4:

[1530] Response Generation

[1531] Based on the analysis results, the server uses a generative AI model to generate an appropriate response.

[1532] Input: User's emotional state, prompt

[1533] Specific operation: The server inputs the prompt sentence into the generative AI model, which generates a response according to the user's emotional state.

[1534] Output: Generated response (text data)

[1535] Step 5:

[1536] Response Sending

[1537] The server generates a response and sends it to the terminal.

[1538] Input: Generated response (text data)

[1539] Specific operation: The text data generated by the server is converted into packets and sent to the terminal via the Internet.

[1540] Output: Response data sent to the terminal

[1541] Step 6:

[1542] Response provision

[1543] The terminal provides the response received from the server to the user.

[1544] Input: Response data sent to the terminal

[1545] Specific operation: The text data received by the terminal is input into a speech synthesis engine and played back as speech to the user.

[1546] Output: The audio response provided to the user.

[1547] (Application example 3)

[1548] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1549] Conventional interactive response systems have difficulty in accurately grasping the subject's emotional state and intentions, making it difficult to generate appropriate responses. In addition, there was a lack of technology to analyze the subject's biological information, such as gaze and tone of voice, in real time and reflect it in the response, which led to a problem of poor response quality. This posed a risk of insufficient response in situations that required a quick and appropriate response, especially in security services.

[1550] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1551] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for analyzing the subject's gaze and tone of voice in real time using a camera and microphone, a means for generating an appropriate response based on the subject's emotional state, and a means for displaying or notifying the generated response. This makes it possible to accurately grasp the subject's emotional state and intentions and generate an appropriate response in real time.

[1552] "Generative AI" is an artificial intelligence system that interactively generates responses based on the subject's statements and biometric information.

[1553] The "complainer model" is a simulation model for providing an appropriate response to a subject's statement.

[1554] A "subject" is a person who provides speech and biometric information to the system.

[1555] "Gaze" is biometric information that refers to the subject's eye movement and point of gaze.

[1556] "Tone of voice" refers to the vocal characteristics of a subject's speech and is an element that indicates emotional state and intention.

[1557] "Biometric information" is information obtained from the subject's body, such as their gaze or tone of voice.

[1558] "Real-time" refers to immediate processing or response without delay.

[1559] A "camera" is a device for capturing images of the subject's gaze and facial expression.

[1560] A "microphone" is a device for picking up the subject's voice.

[1561] "Emotional state" refers to the subject's psychological state and includes emotions such as happiness, anger, sadness, etc.

[1562] A "response" is a reply or reaction generated based on the subject's words or biometric information.

[1563] "Display" refers to a visual indication of the generated response.

[1564] "Notification" refers to notifying the user of the generated response by sound, vibration, etc.

[1565] A system for implementing this invention includes a means for incorporating a generative AI and creating a complainer model that can interactively respond to comments made by a subject, a means for evaluating the subject's response, a means for including biometric information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for analyzing the subject's gaze and tone of voice in real time using a camera and microphone, a means for generating an appropriate response based on the subject's emotional state, and a means for displaying or notifying the generated response.

[1566] The server uses a camera and microphone to capture the subject's gaze and voice tone in real time. The camera captures the subject's eye movements and gaze point, and the microphone captures the subject's voice. This biometric information is analyzed by the server's emotion recognition engine (EmotionRecognizer) and voice tone analysis engine (VoiceToneAnalyzer).

[1567] The analyzed gaze information and tone of voice information are input to a generative AI (Response Generator) to estimate the subject's emotional state and intention. The generative AI generates an appropriate response based on the estimated emotional state. This response is generated in text format and displayed on the smart glasses' display or notified by voice.

[1568] As a concrete example, consider a case where a security guard detects an individual in an agitated state while patrolling a facility. In this case, the server grasps the individual's emotional state in real time and generates a response to the security guard such as "This individual is in an agitated state. Please remain calm."

[1569] Examples of prompts to input to a generative AI model include:

[1570] "This person is agitated. Please remain calm."

[1571] In this way, the server can accurately grasp the subject's emotional state and intentions and generate an appropriate response in real time, improving response times, especially in security services, where a fast and appropriate response is required.

[1572] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1573] Step 1:

[1574] The server uses a camera to obtain the subject's gaze information. The camera captures the subject's eye movements and gaze point, and sends the video data to the server. The server receives this video data as input and analyzes the subject's gaze using a facial recognition algorithm. The output is the coordinate data of the subject's gaze direction and gaze point.

[1575] Step 2:

[1576] The server acquires the subject's voice tone information using a microphone. The microphone picks up the subject's speech and sends the voice data to the server. The server receives this voice data as input and analyzes the voice tone using a voice tone analysis engine (VoiceToneAnalyzer). As output, data on the subject's voice tone characteristics (e.g., pitch, volume, tempo, etc.) are obtained.

[1577] Step 3:

[1578] The server combines the gaze information obtained in step 1 and the tone of voice information obtained in step 2. Using these data as input, the server estimates the subject's emotional state using an emotion recognition engine (EmotionRecognizer). As output, data on the subject's emotional state (e.g., excitement, surprise, relief, etc.) is obtained.

[1579] Step 4:

[1580] The server uses the emotional state data obtained in step 3 as input and generates an appropriate response using a generative AI (Response Generator). The generative AI generates a prompt sentence based on the emotional state and creates a response to the subject. The output is the text data of the generated response.

[1581] Step 5:

[1582] The server displays the text data of the response generated in step 4 on the display of the smart glasses or notifies the user by voice. The smart glasses receive the response data sent from the server and present it to the user visually or audibly, allowing the user to confirm the appropriate response in real time.

[1583] In this way, the server can analyze the subject's gaze and tone of voice in real time and generate appropriate responses based on their emotional state to provide to the user.

[1584] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1585] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1586] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1587] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1588] [Third embodiment]

[1589] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1590] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1591] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1592] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1593] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1594] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1595] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1596] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1597] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1598] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1599] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1600] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1601] "Example 1"

[1602] In one embodiment of the present invention, a generative AI is incorporated into a VR simulation system for the customer service industry. This generative AI creates a complaint model that can interactively respond to the test subject's comments. Specifically, when the test subject says, "Please tell me the details of this product," the generative AI generates a response such as, "This product is ~~~ and has ~~~ characteristics."

[1603] "Example 2"

[1604] Furthermore, in one embodiment of the present invention, the subject's response is evaluated. This evaluation is based on the appropriateness of the response generated by the generative AI and the subject's reaction. For example, if the subject responds with "Thank you, that's enough," the response is highly evaluated.

[1605] "Example 3"

[1606] In one embodiment of the present invention, biological information such as the subject's gaze and tone of voice is included in the response generation / evaluation. This makes it possible to more accurately grasp the subject's emotional state and intentions and generate a response accordingly. For example, if the subject's tone of voice becomes higher, it is determined that the subject is in an excited state and an appropriate response is generated.

[1607] The process flow of each embodiment will be described below.

[1608] "Example 1"

[1609] Step 1: Launch the VR simulation system for hospitality.

[1610] Step 2: Incorporate generative AI. This AI creates a complaint model that can respond interactively to the subject's comments.

[1611] Step 3: If the subject says, "Please tell me more about this product," the generative AI generates a response such as, "This product is ~~ and has ~~ characteristics."

[1612] "Example 2"

[1613] Step 1: Evaluate the appropriateness of the response generated by the generative AI.

[1614] Step 2: Evaluate the subject's response. For example, if the subject responds, "Thank you, that's enough," the response is rated high.

[1615] "Example 3"

[1616] Step 1: Obtain biometric information such as the subject's gaze and tone of voice.

[1617] Step 2: Include the acquired biometric information in the response generation / evaluation, which allows for a more accurate understanding of the subject's emotional state and intentions.

[1618] Step 3: If the subject's voice tone becomes higher, determine that the subject is excited and generate an appropriate response.

[1619] Example 1

[1620] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1621] Conventional simulation systems for the customer service industry have a problem in that responses to the subjects' comments are fixed, making interactive responses difficult. In addition, evaluations do not take into account the subjects' biometric information, making it difficult to simulate a real customer service situation. Furthermore, there is an issue that practical training is difficult because responses cannot be generated in real time.

[1622] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for incorporating a generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for converting the subject's remarks into text data using a voice recognition technology, a means for generating a prompt sentence in the generative AI model and generating an appropriate response, a means for converting the generated response text into voice data using a voice synthesis technology, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, and a means for providing the generated voice data to the subject. This enables interactive response in real time to the subject's remarks, and realizes a simulation close to an actual customer service situation. Furthermore, since evaluation can be performed taking biological information into account, more practical training is possible.

[1623] "Generative AI" is an artificial intelligence technology that interactively generates responses to what a subject says.

[1624] The "complainer model" is a model for simulating a complaint in response to a statement made by a subject.

[1625] "Voice recognition technology" is a technology that converts voice data into text data.

[1626] A "prompt sentence" is an instruction sentence that causes a generative AI model to generate an appropriate response.

[1627] "Speech synthesis technology" is a technology that converts text data into voice data.

[1628] "Biometric information" is information obtained from the subject's body, such as their gaze or tone of voice.

[1629] An "evaluation means" is a means for evaluating the subject's response.

[1630] "Interactive" refers to a state in which two-way communication is possible.

[1631] "Real time" refers to a state in which processing is carried out immediately without delay.

[1632] This invention provides a customer service industry VR simulation system that incorporates generative AI to provide a customer complaint model that responds interactively to the comments of test subjects. A specific embodiment of this system will be described below.

[1633] First, the device collects the subject's speech through a microphone. The collected voice data is converted into text data using voice recognition technology. Specifically, the voice data is converted into text data using the Google Speech-to-Text API. In this conversion process, the voice signal is analyzed and the corresponding text is generated.

[1634] Next, the server generates a prompt sentence for the generative AI model based on the text data received from the terminal. This prompt sentence includes instructions for the generative AI model to generate an appropriate response. For example, the server generates a prompt sentence such as "The user said, 'Please tell me the details of this product.' Please generate a response."

[1635] The server then inputs the prompt sentence into a generative AI model (e.g., GPT-4) to generate an appropriate response. The generative AI model analyzes the prompt sentence and generates a natural response according to the context. For example, it generates a response such as, "This product is made using the latest technology, and has features such as high durability and ease of use."

[1636] The generated response text is converted to voice data by the server using voice synthesis technology. Specifically, Amazon Polly is used to convert text data into voice data. In this conversion process, text data is converted into voice signals to generate natural voice.

[1637] Finally, the device plays back the voice data received from the server and provides the response to the subject, allowing the subject to hear the response generated by the generative AI model.

[1638] As a concrete example, the process is as follows when a subject says "Tell me more about this product" in a VR simulation. The device converts the subject's speech into text using the Google Speech-to-Text API, and the server generates a prompt saying, "The user said, 'Tell me more about this product.' Please generate a response." The generative AI model (GPT-4) generates a response saying, "This product is made using the latest technology, and is durable and easy to use," and the server converts this into voice data using Amazon Polly. Finally, the device plays back the voice data and provides the response to the subject.

[1639] In this way, the server, the terminal, and the user can cooperate to realize an interactive response.

[1640] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1641] Step 1: Audio collection and recognition

[1642] The device collects the subject's speech through a microphone and converts the collected voice data into text data using the Google Speech-to-Text API. This conversion process involves analyzing the voice signal and generating the corresponding text.

[1643] Specific behavior:

[1644] Input: Subject's speech (audio data)

[1645] Data processing: Send the voice data to the Google Speech-to-Text API and analyze the voice signal.

[1646] Output: Text data (e.g. "Please tell me the details of this product")

[1647] Step 2: Generate a prompt

[1648] The server generates a prompt sentence for the generative AI model based on the text data received from the terminal. The prompt sentence includes instructions for generating an appropriate response for the generative AI model.

[1649] Specific behavior:

[1650] Input: Text data (e.g. "Please tell me the details of this product")

[1651] Data processing: Analyze text data and generate prompts suitable for generative AI models

[1652] Output: Prompt statement (e.g. "The user says 'Tell me more about this product.' Generate a response.")

[1653] Step 3: Generate a response

[1654] The server inputs a prompt sentence into a generative AI model (e.g., GPT-4) and generates an appropriate response. The generative AI model analyzes the prompt sentence and generates a natural response according to the context.

[1655] Specific behavior:

[1656] Input: Prompt statement (e.g. "The user says 'Tell me more about this product.' Generate a response.")

[1657] Data computation: A generative AI model analyzes the prompt and generates an appropriate response

[1658] Output: Response text (e.g. "This product is made using the latest technology, is durable, and has easy-to-use features.")

[1659] Step 4: Text-to-speech response

[1660] The server converts the generated response text into voice data using speech synthesis technology. Specifically, it uses Amazon Polly to convert text data into voice data.

[1661] Specific behavior:

[1662] Input: Response text (e.g. "This product is made using the latest technology, is durable, and has easy-to-use features.")

[1663] Data processing: Send text data to Amazon Polly and convert it into voice data

[1664] Output: Audio data

[1665] Step 5: Providing a response

[1666] The device plays the voice data received from the server and provides the response to the subject, allowing the subject to hear the response generated by the generative AI model.

[1667] Specific behavior:

[1668] Input: Audio data

[1669] Data processing: Playing back audio data

[1670] Output: Audio response to the subject (e.g., "This product is made using the latest technology, is durable, and has easy-to-use features.")

[1671] In this way, the server, the terminal, and the user work together to realize interactive responses.

[1672] (Application example 1)

[1673] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1674] In conventional customer service work, it is difficult for store clerks to respond quickly and appropriately to questions and complaints from customers. It is especially difficult for new or inexperienced store clerks to maintain the quality of customer service, which can lead to a decrease in customer satisfaction and an increase in complaints. It is also difficult for store clerks to accurately grasp the emotional state of customers and respond accordingly. To solve these problems, a system is needed that allows store clerks to receive appropriate responses in real time.

[1675] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1676] In this invention, the server includes a means for incorporating a generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for converting the customer's remarks into text using voice recognition, and a means for the generative AI to generate an appropriate response based on the customer's remarks and display it on a display device. This enables store clerks to respond quickly and appropriately to questions and complaints from customers, which is expected to improve customer satisfaction and reduce complaints.

[1677] "Generative AI" refers to artificial intelligence that automatically generates responses and content based on input data.

[1678] The "complainer model" is a simulation model for responding appropriately to complaints and questions from customers.

[1679] "Subject" means a person who is trained and evaluated using the system.

[1680] "Biometric information" refers to physical data such as a subject's gaze, tone of voice, heart rate, etc.

[1681] "Speech recognition" is a technology that converts speech into text data.

[1682] "Display device" means a device for visually displaying a generated response.

[1683] "Real-time" refers to immediate processing or response without delay.

[1684] "Customer satisfaction" refers to the degree of satisfaction a customer feels toward a service or product.

[1685] "Response generation" refers to the process of generating an appropriate response to a customer's utterance.

[1686] "Evaluation" refers to the process of measuring the quality of a subject's actions or responses and providing feedback.

[1687] A system for carrying out this invention incorporates a generative AI and creates a complainer model that can interactively respond to comments made by subjects. The system includes a means for evaluating the response of the subject, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for converting the customer's comments into text using voice recognition, and a means for the generative AI to generate an appropriate response based on the customer's comments and display it on a display device.

[1688] Hardware and Software Used

[1689] Hardware: Smart glasses, microphone

[1690] Software: OpenAI API, SpeechRecognition library, Smart Glasses SDK

[1691] System processing overview

[1692] The server first uses a microphone to capture the customer's speech as audio. It then uses the SpeechRecognition library to convert the speech into text data. This text data is input into a generative AI (OpenAI API) that generates an appropriate response based on the customer's speech. The generated response is displayed on the smart glasses' display device, allowing the store clerk to check it in real time.

[1693] Examples

[1694] For example, if a customer asks, "Please tell me more about this product," the generative AI will generate a response such as, "This product uses the latest technology, is lightweight, and is easy to carry." This response is displayed on the smart glasses' display, and the store clerk can explain it to the customer while checking it.

[1695] Examples of prompt statements

[1696] Customer Question: What are the details of this product?

[1697] Salesperson's response: This product uses the latest technology, is lightweight and easy to carry.

[1698] In this way, the system helps store clerks to respond quickly and appropriately to customer inquiries and complaints, improving customer satisfaction and reducing complaints.

[1699] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1700] Step 1:

[1701] The server uses a microphone to capture the customer's speech as audio: the input is the customer's voice, and the output is audio data, which is converted to text in the next step.

[1702] Step 2:

[1703] The server converts the voice data into text data using the SpeechRecognition library. The input is the voice data obtained in step 1, and the output is text data. This text data is input to the generative AI.

[1704] Step 3:

[1705] The server inputs text data into the generative AI (OpenAI API) and generates an appropriate response based on the customer's utterance. The input is the text data obtained in step 2, and the output is the generated response text. This response text is displayed on a display device in the next step.

[1706] Step 4:

[1707] The server sends the generated response text to the display device of the smart glasses for display. The input is the response text obtained in step 3, and the output is the text displayed on the display of the smart glasses. This allows the user (store clerk) to provide an appropriate response to the customer in real time.

[1708] Step 5:

[1709] The user (store clerk) checks the response displayed on the smart glasses display and provides explanations and responses to the customer based on that. The input is the response text displayed in step 4, and the output is a verbal explanation or response to the customer. This is expected to improve customer satisfaction.

[1710] Example 2

[1711] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1712] Conventional systems using generative AI had insufficient means to evaluate the appropriateness of responses to the subject's comments and the subject's reactions, making it difficult to effectively generate and evaluate interactive responses. In addition, it was difficult to include biological information such as the subject's gaze and tone of voice in response generation and evaluation, making it difficult to accurately estimate the subject's emotional state. Furthermore, the process of the user inputting a reaction to a generated response and evaluating that reaction was complicated, which led to problems with not improving the user experience.

[1713] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1714] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for the user to input a prompt sentence through the terminal, a means for the terminal to transmit the input prompt sentence to the server, a means for the server to generate a response to the prompt sentence using the generative AI model, a means for the server to transmit the generated response to the terminal, a means for the terminal to display the generated response to the user, a means for the user to input a reaction to the generated response, a means for the terminal to transmit the user's reaction to the server, a means for the server to analyze and evaluate the user's reaction, a means for the server to transmit the evaluation result to the terminal, and a means for the terminal to display the evaluation result to the user. This enables interactive response generation and evaluation using generative AI, making it possible to accurately estimate the subject's emotional state and improve the user experience.

[1715] "Generative AI" is a system that uses artificial intelligence technology to automatically generate responses to user input.

[1716] The "complainer model" is a model that responds interactively to comments made by subjects and incorporates generative AI.

[1717] A "subject" refers to a user who makes statements and responds to the system.

[1718] "Evaluation" is the process of analyzing the appropriateness of the responses generated by the generative AI and the subjects' reactions to determine their quality and usefulness.

[1719] "Biometric information" refers to physical data such as the subject's gaze, tone of voice, and heart rate, which are used to generate and evaluate responses.

[1720] A "prompt sentence" refers to a question or instruction that a user inputs to a generative AI.

[1721] "Terminal" refers to a device through which a user can input prompts and display generated responses and evaluation results.

[1722] "Server" refers to a computer system that executes a generative AI model, generates responses to prompts, and evaluates them.

[1723] "Response" refers to the reply or reaction that a generative AI generates based on a prompt.

[1724] "Analysis" refers to the process of processing the user's reactions and biometric information as data and understanding and evaluating its content.

[1725] "Evaluation Results" refers to the conclusions or feedback obtained after the Server analyzes the User's reactions and generated responses.

[1726] This invention is a system that uses generative AI to generate responses to utterances by subjects and evaluates the appropriateness of the responses and the reactions of the subjects. A specific embodiment of this system will be described below.

[1727] Hardware and Software Used

[1728] The server uses a high-performance computing system equipped with an NVIDIA GPU. For software, it uses deep learning frameworks such as TensorFlow and PyTorch to execute generative AI models, and natural language processing (NLP) libraries such as spaCy and NLTK to analyze the data.

[1729] The terminal is a device for a user to input a prompt sentence and display the generated response and the evaluation result. The terminal can be a general personal computer or a smartphone.

[1730] System Operation

[1731] 1. The user enters a prompt through the terminal.

[1732] The user enters a prompt sentence into the input field of the terminal, for example, "If the generating AI responded, 'Thank you, that's enough,' how would you rate that response?"

[1733] 2. The terminal sends the prompt text entered to the server

[1734] The terminal sends the entered prompt text to the server, using HTTP or WebSocket as the communication protocol.

[1735] 3. The server uses the generative AI model to generate a response to the prompt.

[1736] The server runs a generative AI model built using TensorFlow and PyTorch to generate a response to the prompt, such as "That's a very appropriate response."

[1737] 4. The server generates a response and sends it to the device.

[1738] The server then sends the generated response to the terminal, again using HTTP or WebSocket as the communication protocol.

[1739] 5. The terminal displays the generated response to the user.

[1740] The terminal displays the response received from the server to the user. For example, it displays "That response is very appropriate" on the screen.

[1741] 6. The user enters a response to the generated response

[1742] The user enters their reaction to the generated response, for example, "Thanks, that's good enough."

[1743] 7. The device sends the user's response to the server

[1744] The terminal sends the user's response to the server, again using HTTP or WebSocket as the communication protocol.

[1745] 8. The server analyzes the user's response and gives an evaluation.

[1746] The server uses natural language processing libraries (e.g., spaCy or NLTK) to analyze the user's response and rate its appropriateness. For example, a response like "Thanks, that's enough" is highly rated.

[1747] 9. The server sends the evaluation results to the device.

[1748] The server then transmits the evaluation results to the terminal, again using HTTP or WebSocket as the communication protocol.

[1749] 10. The terminal displays the evaluation results to the user.

[1750] The terminal displays the evaluation result received from the server to the user. For example, it displays "Your response is very appropriate" on the screen.

[1751] In this way, a system is realized in which the server, terminal, and user work together to generate and evaluate responses using a generative AI model. This system enables interactive response generation and evaluation using generative AI, accurately estimating the subject's emotional state and improving the user experience.

[1752] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1753] Step 1:

[1754] The user enters a prompt through the terminal.

[1755] Specifically, the user inputs a prompt sentence into the input field of the terminal. For example, the user inputs, "If the generating AI responded, 'Thank you, that's enough,' how would you evaluate that response?" The input data is the prompt sentence in text format.

[1756] Step 2:

[1757] The terminal sends the prompt text entered to the server.

[1758] Specifically, the terminal sends the input prompt text to the server using HTTP or WebSocket. The input data is the prompt text entered by the user, and the output data is the prompt text sent to the server.

[1759] Step 3:

[1760] The server uses a generative AI model to generate a response to the prompt.

[1761] Specifically, the server runs a generative AI model built using TensorFlow and PyTorch to generate a response to the prompt. The input data is the prompt, and the output data is the generated response. For example, it generates a response such as "That response is very appropriate."

[1762] Step 4:

[1763] The server sends the generated response to the terminal.

[1764] Specifically, the server sends the generated response to the terminal using HTTP or WebSocket. The input data is the generated response, and the output data is the response sent to the terminal.

[1765] Step 5:

[1766] The terminal displays the generated response to the user.

[1767] Specifically, the terminal displays the response received from the server to the user. The input data is the response sent from the server, and the output data is the response displayed to the user. For example, "That response is very appropriate" is displayed on the screen.

[1768] Step 6:

[1769] The user enters a reaction to the generated response.

[1770] As a specific operation, the user inputs his / her reaction to the generated response. The input data is the user's reaction, and the output data is the reaction input to the terminal. For example, the user inputs "Thank you, that's enough."

[1771] Step 7:

[1772] The device transmits the user's response to the server.

[1773] Specifically, the terminal transmits the user's response to the server using HTTP or WebSocket. The input data is the user's response, and the output data is the response transmitted to the server.

[1774] Step 8:

[1775] The server analyzes the user's response and provides an evaluation.

[1776] Specifically, the server uses a natural language processing library (e.g., spaCy or NLTK) to analyze the user's response and evaluate its appropriateness. The input data is the user's response, and the output data is the evaluation result. For example, a response like "Thanks, that's enough" is highly evaluated.

[1777] Step 9:

[1778] The server transmits the evaluation results to the terminal.

[1779] Specifically, the server sends the evaluation results to the terminal using HTTP or WebSocket. The input data is the evaluation results, and the output data is the evaluation results sent to the terminal.

[1780] Step 10:

[1781] The terminal displays the evaluation results to the user.

[1782] As a specific operation, the terminal displays the evaluation result received from the server to the user. The input data is the evaluation result sent from the server, and the output data is the evaluation result displayed to the user. For example, "Your response is very appropriate" is displayed on the screen.

[1783] (Application example 2)

[1784] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1785] Conventional customer service systems did not provide sufficient support for store clerks to respond appropriately to customers, making it difficult to improve customer satisfaction. In addition, there was a lack of means to evaluate the appropriateness of responses based on customer reactions, making it difficult to improve the response skills of store clerks. Furthermore, there was a need for a system that could efficiently handle customer service in physical stores.

[1786] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1787] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for generating responses to support customer service in a physical store, a means for evaluating the appropriateness of the response based on the customer's reaction, and a means including an application to be installed on a smartphone. This enables support for store clerks to respond appropriately to customers, thereby improving customer satisfaction and the response skills of store clerks.

[1788] "Generative AI" is a system that uses artificial intelligence technology to generate appropriate responses to user input.

[1789] The "complainer model" is a model for appropriately responding to complaints and inquiries that arise in customer service.

[1790] "Subject" refers to a user who participates in the evaluation and testing of the system.

[1791] "Biometrics" refers to physical data such as a subject's gaze, tone of voice, heart rate, etc.

[1792] "Brick and mortar store" refers to a sales or service location that is located in a physical location.

[1793] "Customer service" refers to the way store clerks respond to and provide services to customers.

[1794] "Response appropriateness" is the measure of how well the generated response meets the customer's requirements and expectations.

[1795] A "smartphone" is a multi-function mobile device that, in addition to the functions of a mobile phone, can connect to the Internet and run applications.

[1796] An "Application" is a software program designed to provide a particular function or service.

[1797] A system for implementing this invention includes a means for incorporating a generative AI and creating a complainer model that can interactively respond to comments made by a subject, a means for evaluating the subject's response, a means for including biometric information such as the subject's gaze and tone of voice in response generation / evaluation, a means for generating responses to assist with customer service in physical stores, a means for evaluating the appropriateness of the response based on the customer's reaction, and a means for including an application to be installed on a smartphone.

[1798] System Programming

[1799] The system is implemented using Python. OpenAI's GPT-3 is used as the generative AI model. The server receives the prompt entered by the store clerk and sends it to the generative AI model to generate a response. The generated response is displayed on the smartphone, providing assistance to the store clerk in providing an appropriate response to the customer.

[1800] Processing Description

[1801] The server receives the prompt entered by the store clerk and sends it to the generative AI model using the OpenAI API. The generative AI model generates an appropriate response based on the prompt and returns the response to the server. The server sends the generated response to the smartphone, and the store clerk responds to the customer. The customer's response is collected through the smartphone's microphone and camera and sent to the server. The server evaluates the appropriateness of the response based on the customer's response and provides feedback to the store clerk.

[1802] Hardware and Software Used

[1803] Hardware: Smartphones, servers

[1804] Software: Python, OpenAI API

[1805] Examples

[1806] For example, if a sales associate types, "Tell me what products you're looking for," the generative AI model responds with, "What kind of products are you looking for? For example, clothing or electronics?" If the customer responds, "Thanks, that's fine," this response is scored as "highly rated."

[1807] Example prompt:

[1808] "Tell us what product you're looking for."

[1809] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1810] Step 1:

[1811] The store clerk enters the prompt text into his / her smartphone.

[1812] Input: Prompt text to be entered by the salesperson (e.g. "Tell me about the product you are looking for.")

[1813] Output: The prompt text is sent to the server.

[1814] Specific actions: The store clerk opens the application on his / her smartphone, enters a prompt in the text box, and presses the send button.

[1815] Step 2:

[1816] The server receives the prompt text and sends it to the generative AI model.

[1817] Input: Prompt text sent from smartphone

[1818] Output: The prompt sent to the generative AI model

[1819] Specific operation: The server creates a request to send the prompt text received from the smartphone to the OpenAI API and sends it to the API endpoint.

[1820] Step 3:

[1821] A generative AI model generates a response based on the prompt.

[1822] Input: Prompt text sent from the server

[1823] Output: The generated response text

[1824] Specific operation: The generative AI model (e.g. GPT-3) analyzes the received prompt and generates an appropriate response, which is returned to the server as an API response.

[1825] Step 4:

[1826] The server receives the generated response and sends it to the smartphone.

[1827] Input: Response text returned by the generative AI model

[1828] Output: Response text sent to smartphone

[1829] Specific operation: The server analyzes the response text received from the generative AI model and creates a message to send to the smartphone. The message is sent to the smartphone application.

[1830] Step 5:

[1831] The smartphone displays the response text, and the store clerk responds to the customer.

[1832] Input: The response text sent by the server

[1833] Output: The response the salesperson gives to the customer

[1834] Specific operation: The smartphone application displays the received response text on the screen. The store clerk uses the displayed response as a reference to respond appropriately to the customer.

[1835] Step 6:

[1836] The smartphone collects customer responses and sends them to the server.

[1837] Input: Customer response (voice, gaze, facial expression, etc.)

[1838] Output: Customer response data sent to the server

[1839] How it works: The smartphone's microphone and camera are used to collect customer responses, and the collected data is sent to a server.

[1840] Step 7:

[1841] The server evaluates the appropriateness of the response based on the customer's reaction and provides feedback to the store attendant.

[1842] Input: Customer response data

[1843] Output: Response appropriateness assessment and feedback

[1844] Specific operation: The server analyzes the received customer response data and evaluates the appropriateness of the response. The evaluation result is sent to the store clerk's smartphone as feedback.

[1845] Example 3

[1846] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1847] In conventional interactive systems, responses to the subject's utterances were uniform, making it difficult to accurately grasp the subject's emotional state and intentions. As a result, it was not possible to generate appropriate responses according to the subject's emotions, resulting in a poor user experience. Furthermore, it was not possible to utilize biological information such as the subject's gaze or tone of voice, resulting in low response accuracy.

[1848] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1849] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for collecting gaze data and tone of voice data of the subject using a high-resolution camera and microphone, an image processing means for analyzing the collected gaze data, a voice analysis means for analyzing the collected tone of voice data, and a means for grasping the emotional state and intention of the subject based on the analyzed gaze data and tone of voice data and generating an appropriate response. This makes it possible to accurately grasp the emotional state and intention of the subject and generate an appropriate response accordingly.

[1850] "Generative AI" is an artificial intelligence system that generates responses in real time based on the subject's statements and biometric information.

[1851] The "complainer model" is a model that responds interactively to comments made by subjects, and is constructed using generative AI.

[1852] "Biometric information" is information obtained from a subject's body, such as their gaze or tone of voice, and is used to understand their emotional state and intentions.

[1853] A "high-resolution camera" is a camera for collecting gaze data from subjects, and is a device that has high resolution and can obtain detailed gaze information.

[1854] A "microphone" is a voice input device for collecting tone data of the subject's voice.

[1855] "Image processing means" refers to technology for analyzing collected gaze data and identifying the subject's gaze direction and movement.

[1856] The "voice analysis means" is a technology for analyzing collected voice tone data and detecting changes in the subject's voice tone.

[1857] The "response generation means" is a technology for understanding the subject's emotional state and intentions based on analyzed gaze data and tone of voice data, and generating an appropriate response.

[1858] This invention is a system that incorporates generative AI and can interactively respond to the subject's comments. Specifically, it is a system that collects biological information such as the subject's gaze and tone of voice, and generates / evaluates a response based on that information. A specific embodiment of this system is described below.

[1859] Hardware and Software Used

[1860] Hardware:

[1861] High-resolution camera: Used to collect subject's gaze data.

[1862] Microphone: Used to collect subject's vocal tone data.

[1863] Server: Equipped with high-performance CPU and GPU, it performs data analysis and response generation.

[1864] software:

[1865] OpenCV: An image processing library for analyzing gaze data.

[1866] PyAudio: An audio processing library for analyzing vocal tone data.

[1867] TensorFlow: A machine learning framework for running generative AI models.

[1868] System processing flow

[1869] Data collection:

[1870] The server activates a high-resolution camera to capture the user's face. The camera tracks the user's eye movements in real time to collect gaze data. The server also activates a microphone to record the user's voice. The microphone collects data for analyzing the tone of the user's voice in real time. The user speaks naturally into the camera and microphone while interacting with the system.

[1871] Data Analysis:

[1872] The server uses OpenCV to analyze the video data obtained from the camera. Specifically, it detects the position of the user's pupils and identifies the direction of gaze. For example, if the user is looking at the top right of the screen, that direction is identified. The server uses PyAudio to analyze the recorded audio data. Specifically, it analyzes the frequency components of the audio and detects changes in the tone of the voice. For example, if the tone of the user's voice becomes higher, it is determined that the user is excited.

[1873] Response generation:

[1874] The server uses TensorFlow to input gaze data and tone of voice data into the generative AI model. The generative AI model understands the user's emotional state and intentions based on this data. For example, if it determines that the user is excited, it generates a response that corresponds to that state. The server uses the generative AI model to generate a response appropriate to the user's state. For example, if the user is excited, it generates a response to respond calmly and chooses words to calm the user.

[1875] Examples of concrete examples and prompts

[1876] Examples:

[1877] Suppose a user is using an online customer support system. When the user asks a question, the system analyzes the user's gaze and tone of voice to determine whether the user is excited. For example, if the user's tone of voice becomes higher, the system determines that the user is excited and generates a response to help the user stay calm.

[1878] Example prompt:

[1879] "When a user is using your online customer support system, analyze their gaze and tone of voice to understand their emotional state and generate a response accordingly."

[1880] In this way, the server can generate a response based on the biometric information of the user, and can take a more accurate and appropriate action. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1881] Step 1:

[1882] Data collection

[1883] The server activates a high-resolution camera and microphone to collect the user's gaze data and tone of voice data. Specifically, the camera captures the user's face and tracks eye movements in real time, and the microphone records the user's voice and collects data for real-time tone of voice analysis.

[1884] Input: User's face and voice

[1885] Output: Gaze data and tone of voice data

[1886] Step 2:

[1887] Analysis of gaze data

[1888] The server uses OpenCV to analyze the video data obtained from the camera. Specifically, it detects the position of the user's pupils and identifies the direction of the user's gaze. For example, if the user is looking at the upper right corner of the screen, that direction is identified.

[1889] Input: Gaze data

[1890] Output: Gaze direction information

[1891] Step 3:

[1892] Analysis of voice tone data

[1893] The server uses PyAudio to analyze the recorded voice data. Specifically, it analyzes the frequency components of the voice and detects changes in the tone of the voice. For example, if the user's voice tone becomes higher, it is determined that the user is excited.

[1894] Input: Voice tone data

[1895] Output: Voice tone change information

[1896] Step 4:

[1897] Emotional state estimation

[1898] The server understands the user's emotional state and intentions based on the analyzed gaze data and tone of voice data. Specifically, it determines whether the user is excited by combining gaze direction information and tone of voice change information.

[1899] Input: Gaze direction information, voice tone change information

[1900] Output: Emotional state information

[1901] Step 5:

[1902] Response Generation

[1903] The server uses TensorFlow to input emotional state information into a generative AI model to generate an appropriate response. For example, if the server determines that the user is excited, it generates a response according to that state.

[1904] Input: Emotional state information

[1905] Output: Response message

[1906] Step 6:

[1907] Providing a response

[1908] The server provides the generated response message to the user, either by displaying it in text form or by responding audibly using speech synthesis.

[1909] Input: Response message

[1910] Output: Response to the user

[1911] In this way, the server can generate a response based on the user's biometric information, enabling more accurate and appropriate action to be taken.

[1912] (Application example 3)

[1913] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1914] Conventional customer service systems have the problem that it is difficult to accurately grasp the emotional state and intentions of customers, and they are unable to generate appropriate responses. In addition, there is a lack of technology that utilizes biometric information such as gaze and tone of voice, and effective means to improve customer satisfaction are required. Furthermore, when introducing customer service robots into brick-and-mortar stores, flexible responses according to the customer's emotions and intentions are required.

[1915] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1916] In this invention, the server includes a means for incorporating generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for grasping the subject's emotional state and intention and generating a response accordingly, and a means for being installed in a customer service robot in a brick-and-mortar store. This makes it possible to accurately grasp the customer's emotional state and intention and generate the optimal response accordingly.

[1917] "Generative AI" is an artificial intelligence system that interactively generates responses based on the subjects' statements and actions.

[1918] The "complainer model" is a model for generating appropriate responses to the statements or actions of customers or subjects.

[1919] An "evaluation means" is a means for evaluating the subject's response or behavior.

[1920] "Biometric information" is information obtained from the subject's body, such as their gaze or tone of voice.

[1921] The "response generating means" is a means for generating an appropriate response based on the subject's utterances and biometric information.

[1922] "Emotional state" refers to the subject's current emotional state.

[1923] "Intention" refers to what the subject is thinking and intending to do.

[1924] A "brick and mortar store" is a store that exists in a physical location.

[1925] A "customer service robot" is a robot that provides service to customers in a physical store.

[1926] A system for implementing this invention includes a means for incorporating generative AI and creating a complainer model that can interactively respond to comments made by a subject, a means for evaluating the subject's response, a means for including biometric information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for grasping the subject's emotional state and intentions and generating a response accordingly, and a means for being installed in a customer service robot in a physical store.

[1927] System Programming

[1928] The program of this system uses the following hardware and software:

[1929] Hardware: Camera (for gaze detection), microphone (for voice recognition)

[1930] Software: OpenCV (for face recognition and gaze detection), SpeechRecognition (for voice recognition), Transformers (for sentiment analysis)

[1931] Processing flow

[1932] The server captures video and audio in real time using a camera and microphone. It uses OpenCV to detect faces and eyes from the camera footage, and SpeechRecognition to convert the audio captured from the microphone into text. The converted text is analyzed for sentiment using a generative AI model from Transformers. Based on the analysis results, an appropriate response is generated.

[1933] Examples

[1934] For example, if a customer is excited, it is determined that the customer's voice tone is high and their gaze is focused on the product. In this case, the customer service robot will respond, "Sir, what product are you looking for today?" On the other hand, if a customer is in trouble, it is determined that the customer's voice tone is low and their gaze is unsteady. In this case, the customer service robot will respond, "If there's anything I can help you with, I'd like to help you."

[1935] Examples of prompt statements

[1936] "Generate responses where the customer's voice tone is high and their gaze is focused on the product."

[1937] In this way, customer service robots in physical stores will be able to accurately grasp the emotional state and intentions of customers and provide the most appropriate response accordingly.

[1938] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1939] Step 1:

[1940] The server uses a camera and microphone to capture video and audio in real time.

[1941] Input: Camera video, microphone audio

[1942] Output: Video data, audio data

[1943] Specific operation: Captures the customer's face and gaze through a camera, and records what the customer says through a microphone.

[1944] Step 2:

[1945] The server uses OpenCV to detect faces and eyes from camera footage.

[1946] Input: Video data

[1947] Output: Face and eye position information

[1948] Specific operation: Apply OpenCV's facial recognition algorithm to identify the face and eye positions from video data.

[1949] Step 3:

[1950] The server uses SpeechRecognition to convert the audio data into text.

[1951] Input: Audio data

[1952] Output: Text data

[1953] What it does: Apply a speech recognition algorithm to convert the audio data into text.

[1954] Step 4:

[1955] The server uses Transformers' generative AI models to parse sentiment from text data.

[1956] Input: Text data

[1957] Output: Emotion analysis results

[1958] How it works: Input text data into a generative AI model, and output an emotional state (e.g. positive, negative, neutral).

[1959] Step 5:

[1960] The server generates an appropriate response based on the face and eye position information and emotion analysis results.

[1961] Input: Face and eye position information, emotion analysis results

[1962] Output: Response text

[1963] Specific operation: Determine the direction of gaze from face and eye position information, combine it with emotion analysis results, and generate an appropriate response using a generative AI model.

[1964] Step 6:

[1965] The server transmits the generated response text to a customer service robot, which provides a response to the customer.

[1966] Input: Response text

[1967] Output: Audio response

[1968] Specific operation: The response text is input to the speech synthesis system and output as a voice. The customer service robot plays this voice and provides the response to the customer.

[1969] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1970] "Example 1"

[1971] One embodiment of the present invention is to incorporate an emotion engine into the generative AI. This emotion engine recognizes the emotional state of the subject from the tone of voice, facial expression, eye movement, etc., and feeds this information back to the generative AI. The generative AI generates a response to the subject based on this emotion information. For example, if the subject shows an emotion of anger, the generative AI receives that information and generates a response that soothes the subject.

[1972] "Example 2"

[1973] The emotion engine also has the ability to adjust the evaluation based on the subject's emotional state. Specifically, if the subject is happy, the response is rated high. Conversely, if the subject is unhappy, the response is rated low. In this way, the emotion engine grasps the subject's emotional state in real time and adjusts the evaluation accordingly.

[1974] "Example 3"

[1975] In addition, the emotion engine also has the ability to adjust responses based on the subject's emotional state. For example, if the subject shows signs of surprise, the generative AI will receive that information and generate a response that will reassure the subject. In this way, the emotion engine understands the subject's emotional state in real time and adjusts responses accordingly.

[1976] The process flow of each embodiment will be described below.

[1977] "Example 1"

[1978] Step 1: The generative AI collects information from the subject's tone of voice, facial expressions, eye movements, etc.

[1979] Step 2: Send the collected information to the emotion engine.

[1980] Step 3: The emotion engine recognizes the subject's emotional state and feeds that information back to the generative AI.

[1981] Step 4: The generative AI generates a response to the subject based on the feedback information. "Example 2"

[1982] Step 1: The generative AI sends the subject’s emotional state to the emotion engine.

[1983] Step 2: The emotion engine adjusts the rating based on the subject's emotional state.

[1984] Step 3: The adjusted evaluation is fed back to the generative AI.

[1985] Step 4: The generative AI generates a response to the subject based on the feedback information. "Example 3"

[1986] Step 1: The generative AI sends the subject’s emotional state to the emotion engine.

[1987] Step 2: The emotion engine adjusts the response based on the subject's emotional state.

[1988] Step 3: The adjusted response is fed back to the generative AI.

[1989] Step 4: The generative AI generates a response for the subject based on the feedback information.

[1990] Example 1

[1991] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1992] In conventional customer service simulation systems, responses to comments made by subjects were fixed, making interactive responses difficult. In addition, responses could not be generated that took into account the subject's emotional state, making it impossible to realize a realistic customer service simulation. Furthermore, evaluations of the subjects' responses were subjective, making it difficult to make objective evaluations.

[1993] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1994] In this invention, the server includes a means for incorporating a generative AI and creating a complainer model capable of interactively responding to the subject's remarks, a means for evaluating the subject's response, a means for including biological information such as the subject's gaze and tone of voice in the response generation / evaluation, a means for capturing...

Claims

1. A means for acquiring voice input from a user through a smart device worn by a staff member and converting the voice data into text data using a voice recognition system; means for transmitting the converted text data to a generating AI and generating an appropriate response based on the text data; means for converting the response into voice data using a voice synthesis system; means for analyzing a reaction to the response with an emotion engine; means for evaluating the appropriateness of the response based on the emotional state of the user obtained from the analysis result of the emotion engine, and providing feedback to the staff; A system including:

2. The system of claim 1 , wherein the generating AI generates the response in real time.

3. The system of claim 1 , wherein biometric information of the user in reaction to the response is used to infer an emotional state of the user.

Images