system

Abstract

We provide the system. [Solution] A means for training a generative artificial intelligence model using past customer service records and operation manuals, A means of converting customer inquiry audio received in real time into text, A means for receiving the converted text or chat text and generating a model answer using a generative artificial intelligence model, A system including means for displaying the generated model answer on an operator display.

Description

Technical Field

[0004] , , , ,

[0005] , , , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention relates to a system for reducing the load on customer service operators and shortening the customer response time. Specifically, in conventional customer service, in order for an operator to quickly and appropriately respond to customer inquiries, it is necessary to refer to a large amount of manuals and past records, which requires a great deal of time and labor. In addition, there is also a problem that the response quality varies due to differences in the expertise and experience of operators. Therefore, it has been required to reduce the burden on operators and improve the consistency and efficiency of inquiry responses.

Means for Solving the Problems

[0005] The present invention solves these problems by including means for training a generative artificial intelligence model using past customer service records and operation manuals, means for converting customer inquiry voice received in real time into text, means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model, and means for displaying the generated model answer on an operator display (Claim 1). Furthermore, by utilizing speech recognition technology when converting voice to text in real time, customer voice inquiries can be responded to quickly (Claim 2). In addition, by adding a function to display a screen for viewing relevant information based on the customer's inquiry when generating a model answer, operators can quickly access relevant information and provide consistent responses (Claim 3). This reduces the burden on operators and improves response time and quality.

[0006] "Past customer service records" refer to records of responses made by operators to customer inquiries, and include data in various formats such as voice, text, and chat logs.

[0007] An "operation manual" is a document that outlines procedures and guidelines for operators to refer to when handling customer inquiries.

[0008] A "generative artificial intelligence model" refers to an algorithm and system that is trained using historical data to generate answers in natural language.

[0009] "Converting to text" refers to the process of converting audio data into text information using speech recognition technology.

[0010] A "model answer" refers to text information that shows the optimal response to a customer inquiry, as determined by a generative artificial intelligence model.

[0011] An "operator display" refers to an information display screen used by operators to provide appropriate responses to customer inquiries.

[0012] An "information viewing screen" refers to a screen element that visually displays data that can be referenced by the operator.

[0013] "Speech recognition technology" refers to the technology that analyzes speech data and converts it into text format. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then uses the generative artificial intelligence model to generate a model answer, and finally displays it on an operator display.

[0036] Initial setup and learning phase

[0037] 1. The server retrieves past customer interaction records in both audio and text format from the database and converts these records to text. In the case of audio records, speech recognition technology is used to convert them to text.

[0038] 2. The server uses the converted text data to train a generative artificial intelligence model (e.g., a natural language processing model). This training process includes data preprocessing (noise removal, formatting standardization, etc.).

[0039] 3. The server deploys the trained generative artificial intelligence model and configures API endpoints so that they can be used on each operator display terminal.

[0040] Real-time response support phase

[0041] 1. The user contacts customer support by phone or chat.

[0042] In the case of phone inquiries, the terminal captures the user's voice in real time.

[0043] In the case of chat support, the device directly receives the text message sent by the user.

[0044] 2. In the case of a phone inquiry, the terminal uses speech recognition software to convert the user's voice into text.

[0045] The converted text is stored in the device's temporary memory.

[0046] 3. The device sends the text converted from the voice or the received chat text to the server.

[0047] This transmission is in the form of an HTTP request and includes the query content and metadata (query type, time, etc.).

[0048] 4. The server inputs the received text into a generative artificial intelligence model and generates a model answer.

[0049] For example, in response to an inquiry such as "Please tell me about returns," a generative artificial intelligence model will generate a model answer such as, "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0050] 5. The server sends the generated model answer back to the terminal.

[0051] This response is also sent in HTTP response format and includes not only the model answer but also related information (for example, a link to the relevant page in the operation manual).

[0052] 6. The terminal displays the model answer received from the server on the operator's display.

[0053] The information displayed includes the answers and related documents the user is looking for.

[0054] 7. The operator will provide appropriate assistance to the user, referring to the provided model answer.

[0055] If necessary, the operator will assist the customer while supplementing the information with relevant details.

[0056] 8. The terminal also controls the information viewing screen, assisting operators in quickly accessing relevant information.

[0057] For example, if an inquiry is received regarding the return process, the detailed return procedure page will be automatically displayed.

[0058] Specific example

[0059] Example 1: Telephone inquiry

[0060] 1. The user inquires about the return procedure for the product.

[0061] 2. The device converts the user's voice into text in real time (e.g., "I would like to return the item, how do I do that?").

[0062] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[0063] 4. The server sends this model answer back to the terminal, which then displays it on the operator's display.

[0064] 5. The operator provides appropriate instructions to the user based on the displayed answer.

[0065] Example 2: Chat support

[0066] 1. The user sends a question via chat about a network connection problem.

[0067] 2. The terminal forwards this text to the server as is.

[0068] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[0069] 4. The server sends the generated model answer back to the terminal, which then displays it on the operator's chat screen.

[0070] 5. The operator replies to the user with the displayed information.

[0071] Thus, the system of the present invention can reduce the burden on operators, shorten response times, and improve customer satisfaction.

[0072] The following describes the processing flow.

[0073] Step 1:

[0074] The server retrieves past customer interaction records in both audio and text format from the database and converts these records to text. For retrieved audio records, speech recognition software is used to convert them to text.

[0075] Step 2:

[0076] The server preprocesses this text data. Preprocessing includes text cleanup (noise removal, formatting standardization, etc.).

[0077] Step 3:

[0078] The server trains a generative artificial intelligence model using pre-processed data. During the training process, customer interaction records and manual datasets are input into the model, and natural language processing techniques are applied to optimize the model's parameters.

[0079] Step 4:

[0080] The server stores the trained generative artificial intelligence model and deploys it. It then configures an API endpoint to make it available from the operator display terminal.

[0081] Step 5:

[0082] A user contacts customer support. For phone inquiries, the device captures the user's voice in real time. For chat inquiries, the device receives text messages directly.

[0083] Step 6:

[0084] When a user makes a voice inquiry, the device uses speech recognition software to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[0085] Step 7:

[0086] The terminal sends the transcribed query content to the server. This transmission is in the form of an HTTP request and includes the query content and metadata (e.g., query type, time, etc.).

[0087] Step 8:

[0088] The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0089] Step 9:

[0090] The server sends the generated model answer back to the terminal in HTTP response format. The response includes the model answer and any related additional information (e.g., links to relevant manual pages).

[0091] Step 10:

[0092] The terminal displays the received model answer on the operator's display. The displayed information includes the appropriate answer to the user's inquiry and related documents.

[0093] Step 11:

[0094] The operator provides the user with an answer, referring to the displayed model answer. If necessary, the operator provides the user with appropriate instructions, supplementing them with relevant information.

[0095] Step 12:

[0096] The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, the terminal automatically displays the detailed return procedure page.

[0097] This series of steps reduces the burden on operators and shortens response times.

[0098] (Example 1)

[0099] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0100] In modern customer support operations, responding quickly and accurately to customer inquiries is crucial. However, customer inquiries cover a wide range of topics, requiring operators to provide appropriate responses each time. This is time-consuming and labor-intensive, and the quality of service can vary from operator to operator. Furthermore, a lack of systems that allow for quick access to information related to inquiries leads to longer response times and a decline in customer satisfaction.

[0101] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0102] In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry audio received in real time into text; means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model; means for automatically displaying an information viewing screen related to the inquiry; means for constructing prompt sentences for generating a model answer using the generative artificial intelligence model; and means for providing additional information related to the model answer (such as a link to the relevant page in the operation manual). This enables operators to respond to customer inquiries quickly and accurately, reduces variability in the quality of responses, and improves customer satisfaction.

[0103] "Past customer interaction records" refer to all communications and interaction history recorded between customers and support operators in the past.

[0104] An "operator's manual" is a document that describes how to use a product or service, troubleshooting procedures, and other related information.

[0105] A "generative artificial intelligence model" is a machine learning model that has the ability to learn from large amounts of text data and perform natural language generation tasks.

[0106] "Inquiry voice messages" refer to audio information transmitted by customers via telephone or voice message.

[0107] "Converting to text" refers to the process of converting audio data into written text.

[0108] "Chat text" refers to the text information that customers send via text chat.

[0109] A "model answer" is the optimal response to a customer inquiry generated by a generative artificial intelligence model.

[0110] An "operator display" refers to a display device or interface used by an operator.

[0111] "Speech recognition technology" is a technology that analyzes speech data and converts it into text data.

[0112] An "information viewing screen" is a display screen used by operators to check relevant information.

[0113] A "prompt statement" is a guide statement that is input to a generative artificial intelligence model.

[0114] This invention relates to a system that trains a generative artificial intelligence model using past customer service records and operation manuals, converts customer inquiries into text in real time, then uses the generative artificial intelligence model to generate model answers, and finally displays them on the operator's display. This system enables operators to provide customer service quickly and effectively.

[0115] First, I will explain the system's initial setup and learning phase.

[0116] The server retrieves past customer interaction records in both audio and text format from a database. Common databases used for this purpose include MySQL® and PostgreSQL. Next, the server converts the retrieved audio records into text using speech recognition technologies such as Google® Cloud Speech-to-Text or Amazon Transcribe. This conversion process extracts the audio data as text information.

[0117] The server then preprocesses this text data. Preprocessing involves removing noise from the data and standardizing its format. The Python pandas library can be used for preprocessing. The preprocessed data is then used to train a generative artificial intelligence model (for example, OpenAI's GPT-4 or Google's BERT). Deep learning frameworks such as Tensorflow or PyTorch are commonly used for this training process.

[0118] The trained generative artificial intelligence model is deployed to a cloud platform such as AWS® EC2 or Google Cloud. After deployment, an API endpoint is configured so that it can be used on each operator display terminal. This API is often built using web frameworks such as Flask or Django.

[0119] Next, I will explain the real-time response support phase.

[0120] When a user contacts customer support via phone or chat, the device captures the user's voice in real time using a digital microphone in the case of a phone call. In the case of a chat, the device directly receives the text message sent by the user. Slack API and Zendesk Chat are used to receive chat messages.

[0121] The device then uses Google Cloud Speech-to-Text to convert the user's speech to text, in the case of a phone call. The converted text is stored in temporary memory. The device then sends the converted text or chat text to the server. This transmission is in the form of an HTTP POST request and includes metadata such as the content of the inquiry, the type of inquiry, and the time.

[0122] The server inputs the received text into a generative artificial intelligence model. A prompt is used in this process. For example, by inputting a prompt such as "The user asked: Please tell me about the return procedure," the generative AI model generates a model answer. This model answer might be something like, "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0123] The generated model answer is sent back to the terminal in HTTP response format. In addition to the model answer, it also includes relevant information (e.g., a link to the relevant page in the operation manual). The terminal displays this model answer on the operator display. The display includes the answer the user is looking for and related documents. The operator uses the provided model answer as a reference to respond appropriately to the user. If necessary, the operator will provide additional information to assist the customer.

[0124] Furthermore, the terminal also controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry about a return is received, it automatically displays the detailed return procedure page. In this way, the system can reduce the burden on operators, shorten response times, and improve customer satisfaction.

[0125] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0126] Step 1:

[0127] The server retrieves past customer interaction records in audio and text format from a database. MySQL is used as the database for retrieval. The input data consists of customer interaction records, and the output is the retrieved record data.

[0128] Step 2:

[0129] The server converts audio data from acquired customer interaction records into text using Google Cloud Speech-to-Text. The input data is the audio recording, and the output is the text data converted from the audio. Specifically, the audio file is sent to the API, and a reply is received in text format.

[0130] Step 3:

[0131] The server preprocesses the text data. Here, the Python pandas library is used to remove noise and standardize the text format. The input data is the converted text data, and the output is the preprocessed text data.

[0132] Step 4:

[0133] The server trains a generative artificial intelligence model (e.g., GPT-4) using preprocessed text data. TensorFlow is used for this training. The input data is preprocessed text data, and the output is the trained generative artificial intelligence model.

[0134] Step 5:

[0135] The server deploys the trained generative AI model to a cloud platform such as AWS EC2. Furthermore, it configures API endpoints for use on each operator display terminal. The input data is the trained generative AI model, and the output is the deployed model and the available API endpoints.

[0136] Step 6:

[0137] Users contact customer support via phone or chat. For phone inquiries, the device uses a digital microphone to capture the user's voice in real time. Input data is the user's voice inquiry or chat text, and output is the captured audio or directly received chat messages.

[0138] Step 7:

[0139] For voice inquiries, the device uses Google Cloud Speech-to-Text to convert speech to text. The input data is the captured audio, and the output is the converted text.

[0140] Step 8:

[0141] The terminal sends the converted text or received chat text to the server. The transmission is done via an HTTP POST request. The input data is the converted text or chat text, and the output is the text sent to the server.

[0142] Step 9:

[0143] The server inputs the received text into a generative artificial intelligence model. In doing so, it constructs and inputs a prompt. The input data consists of the received text and the prompt, and the output is the generated model answer. A concrete example of a prompt is: "The user asked: Please tell me about the return procedure."

[0144] Step 10:

[0145] The server sends the generated model answer back to the terminal. The return is also done in HTTP response format. The input data is the generated model answer and related information (such as a link to the operation manual), and the output is the model answer sent to the terminal.

[0146] Step 11:

[0147] The terminal displays the model answer received from the server on the operator's display. The input data is the model answer and related documents, and the output is the displayed information.

[0148] Step 12:

[0149] The operator responds to the user while referring to the provided model answer. Additional information is provided as needed to assist the customer. The input data consists of the displayed model answer and related information, while the output is the operator's response action.

[0150] Step 13:

[0151] The terminal automatically controls the information viewing screen so that operators can quickly access relevant information. For example, if there is an inquiry about the return procedure, it will automatically display the page detailing the return procedure. The input data is the inquiry content, and the output is the automatically displayed page of relevant information.

[0152] (Application Example 1)

[0153] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0154] A support system is needed in logistics centers to enable workers to efficiently process customer inquiries and work instructions. Existing systems require workers to manually verify information, which is time-consuming and can lead to misunderstandings and delays in responses. Furthermore, providing accurate instructions to workers in real time is difficult, resulting in insufficient improvements in work efficiency and reduction of errors. To address these issues, a real-time support system utilizing speech recognition technology and generative artificial intelligence is required.

[0155] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0156] In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry voices received in real time into text; means for receiving the converted text or chat text and generating model answers using the generative artificial intelligence model; means for displaying the generated model answers on a visual display device; and means for acquiring voice instructions from workers in the logistics center in real time using a basic speech recognition library and an external API, and displaying the generated model answers on smart glasses. This enables improved work efficiency, reduced errors, and faster customer response in the logistics center.

[0157] "Past customer service records" refers to data that records past customer inquiries and the details of the responses received.

[0158] An "operation manual" refers to a document or guide that describes specific procedures or operating methods.

[0159] A "generative artificial intelligence model" refers to an AI model that performs natural language processing based on a large amount of data to generate model answers or recommended responses.

[0160] "Real-time customer inquiry audio" refers to the immediate reception of audio data sent by customers via telephone or voice chat.

[0161] "Means of converting to text" refers to technologies and devices for converting audio data into text data.

[0162] "Chat text" refers to text data sent in text chat format.

[0163] A "visual display device" refers to a device used to visually display the generated model answer.

[0164] A "speech recognition library" refers to a software component used to convert speech data into text.

[0165] An "external API" refers to an interface used to connect with other systems or services.

[0166] "Smart glasses" refer to glasses-type wearable devices with a built-in display that can show information in real time.

[0167] A "logistics center" refers to a facility that stores, manages, and ships goods.

[0168] A "worker" refers to a person who is responsible for tasks such as receiving, managing, and shipping goods at a logistics center.

[0169] The system that implements this application example includes the following components.

[0170] Components

[0171] 1. Server

[0172] Database: Stores past customer service records and operation manuals.

[0173] Generative artificial intelligence models: Trained using past customer interaction records and operation manuals.

[0174] API endpoint: Provides an interface for accessing trained generative artificial intelligence models.

[0175] 2. Terminal

[0176] Smart glasses: Used by workers in logistics centers to display information in real time.

[0177] Speech recognition library: Converts worker speech into text in real time.

[0178] Data transmission / reception function: Exchanges text and model answers with the server.

[0179] 3. User

[0180] Worker: At the logistics center, they use the system to handle customer inquiries, receive work instructions, and respond to them.

[0181] Explanation of the process

[0182] 1. Preparing the database

[0183] The server retrieves past customer service records and operation manuals from a database and uses this data to train a generative artificial intelligence model.

[0184] 2. Training and Deployment

[0185] The server converts audio recordings into text using speech recognition technology, preprocesses the data using a generative artificial intelligence model, and trains the model. The trained generative AI model becomes accessible through an API endpoint.

[0186] 3. Real-time processing

[0187] Users wear smart glasses within the logistics center and make inquiries and give instructions using voice commands.

[0188] The speech recognition library converts the speech to text, and the device sends the text to the server.

[0189] The server inputs the received text into a generative artificial intelligence model and generates a model answer in real time.

[0190] The model answer is sent to the device and displayed on the smart glasses' screen.

[0191] Hardware and software to be used

[0192] 1. Speech recognition library

[0193] Libraries used: Python's speech_recognition

[0194] Purpose: Converts audio data to text.

[0195] 2. External API

[0196] How to use: Send a query to a generative artificial intelligence model using an HTTP request and obtain a model answer.

[0197] 3. Smart Glasses

[0198] Device used: A wearable device in the form of glasses with a built-in display, worn by workers at a logistics center.

[0199] Specific example

[0200] In a logistics center, when a worker wears smart glasses to retrieve a specific item from a storage shelf, they are asked aloud, "What is the next item?" A speech recognition library captures this audio and converts it to text. The converted text is sent to a server, where a generative artificial intelligence model generates the optimal response. This response is displayed on the smart glasses' screen in the form of, "Please retrieve the item on shelf B5."

[0201] Example of a prompt

[0202] The following is an example of a prompt message that describes the specific operation of this system.

[0203] "Please explain how to create a system that uses speech recognition technology to convert a user's voice into text in real time, inputs that text into a generative AI model to generate a model answer, and displays the result on smart glasses."

[0204] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0205] Step 1:

[0206] Users wear smart glasses and make inquiries and give instructions by voice within the logistics center.

[0207] Input: User's voice (e.g., "What is the next product?")

[0208] Operation:

[0209] The user speaks into the microphone of the smart glasses. The smart glasses capture the audio.

[0210] Step 2:

[0211] The device uses a speech recognition library to convert the user's speech into text.

[0212] Input: User's voice captured in Step 1

[0213] Data processing: A speech recognition library is used to convert audio data into text.

[0214] Output: Text (Example: "What is the next product?")

[0215] Operation:

[0216] The speech recognition library (Python's speech_recognition) converts the speech into text data and stores it in temporary memory.

[0217] Step 3:

[0218] The terminal sends the converted text to the server in the form of an HTTP request.

[0219] Input: Text generated in Step 2

[0220] Data processing: Converting text data into HTTP request format.

[0221] Output: HTTP request sent to the server

[0222] Operation:

[0223] The text data is sent as an HTTP request to the server's API endpoint.

[0224] Step 4:

[0225] The server inputs the received text into a generative artificial intelligence model and generates a model answer.

[0226] Input: Text received as an HTTP request

[0227] Data processing: Generative artificial intelligence models generate the optimal model answer from text.

[0228] Output: Generated model answer (Example: "Please take the item from shelf B5.")

[0229] Operation:

[0230] A generative artificial intelligence model on the server generates a model answer based on the received text.

[0231] Step 5:

[0232] The server sends the generated model answer back to the terminal in HTTP response format.

[0233] Input: Generated model answer

[0234] Data processing: Convert model answers into HTTP response format.

[0235] Output: HTTP response sent to the terminal

[0236] Operation:

[0237] The server sends an HTTP response containing the model answer back to the terminal.

[0238] Step 6:

[0239] The device displays the received model answer on the smart glasses' screen.

[0240] Input: Model answer included in the HTTP response received in Step 5

[0241] Data processing: Convert model answers to a display format.

[0242] Output: Model answer displayed on the smart glasses screen (Example: "Please take the item from shelf B5.")

[0243] Operation:

[0244] The device converts the model answer data into an appropriate display format and displays it on the smart glasses' screen.

[0245] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0246] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then combines this with an emotion engine that recognizes the user's emotions, generates a model answer using the generative artificial intelligence model, and finally displays it on an operator display.

[0247] Initial setup and learning phase

[0248] 1. The server retrieves past customer interaction records in audio and text format from the database and converts these records to text. For retrieved audio records, speech recognition software is used to convert them to text.

[0249] 2. The server preprocesses this text data. Preprocessing includes text cleanup (noise removal, formatting standardization, etc.).

[0250] 3. The server trains a generative artificial intelligence model using pre-processed data. During the training process, the model is fed a dataset of customer interaction records and manuals, and natural language processing techniques are applied to optimize the model's parameters.

[0251] 4. The server saves the trained generative artificial intelligence model and deploys it. It configures an API endpoint to make it available from the operator display terminal.

[0252] Real-time response support phase

[0253] 1. The user contacts customer support. In the case of a phone inquiry, the device captures the user's voice in real time. In the case of a chat inquiry, the device receives the text message directly.

[0254] 2. When the device receives a phone inquiry, it uses speech recognition software to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[0255] 3. The terminal sends the text-based query to the server. This transmission is in the form of an HTTP request and includes the query content and metadata (e.g., query type, time, etc.).

[0256] 4. The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0257] 5. The server uses an emotion engine to analyze the user's emotions. In the case of voice queries, the server recognizes the user's emotions from the voice data; in the case of text queries, it recognizes them from the text data.

[0258] 6. Based on the sentiment data analyzed by the server, the generative artificial intelligence model adjusts the model answer. For example, if the user is angry, it will generate a model answer that includes a more courteous response.

[0259] 7. The server sends the generated model answer and sentiment data back to the terminal. The response includes the model answer and any additional relevant information (e.g., links to relevant manual pages).

[0260] 8. The terminal displays the received model answer and sentiment data on the operator display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and relevant documents.

[0261] 9. The operator provides the user with a response, referring to the displayed model answer and sentiment data. The operator provides appropriate instructions to the user, supplementing relevant information as needed.

[0262] 10. The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, the terminal automatically displays the detailed page for the return procedure.

[0263] Specific example

[0264] Example 1: Telephone inquiry

[0265] 1. The user inquires about the return procedure for the product.

[0266] 2. The device converts the user's voice into text in real time (e.g., "I would like to return the item, how do I do that?").

[0267] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[0268] 4. The server uses an emotion engine to analyze the user's voice and recognize that the user is dissatisfied.

[0269] 5. The server adjusts the model answer based on the analysis results and generates a more detailed explanation.

[0270] 6. The server sends this model answer and sentiment data back to the terminal, which then displays it on the operator's display.

[0271] 7. The operator will provide appropriate instructions to the user based on the displayed answer and adjust their response based on the displayed sentiment data.

[0272] Example 2: Chat support

[0273] 1. The user sends a question via chat about a network connection problem.

[0274] 2. The terminal forwards this text to the server as is.

[0275] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[0276] 4. The server uses an emotion engine to analyze the user's emotions from their text and recognizes that the user is anxious.

[0277] 5. The server adjusts the model answer based on the analysis results and generates an answer that includes calming language.

[0278] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays it on the operator's chat screen.

[0279] 7. Operators will respond to users quickly and appropriately, taking into account the displayed information. In doing so, they will strive to reassure users by considering emotional data.

[0280] In this way, the system of the present invention reduces the burden on the operator, shortens the response time, improves customer satisfaction, and enables appropriate responses according to the user's emotions.

[0281] The following describes the processing flow.

[0282] Step 1:

[0283] The server retrieves the past customer response records in voice and text formats from the database, and converts the voice records into text using voice recognition software. The obtained text data is pre-processed to remove noise and unify the format.

[0284] Step 2:

[0285] The server trains the generative artificial intelligence model using the pre-processed data. Natural language processing technology is utilized for training to optimize the model parameters. The trained model is saved, and an API endpoint is set so that it can be used on each operator display terminal.

[0286] Step 3:

[0287] The user makes an inquiry to customer support. In the case of a phone inquiry, the terminal captures the user's voice in real time. In the case of a chat inquiry, the terminal directly receives the text message.

[0288] Step 4:

[0289] When it is a phone inquiry, the terminal converts the user's voice into text in real time using voice recognition software. The converted text is temporarily saved in the terminal's memory.

[0290] Step 5:

[0291] The terminal sends the text-based query content to the server. The transmission is in HTTP request format and includes the query content and metadata (e.g., query type, time, etc.).

[0292] Step 6:

[0293] The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0294] Step 7:

[0295] The server uses an emotion engine to analyze the user's emotions. If audio data is available, it recognizes emotions from the audio; if text data is available, it analyzes emotions from the text. For example, it analyzes tone and speed of voice from audio, and word choice and expression from text.

[0296] Step 8:

[0297] The server adjusts the model response based on the analyzed sentiment data. If the user is clearly dissatisfied, it generates a more polite and helpful message. For example, it might add phrases like, "We apologize for your dissatisfaction with the return process..."

[0298] Step 9:

[0299] The server returns the generated model answer and sentiment data to the terminal in HTTP response format. The response includes the model answer and related additional information (e.g., links to relevant manual pages).

[0300] Step 10:

[0301] The terminal displays the received model answer and emotion data on the operator display. The information to be displayed includes the appropriate answer to the user's inquiry, emotion data, and relevant documents.

[0302] Step 11:

[0303] The operator refers to the displayed model answer and emotion data and answers the user. If necessary, the operator adjusts the response while referring to the displayed emotion data and provides appropriate instructions to the user.

[0304] Step 12:

[0305] The terminal controls the information browsing screen and assists the operator in quickly accessing relevant information. For example, when there is an inquiry regarding the return procedure, the detailed page of the return procedure is automatically displayed. Through this series of steps, the system can reduce the burden on the operator and shorten the response time. Also, appropriate responses according to the user's emotion can be achieved.

[0306] (Example 2)

[0307] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart device 14 is referred to as the "terminal".

[0308] Conventional customer support systems have the problem that it places a great burden on the operator when responding to customers' inquiries in real time. Also, it is difficult to respond according to the customers' emotions, and there has been a concern that customer satisfaction will decrease. Furthermore, due to the difficulty in quickly accessing relevant information, there has been a problem that the response time is extended and efficient support cannot be provided.

[0309] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry voice received in real time into text; means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model; means for analyzing the user's emotions and adjusting the generated model answer; means for displaying the generated model answer on an operator display; and means for automatically displaying a related information viewing screen so that the operator can quickly access related information. This makes it possible to reduce the burden on operators, shorten response times, and improve customer satisfaction.

[0310] "Past customer interaction records" refer to data that includes details of previous interactions with customers.

[0311] An "operator's manual" is a document that explains how to use a system or product.

[0312] A "generative artificial intelligence model" is a machine learning model that has the ability to generate natural language based on a large amount of data.

[0313] "Converting to text" is the process of converting non-text data, such as audio or images, into written text.

[0314] "Speech recognition technology" is a technology that converts speech data into text in real time.

[0315] A "model answer" is a standard or recommended response provided to a customer inquiry.

[0316] "Emotional analysis" is the process of identifying a customer's emotional state from their voice or text.

[0317] An "operator display" is a display device used by operators to view and manipulate information.

[0318] The "Related Information Viewing Screen" is a screen that displays the necessary information in response to customer inquiries.

[0319] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then combines it with an emotion engine to recognize the user's emotions, generates a model answer using the generative artificial intelligence model, and finally displays it on an operator display.

[0320] System Configuration

[0321] Hardware and software

[0322] The server manages the database, trains and deploys generative artificial intelligence models, and performs speech recognition and sentiment analysis. Software used includes Python, SQL, OpenAI GPT-3®, IBM Watson® Tone Analyzer, and Google Cloud Speech-to-Text.

[0323] Terminal: Captures user inquiries and exchanges data with the server. The terminal is equipped with voice recognition software, a browser, and an operator display.

[0324] User: Contact us by phone or chat.

[0325] Processing of the invention

[0326] Initial setup and learning phase

[0327] 1. The server retrieves past customer interaction records (in audio and text format) from the database and converts them to text using speech recognition software (Google Cloud Speech-to-Text).

[0328] 2. The server cleans up the acquired text and performs preprocessing to train a generative artificial intelligence model (OpenAI GPT-3).

[0329] 3. The server trains a generative artificial intelligence model using preprocessed data, and saves and deploys the model after training is complete. An API endpoint is configured to allow access from the operator's display terminal.

[0330] Real-time response support phase

[0331] 1. The user makes an inquiry via phone or chat.

[0332] 2. If the device is handling a phone inquiry, it captures the audio in real time and converts it to text using Google Cloud Speech-to-Text.

[0333] 3. The terminal sends the converted text to the server.

[0334] 4. The server inputs the received text into a generative artificial intelligence model and generates a model answer.

[0335] 5. The server uses an emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotions and generate an appropriate model answer.

[0336] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays them on the operator's display.

[0337] 7. The operator will respond to the user based on the displayed information and help them quickly access relevant information.

[0338] Specific example

[0339] Example 1: Telephone inquiry

[0340] 1. The user inquires about the return procedure for the product.

[0341] 2. The device converts the user's voice into text in real time (e.g., "I would like to return this item, how do I do that?").

[0342] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[0343] 4. The server uses an emotion engine to analyze the user's voice and recognizes that the user is dissatisfied.

[0344] 5. The server adjusts the model answer based on the analysis results and generates a more polite instruction message (e.g., "We sincerely apologize for the inconvenience. For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[0345] 6. The server sends this model answer and sentiment data back to the terminal, which then displays it on the operator's display.

[0346] Example 2: Chat support

[0347] 1. The user sends a question via chat about a network connection problem.

[0348] 2. The terminal forwards this text to the server as is.

[0349] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[0350] 4. The server uses an emotion engine to analyze the user's emotions from their text and recognizes that the user is anxious.

[0351] 5. The server adjusts the model answer based on the analysis results and generates an answer that includes calming language (e.g., "First, please stay calm. Try restarting your router. If that doesn't solve the problem, contact your provider to check your internet connection.").

[0352] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays it on the operator's chat screen.

[0353] This will reduce the burden on operators, shorten response times, and improve customer satisfaction.

[0354] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0355] Step 1:

[0356] The server retrieves past customer interaction records (in audio and text format) from the database. Specifically, it uses SQL queries to retrieve customer interaction records from the database and saves them as audio and text files. The input is the customer interaction records from the database, and the output is the audio and text files stored on the server.

[0357] Step 2:

[0358] The server calls speech recognition software (Google Cloud Speech-to-Text) to convert the audio recording into text. The input is an audio file, and the output is the converted text data. The server sends the audio file to the API and receives and temporarily stores the converted text data.

[0359] Step 3:

[0360] The server preprocesses the acquired text data. Preprocessing includes noise reduction, formatting standardization, and spell checking. Specifically, it removes unwanted characters using regular expressions and cleanses the text using Python's "nltk" library. The input is the converted text data, and the output is the preprocessed text data.

[0361] Step 4:

[0362] The server trains a generative artificial intelligence model (such as OpenAI's GPT-3) using preprocessed text data. The input is preprocessed text data, and the output is the trained generative AI model. The server inputs the data into the model and optimizes the model's parameters.

[0363] Step 5:

[0364] The server stores the trained generative AI model and deploys it as an API endpoint. The input is the trained generative AI model, and the output is the API endpoint. This allows the model to be accessed from the operator display terminal.

[0365] Step 6:

[0366] Users contact customer support via phone or chat. In the case of phone inquiries, the device captures the user's voice in real time and converts it to text using speech recognition software such as Google Cloud Speech-to-Text. The input is the user's voice, and the output is the converted text.

[0367] Step 7:

[0368] The terminal sends the converted text to the server. The input is the converted text, and the output is an HTTP request sent to the server. This includes the query content and metadata.

[0369] Step 8:

[0370] The server inputs the received text into a generative artificial intelligence model and generates a model answer based on the inquiry. The input is the received text data, and the output is the model answer. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0371] Step 9:

[0372] The server analyzes the user's emotions using an emotion engine (such as IBM Watson Tone Analyzer). The input is received text or audio data, and the output is the analyzed emotion data. Based on the analysis results, the server adjusts the generated model answer.

[0373] Step 10:

[0374] The server sends the final model answer and sentiment data back to the terminal. The input is the model answer and sentiment data, and the output is an HTTP response to the terminal, which includes the model answer and any associated additional information.

[0375] Step 11:

[0376] The terminal displays the received model answer and sentiment data on the operator's display. The input is an HTTP response from the server, and the output is the displayed model answer and sentiment information. Based on the displayed information, the operator provides an answer to the customer.

[0377] Step 12:

[0378] The terminal automatically displays a related information viewing screen to support operator operations and allow for quick access to relevant information. Input is the inquiry content or model answer, and output is the related information viewing screen. For example, if there is an inquiry about the return procedure, the detailed return procedure page will be automatically displayed.

[0379] Through these steps, the system of the present invention can reduce the burden on operators, shorten response times, and improve customer satisfaction.

[0380] (Application Example 2)

[0381] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0382] Traditional customer service systems have made it difficult to grasp customer emotions in real time, hindering the ability to provide appropriate responses quickly. To improve customer satisfaction, it is necessary to provide timely and appropriate answers to customer inquiries while simultaneously responding in a way that respects the customer's emotions.

[0383] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry voice received in real time into text; means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model; means for displaying the model answer on an operator display; means for analyzing the customer's emotions together with the generated model answer and adjusting the model answer based on the emotion data; and means for displaying the generated model answer and the adjusted model answer on an operator display and providing links to related information. This makes it possible to quickly provide appropriate responses that correspond to the customer's emotions.

[0384] "Past customer service records" refer to data that records the history of customer inquiries and the status of those responses.

[0385] An "operation manual" is a set of instructions that describes how to operate a specific task or system.

[0386] A "generative artificial intelligence model" is an algorithm that uses artificial intelligence technology to learn from data and generate predictions and responses.

[0387] "Real-time" means processing or reacting immediately in accordance with the current timeline.

[0388] "Converting to text" is the process of converting audio data into written information.

[0389] "Chat text" refers to inquiries made using text messages.

[0390] A "model answer" refers to an ideal response or answer that has been generated.

[0391] An "operator display" is a display device used by an operator.

[0392] "Analyzing emotions" is the process of extracting emotional elements from text or audio and interpreting them.

[0393] A "link to related information" is a reference URL that provides access to necessary information sources or additional materials.

[0394] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then combines this with an emotion engine that recognizes the user's emotions, generates a model answer using the generative artificial intelligence model, and finally displays it on an operator display.

[0395] Initial setup and learning phase

[0396] 1. The server retrieves past customer interaction records in audio and text format from the database and converts these records to text. For retrieved audio records, it uses speech recognition software (e.g., SpeechRecognition) to convert them to text.

[0397] 2. The server preprocesses this text data. Preprocessing includes text cleanup (noise removal, formatting standardization, etc.).

[0398] 3. The server trains a generative artificial intelligence model (e.g., the Transformers library) using pre-processed data. During the training process, the model is input with customer interaction records and manual datasets, and natural language processing techniques are applied to optimize the model's parameters.

[0399] 4. The server stores the trained generative artificial intelligence model and deploys it. It configures an API endpoint to make it available from the operator display terminal.

[0400] Real-time response support phase

[0401] 1. The user contacts customer support. In the case of a phone inquiry, the device captures the user's voice in real time. In the case of a chat inquiry, the device receives the text message directly.

[0402] 2. In the case of a phone inquiry, the device uses speech recognition software (e.g., Google Speech-to-Text API) to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[0403] 3. The terminal sends the transcribed query to the server. This transmission is in the form of an HTTP request and includes the query content and metadata (e.g., query type, time, etc.).

[0404] 4. The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0405] 5. The server analyzes the user's emotions using an emotion engine (e.g., an EmotionRecognizer engine). It recognizes the user's emotions from the voice data in the case of voice queries, and from the text data in the case of text queries.

[0406] 6. The server uses the analyzed sentiment data to refine the model response using a generative artificial intelligence model. For example, if the user is angry, it will generate a model response that includes a more courteous response.

[0407] 7. The server sends the generated model answer and sentiment data back to the terminal. The response includes the model answer and any additional relevant information (e.g., links to relevant manual pages).

[0408] 8. The terminal displays the received model answer and sentiment data on the operator display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and relevant documents.

[0409] 9. The operator provides the user with a response, referring to the displayed model answer and sentiment data. The operator provides appropriate instructions to the user, supplementing relevant information as needed.

[0410] 10. The terminal controls the information viewing screen and helps operators quickly access relevant information. For example, if there is an inquiry about the return procedure, it will automatically display the detailed page for the return procedure.

[0411] Specific example

[0412] Example 1: Telephone inquiry

[0413] 1. The user inquires about the return procedure for the product.

[0414] 2. The device converts the user's voice into text in real time (e.g., "I would like to return the item, how do I do that?").

[0415] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[0416] 4. The server uses an emotion engine to analyze the user's voice and recognize that the user is dissatisfied.

[0417] 5. The server adjusts the model answer based on the analysis results and generates a more detailed explanation.

[0418] 6. The server sends this model answer and sentiment data back to the terminal, which then displays it on the operator's display.

[0419] 7. The operator will provide appropriate instructions to the user based on the displayed answer and adjust their response based on the displayed sentiment data.

[0420] Example 2: Chat support

[0421] 1. The user sends a question via chat about a network connection problem.

[0422] 2. The terminal forwards this text to the server as is.

[0423] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[0424] 4. The server uses an emotion engine to analyze the user's emotions from their text and recognizes that the user is anxious.

[0425] 5. The server adjusts the model answer based on the analysis results and generates an answer that includes calming language.

[0426] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays it on the operator's chat screen.

[0427] 7. Operators will respond to users quickly and appropriately, taking into account the displayed information. In doing so, they will strive to reassure users by considering emotional data.

[0428] The system of the present invention can provide a quick and emotionally responsive response to prompts such as "I want to return this item, what should I do?" or "I bought this, but it doesn't work, what should I do?", thereby improving customer satisfaction.

[0429] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0430] Step 1:

[0431] A user contacts customer support. Input is either voice or text. Output is either audio data or a text message.

[0432] Step 2:

[0433] For phone inquiries, the device uses speech recognition software (e.g., Google Speech-to-Text API) to convert the user's voice into text in real time. Input is audio data. Output is text data.

[0434] Step 3:

[0435] The terminal sends the converted text or chat text to the server. This transmission is in the form of an HTTP request and includes the inquiry content and metadata (e.g., inquiry type, time, etc.). The input is text data and metadata. The output is an HTTP request.

[0436] Step 4:

[0437] The server inputs the received text data into a generative artificial intelligence model and generates a model answer based on the query. The input is text data. The output is the model answer.

[0438] Step 5:

[0439] The server analyzes the user's emotions using an emotion engine (e.g., an EmotionRecognizer engine). It recognizes the user's emotions from audio data in the case of voice queries, and from text data in the case of text queries. Input is either text data or audio data. Output is emotion data.

[0440] Step 6:

[0441] The server uses a generative artificial intelligence model to adjust the model answer based on the analyzed emotion data. For example, if the user is angry, it will generate a model answer that includes a more courteous response. The input is emotion data and a model answer. The output is the adjusted model answer.

[0442] Step 7:

[0443] The server sends the generated model answer and sentiment data back to the terminal. The response includes the model answer and related additional information (e.g., links to relevant manual pages). The input is the adjusted model answer and sentiment data. The output is an HTTP response.

[0444] Step 8:

[0445] The terminal displays the received model answer and sentiment data on the operator's display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and related documentation. Input is an HTTP response. Output is the display.

[0446] Step 9:

[0447] The operator provides answers to the user, referring to the displayed model answers and sentiment data. As needed, the operator provides appropriate instructions to the user, supplementing them with relevant information. Input is the information displayed on the screen. Output is the response to the user.

[0448] Step 10:

[0449] The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, it automatically displays the detailed return procedure page. The input is the user's inquiry. The output is an automatically displayed information.

[0450] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0451] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0452] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0453] [Second Embodiment]

[0454] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0455] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0456] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0457] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0458] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0459] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0460] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0461] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0462] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0463] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0464] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0465] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0466] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then uses the generative artificial intelligence model to generate a model answer, and finally displays it on an operator display.

[0467] Initial setup and learning phase

[0468] 1. The server retrieves past customer interaction records in both audio and text format from the database and converts these records to text. In the case of audio records, speech recognition technology is used to convert them to text.

[0469] 2. The server uses the converted text data to train a generative artificial intelligence model (e.g., a natural language processing model). This training process includes data preprocessing (noise removal, formatting standardization, etc.).

[0470] 3. The server deploys the trained generative artificial intelligence model and configures API endpoints so that they can be used on each operator display terminal.

[0471] Real-time response support phase

[0472] 1. The user contacts customer support by phone or chat.

[0473] In the case of phone inquiries, the terminal captures the user's voice in real time.

[0474] In the case of chat support, the device directly receives the text message sent by the user.

[0475] 2. In the case of a phone inquiry, the terminal uses speech recognition software to convert the user's voice into text.

[0476] The converted text is stored in the device's temporary memory.

[0477] 3. The device sends the text converted from the voice or the received chat text to the server.

[0478] This transmission is in the form of an HTTP request and includes the query content and metadata (query type, time, etc.).

[0479] 4. The server inputs the received text into a generative artificial intelligence model and generates a model answer.

[0480] For example, in response to an inquiry such as "Please tell me about returns," a generative artificial intelligence model will generate a model answer such as, "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0481] 5. The server sends the generated model answer back to the terminal.

[0482] This response is also sent in HTTP response format and includes not only the model answer but also related information (for example, a link to the relevant page in the operation manual).

[0483] 6. The terminal displays the model answer received from the server on the operator's display.

[0484] The information displayed includes the answers and related documents the user is looking for.

[0485] 7. The operator will provide appropriate assistance to the user, referring to the provided model answer.

[0486] If necessary, the operator will assist the customer while supplementing the information with relevant details.

[0487] 8. The terminal also controls the information viewing screen, assisting operators in quickly accessing relevant information.

[0488] For example, if an inquiry is received regarding the return process, the detailed return procedure page will be automatically displayed.

[0489] Specific example

[0490] Example 1: Telephone inquiry

[0491] 1. The user inquires about the return procedure for the product.

[0492] 2. The device converts the user's voice into text in real time (e.g., "I would like to return the item, how do I do that?").

[0493] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[0494] 4. The server sends this model answer back to the terminal, which then displays it on the operator's display.

[0495] 5. The operator provides appropriate instructions to the user based on the displayed answer.

[0496] Example 2: Chat support

[0497] 1. The user sends a question via chat about a network connection problem.

[0498] 2. The terminal forwards this text to the server as is.

[0499] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[0500] 4. The server sends the generated model answer back to the terminal, which then displays it on the operator's chat screen.

[0501] 5. The operator replies to the user with the displayed information.

[0502] Thus, the system of the present invention can reduce the burden on operators, shorten response times, and improve customer satisfaction.

[0503] The following describes the processing flow.

[0504] Step 1:

[0505] The server retrieves past customer interaction records in both audio and text format from the database and converts these records to text. For retrieved audio records, speech recognition software is used to convert them to text.

[0506] Step 2:

[0507] The server preprocesses this text data. Preprocessing includes text cleanup (noise removal, formatting standardization, etc.).

[0508] Step 3:

[0509] The server trains a generative artificial intelligence model using pre-processed data. During the training process, customer interaction records and manual datasets are input into the model, and natural language processing techniques are applied to optimize the model's parameters.

[0510] Step 4:

[0511] The server stores the trained generative artificial intelligence model and deploys it. It then configures an API endpoint to make it available from the operator display terminal.

[0512] Step 5:

[0513] A user contacts customer support. For phone inquiries, the device captures the user's voice in real time. For chat inquiries, the device receives text messages directly.

[0514] Step 6:

[0515] When a user makes a voice inquiry, the device uses speech recognition software to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[0516] Step 7:

[0517] The terminal sends the transcribed query content to the server. This transmission is in the form of an HTTP request and includes the query content and metadata (e.g., query type, time, etc.).

[0518] Step 8:

[0519] The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0520] Step 9:

[0521] The server sends the generated model answer back to the terminal in HTTP response format. The response includes the model answer and any related additional information (e.g., links to relevant manual pages).

[0522] Step 10:

[0523] The terminal displays the received model answer on the operator's display. The displayed information includes the appropriate answer to the user's inquiry and related documents.

[0524] Step 11:

[0525] The operator provides the user with an answer, referring to the displayed model answer. If necessary, the operator provides the user with appropriate instructions, supplementing them with relevant information.

[0526] Step 12:

[0527] The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, the terminal automatically displays the detailed return procedure page.

[0528] This series of steps reduces the burden on operators and shortens response times.

[0529] (Example 1)

[0530] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0531] In modern customer support operations, responding quickly and accurately to customer inquiries is crucial. However, customer inquiries cover a wide range of topics, requiring operators to provide appropriate responses each time. This is time-consuming and labor-intensive, and the quality of service can vary from operator to operator. Furthermore, a lack of systems that allow for quick access to information related to inquiries leads to longer response times and a decline in customer satisfaction.

[0532] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0533] In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry audio received in real time into text; means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model; means for automatically displaying an information viewing screen related to the inquiry; means for constructing prompt sentences for generating a model answer using the generative artificial intelligence model; and means for providing additional information related to the model answer (such as a link to the relevant page in the operation manual). This enables operators to respond to customer inquiries quickly and accurately, reduces variability in the quality of responses, and improves customer satisfaction.

[0534] "Past customer interaction records" refer to all communications and interaction history recorded between customers and support operators in the past.

[0535] An "operator's manual" is a document that describes how to use a product or service, troubleshooting procedures, and other related information.

[0536] A "generative artificial intelligence model" is a machine learning model that has the ability to learn from large amounts of text data and perform natural language generation tasks.

[0537] "Inquiry voice messages" refer to audio information transmitted by customers via telephone or voice message.

[0538] "Converting to text" refers to the process of converting audio data into written text.

[0539] "Chat text" refers to the text information that customers send via text chat.

[0540] A "model answer" is the optimal response to a customer inquiry generated by a generative artificial intelligence model.

[0541] An "operator display" refers to a display device or interface used by an operator.

[0542] "Speech recognition technology" is a technology that analyzes speech data and converts it into text data.

[0543] An "information viewing screen" is a display screen used by operators to check relevant information.

[0544] A "prompt statement" is a guide statement that is input to a generative artificial intelligence model.

[0545] This invention relates to a system that trains a generative artificial intelligence model using past customer service records and operation manuals, converts customer inquiries into text in real time, then uses the generative artificial intelligence model to generate model answers, and finally displays them on the operator's display. This system enables operators to provide customer service quickly and effectively.

[0546] First, I will explain the system's initial setup and learning phase.

[0547] The server retrieves past customer interaction records in both audio and text format from a database. Common databases used for this purpose include MySQL and PostgreSQL. Next, the server converts the retrieved audio records into text using speech recognition technologies such as Google Cloud Speech-to-Text or Amazon Transcribe. This conversion process extracts the audio data as text information.

[0548] The server then preprocesses this digitized data. Preprocessing involves removing noise from the data and standardizing its format. The Python pandas library can be used for preprocessing. The preprocessed data is then used to train a generative artificial intelligence model (e.g., OpenAI's GPT-4 or Google's BERT). Deep learning frameworks such as TensorFlow or PyTorch are commonly used for this training process.

[0549] Trained generative artificial intelligence models are deployed to cloud platforms such as AWS EC2 or Google Cloud. After deployment, API endpoints are configured so that they can be used on each operator display terminal. These APIs are often built using web frameworks such as Flask or Django.

[0550] Next, I will explain the real-time response support phase.

[0551] When a user contacts customer support via phone or chat, the device captures the user's voice in real time using a digital microphone in the case of a phone call. In the case of a chat, the device directly receives the text message sent by the user. Slack API and Zendesk Chat are used to receive chat messages.

[0552] The device then uses Google Cloud Speech-to-Text to convert the user's speech to text, in the case of a phone call. The converted text is stored in temporary memory. The device then sends the converted text or chat text to the server. This transmission is in the form of an HTTP POST request and includes metadata such as the content of the inquiry, the type of inquiry, and the time.

[0553] The server inputs the received text into a generative artificial intelligence model. A prompt is used in this process. For example, by inputting a prompt such as "The user asked: Please tell me about the return procedure," the generative AI model generates a model answer. This model answer might be something like, "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0554] The generated model answer is sent back to the terminal in HTTP response format. In addition to the model answer, it also includes relevant information (e.g., a link to the relevant page in the operation manual). The terminal displays this model answer on the operator display. The display includes the answer the user is looking for and related documents. The operator uses the provided model answer as a reference to respond appropriately to the user. If necessary, the operator will provide additional information to assist the customer.

[0555] Furthermore, the terminal also controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry about a return is received, it automatically displays the detailed return procedure page. In this way, the system can reduce the burden on operators, shorten response times, and improve customer satisfaction.

[0556] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0557] Step 1:

[0558] The server retrieves past customer interaction records in audio and text format from a database. MySQL is used as the database for retrieval. The input data consists of customer interaction records, and the output is the retrieved record data.

[0559] Step 2:

[0560] The server converts audio data from acquired customer interaction records into text using Google Cloud Speech-to-Text. The input data is the audio recording, and the output is the text data converted from the audio. Specifically, the audio file is sent to the API, and a reply is received in text format.

[0561] Step 3:

[0562] The server preprocesses the text data. Here, the Python pandas library is used to remove noise and standardize the text format. The input data is the converted text data, and the output is the preprocessed text data.

[0563] Step 4:

[0564] The server trains a generative artificial intelligence model (e.g., GPT-4) using preprocessed text data. TensorFlow is used for this training. The input data is preprocessed text data, and the output is the trained generative artificial intelligence model.

[0565] Step 5:

[0566] The server deploys the trained generative AI model to a cloud platform such as AWS EC2. Furthermore, it configures API endpoints for use on each operator display terminal. The input data is the trained generative AI model, and the output is the deployed model and the available API endpoints.

[0567] Step 6:

[0568] Users contact customer support via phone or chat. For phone inquiries, the device uses a digital microphone to capture the user's voice in real time. Input data is the user's voice inquiry or chat text, and output is the captured audio or directly received chat messages.

[0569] Step 7:

[0570] For voice inquiries, the device uses Google Cloud Speech-to-Text to convert speech to text. The input data is the captured audio, and the output is the converted text.

[0571] Step 8:

[0572] The terminal sends the converted text or received chat text to the server. The transmission is done via an HTTP POST request. The input data is the converted text or chat text, and the output is the text sent to the server.

[0573] Step 9:

[0574] The server inputs the received text into a generative artificial intelligence model. In doing so, it constructs and inputs a prompt. The input data consists of the received text and the prompt, and the output is the generated model answer. A concrete example of a prompt is: "The user asked: Please tell me about the return procedure."

[0575] Step 10:

[0576] The server sends the generated model answer back to the terminal. The return is also done in HTTP response format. The input data is the generated model answer and related information (such as a link to the operation manual), and the output is the model answer sent to the terminal.

[0577] Step 11:

[0578] The terminal displays the model answer received from the server on the operator's display. The input data is the model answer and related documents, and the output is the displayed information.

[0579] Step 12:

[0580] The operator responds to the user while referring to the provided model answer. Additional information is provided as needed to assist the customer. The input data consists of the displayed model answer and related information, while the output is the operator's response action.

[0581] Step 13:

[0582] The terminal automatically controls the information viewing screen so that operators can quickly access relevant information. For example, if there is an inquiry about the return procedure, it will automatically display the page detailing the return procedure. The input data is the inquiry content, and the output is the automatically displayed page of relevant information.

[0583] (Application Example 1)

[0584] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0585] A support system is needed in logistics centers to enable workers to efficiently process customer inquiries and work instructions. Existing systems require workers to manually verify information, which is time-consuming and can lead to misunderstandings and delays in responses. Furthermore, providing accurate instructions to workers in real time is difficult, resulting in insufficient improvements in work efficiency and reduction of errors. To address these issues, a real-time support system utilizing speech recognition technology and generative artificial intelligence is required.

[0586] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0587] In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry voices received in real time into text; means for receiving the converted text or chat text and generating model answers using the generative artificial intelligence model; means for displaying the generated model answers on a visual display device; and means for acquiring voice instructions from workers in the logistics center in real time using a basic speech recognition library and an external API, and displaying the generated model answers on smart glasses. This enables improved work efficiency, reduced errors, and faster customer response in the logistics center.

[0588] "Past customer service records" refers to data that records past customer inquiries and the details of the responses received.

[0589] An "operation manual" refers to a document or guide that describes specific procedures or operating methods.

[0590] A "generative artificial intelligence model" refers to an AI model that performs natural language processing based on a large amount of data to generate model answers or recommended responses.

[0591] "Real-time customer inquiry audio" refers to the immediate reception of audio data sent by customers via telephone or voice chat.

[0592] "Means of converting to text" refers to technologies and devices for converting audio data into text data.

[0593] "Chat text" refers to text data sent in text chat format.

[0594] A "visual display device" refers to a device used to visually display the generated model answer.

[0595] A "speech recognition library" refers to a software component used to convert speech data into text.

[0596] An "external API" refers to an interface used to connect with other systems or services.

[0597] "Smart glasses" refer to glasses-type wearable devices with a built-in display that can show information in real time.

[0598] A "logistics center" refers to a facility that stores, manages, and ships goods.

[0599] A "worker" refers to a person who is responsible for tasks such as receiving, managing, and shipping goods at a logistics center.

[0600] The system that implements this application example includes the following components.

[0601] Components

[0602] 1. Server

[0603] Database: Stores past customer service records and operation manuals.

[0604] Generative artificial intelligence models: Trained using past customer interaction records and operation manuals.

[0605] API endpoint: Provides an interface for accessing trained generative artificial intelligence models.

[0606] 2. Terminal

[0607] Smart glasses: Used by workers in logistics centers to display information in real time.

[0608] Speech recognition library: Converts worker speech into text in real time.

[0609] Data transmission / reception function: Exchanges text and model answers with the server.

[0610] 3. User

[0611] Worker: At the logistics center, they use the system to handle customer inquiries, receive work instructions, and respond to them.

[0612] Explanation of the process

[0613] 1. Preparing the database

[0614] The server retrieves past customer service records and operation manuals from a database and uses this data to train a generative artificial intelligence model.

[0615] 2. Training and Deployment

[0616] The server converts audio recordings into text using speech recognition technology, preprocesses the data using a generative artificial intelligence model, and trains the model. The trained generative AI model becomes accessible through an API endpoint.

[0617] 3. Real-time processing

[0618] Users wear smart glasses within the logistics center and make inquiries and give instructions using voice commands.

[0619] The speech recognition library converts the speech to text, and the device sends the text to the server.

[0620] The server inputs the received text into a generative artificial intelligence model and generates a model answer in real time.

[0621] The model answer is sent to the device and displayed on the smart glasses' screen.

[0622] Hardware and software to be used

[0623] 1. Speech recognition library

[0624] Libraries used: Python's speech_recognition

[0625] Purpose: Converts audio data to text.

[0626] 2. External API

[0627] How to use: Send a query to a generative artificial intelligence model using an HTTP request and obtain a model answer.

[0628] 3. Smart Glasses

[0629] Device used: A wearable device in the form of glasses with a built-in display, worn by workers at a logistics center.

[0630] Specific example

[0631] In a logistics center, when a worker wears smart glasses to retrieve a specific item from a storage shelf, they are asked aloud, "What is the next item?" A speech recognition library captures this audio and converts it to text. The converted text is sent to a server, where a generative artificial intelligence model generates the optimal response. This response is displayed on the smart glasses' screen in the form of, "Please retrieve the item on shelf B5."

[0632] Example of a prompt

[0633] The following is an example of a prompt message that describes the specific operation of this system.

[0634] "Please explain how to create a system that uses speech recognition technology to convert a user's voice into text in real time, inputs that text into a generative AI model to generate a model answer, and displays the result on smart glasses."

[0635] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0636] Step 1:

[0637] Users wear smart glasses and make inquiries and give instructions by voice within the logistics center.

[0638] Input: User's voice (e.g., "What is the next product?")

[0639] Operation:

[0640] The user speaks into the microphone of the smart glasses. The smart glasses capture the audio.

[0641] Step 2:

[0642] The device uses a speech recognition library to convert the user's speech into text.

[0643] Input: User's voice captured in Step 1

[0644] Data processing: A speech recognition library is used to convert audio data into text.

[0645] Output: Text (Example: "What is the next product?")

[0646] Operation:

[0647] The speech recognition library (Python's speech_recognition) converts the speech into text data and stores it in temporary memory.

[0648] Step 3:

[0649] The terminal sends the converted text to the server in the form of an HTTP request.

[0650] Input: Text generated in Step 2

[0651] Data processing: Converting text data into HTTP request format.

[0652] Output: HTTP request sent to the server

[0653] Operation:

[0654] The text data is sent as an HTTP request to the server's API endpoint.

[0655] Step 4:

[0656] The server inputs the received text into a generative artificial intelligence model and generates a model answer.

[0657] Input: Text received as an HTTP request

[0658] Data processing: Generative artificial intelligence models generate the optimal model answer from text.

[0659] Output: Generated model answer (Example: "Please take the item from shelf B5.")

[0660] Operation:

[0661] A generative artificial intelligence model on the server generates a model answer based on the received text.

[0662] Step 5:

[0663] The server sends the generated model answer back to the terminal in HTTP response format.

[0664] Input: Generated model answer

[0665] Data processing: Convert model answers into HTTP response format.

[0666] Output: HTTP response sent to the terminal

[0667] Operation:

[0668] The server sends an HTTP response containing the model answer back to the terminal.

[0669] Step 6:

[0670] The device displays the received model answer on the smart glasses' screen.

[0671] Input: Model answer included in the HTTP response received in Step 5

[0672] Data processing: Convert model answers to a display format.

[0673] Output: Model answer displayed on the smart glasses screen (Example: "Please take the item from shelf B5.")

[0674] Operation:

[0675] The device converts the model answer data into an appropriate display format and displays it on the smart glasses' screen.

[0676] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0677] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then combines this with an emotion engine that recognizes the user's emotions, generates a model answer using the generative artificial intelligence model, and finally displays it on an operator display.

[0678] Initial setup and learning phase

[0679] 1. The server retrieves past customer interaction records in audio and text format from the database and converts these records to text. For retrieved audio records, speech recognition software is used to convert them to text.

[0680] 2. The server preprocesses this text data. Preprocessing includes text cleanup (noise removal, formatting standardization, etc.).

[0681] 3. The server trains a generative artificial intelligence model using pre-processed data. During the training process, the model is fed a dataset of customer interaction records and manuals, and natural language processing techniques are applied to optimize the model's parameters.

[0682] 4. The server saves the trained generative artificial intelligence model and deploys it. It configures an API endpoint to make it available from the operator display terminal.

[0683] Real-time response support phase

[0684] 1. The user contacts customer support. In the case of a phone inquiry, the device captures the user's voice in real time. In the case of a chat inquiry, the device receives the text message directly.

[0685] 2. When the device receives a phone inquiry, it uses speech recognition software to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[0686] 3. The terminal sends the text-based query to the server. This transmission is in the form of an HTTP request and includes the query content and metadata (e.g., query type, time, etc.).

[0687] 4. The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0688] 5. The server uses an emotion engine to analyze the user's emotions. In the case of voice queries, the server recognizes the user's emotions from the voice data; in the case of text queries, it recognizes them from the text data.

[0689] 6. Based on the sentiment data analyzed by the server, the generative artificial intelligence model adjusts the model answer. For example, if the user is angry, it will generate a model answer that includes a more courteous response.

[0690] 7. The server sends the generated model answer and sentiment data back to the terminal. The response includes the model answer and any additional relevant information (e.g., links to relevant manual pages).

[0691] 8. The terminal displays the received model answer and sentiment data on the operator display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and relevant documents.

[0692] 9. The operator provides the user with a response, referring to the displayed model answer and sentiment data. The operator provides appropriate instructions to the user, supplementing relevant information as needed.

[0693] 10. The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, the terminal automatically displays the detailed page for the return procedure.

[0694] Specific example

[0695] Example 1: Telephone inquiry

[0696] 1. The user inquires about the return procedure for the product.

[0697] 2. The device converts the user's voice into text in real time (e.g., "I would like to return the item, how do I do that?").

[0698] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[0699] 4. The server uses an emotion engine to analyze the user's voice and recognize that the user is dissatisfied.

[0700] 5. The server adjusts the model answer based on the analysis results and generates a more detailed explanation.

[0701] 6. The server sends this model answer and sentiment data back to the terminal, which then displays it on the operator's display.

[0702] 7. The operator will provide appropriate instructions to the user based on the displayed answer and adjust their response based on the displayed sentiment data.

[0703] Example 2: Chat support

[0704] 1. The user sends a question via chat about a network connection problem.

[0705] 2. The terminal forwards this text to the server as is.

[0706] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[0707] 4. The server uses an emotion engine to analyze the user's emotions from their text and recognizes that the user is anxious.

[0708] 5. The server adjusts the model answer based on the analysis results and generates an answer that includes calming language.

[0709] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays it on the operator's chat screen.

[0710] 7. Operators will respond to users quickly and appropriately, taking into account the displayed information. In doing so, they will strive to reassure users by considering emotional data.

[0711] Thus, the system of the present invention reduces the burden on operators, shortens response times, improves customer satisfaction, and enables appropriate responses that respond to the user's emotions.

[0712] The following describes the processing flow.

[0713] Step 1:

[0714] The server retrieves past customer interaction records in both audio and text format from the database, and converts the audio records to text using speech recognition software. The retrieved text data is preprocessed to remove noise and standardize the format.

[0715] Step 2:

[0716] The server trains a generative artificial intelligence model using pre-processed data. Natural language processing techniques are used to optimize the model's parameters during training. Once the training is complete, the model is saved, and an API endpoint is configured for use on each operator's display terminal.

[0717] Step 3:

[0718] A user contacts customer support. For phone inquiries, the device captures the user's voice in real time. For chat inquiries, the device receives text messages directly.

[0719] Step 4:

[0720] When a call is made, the device uses speech recognition software to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[0721] Step 5:

[0722] The terminal sends the text-based query content to the server. The transmission is in HTTP request format and includes the query content and metadata (e.g., query type, time, etc.).

[0723] Step 6:

[0724] The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0725] Step 7:

[0726] The server uses an emotion engine to analyze the user's emotions. If audio data is available, it recognizes emotions from the audio; if text data is available, it analyzes emotions from the text. For example, it analyzes tone and speed of voice from audio, and word choice and expression from text.

[0727] Step 8:

[0728] The server adjusts the model response based on the analyzed sentiment data. If the user is clearly dissatisfied, it generates a more polite and helpful message. For example, it might add phrases like, "We apologize for your dissatisfaction with the return process..."

[0729] Step 9:

[0730] The server returns the generated model answer and sentiment data to the terminal in HTTP response format. The response includes the model answer and related additional information (e.g., links to relevant manual pages).

[0731] Step 10:

[0732] The terminal displays the received model answer and sentiment data on the operator's display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and relevant documents.

[0733] Step 11:

[0734] The operator provides answers to the user, referring to the displayed model answers and sentiment data. If necessary, they adjust their response based on the displayed sentiment data and provide appropriate instructions to the user.

[0735] Step 12:

[0736] The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, the system automatically displays the detailed return procedure page. This series of steps reduces the burden on operators and shortens response times. It also enables appropriate responses that are tailored to the user's emotions.

[0737] (Example 2)

[0738] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0739] Traditional customer support systems have several drawbacks, including the significant burden placed on operators when responding to customer inquiries in real time. Furthermore, the difficulty in responding to customer emotions and the resulting decrease in customer satisfaction are concerns. Additionally, the difficulty in quickly accessing relevant information leads to longer response times and inefficient support.

[0740] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry voice received in real time into text; means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model; means for analyzing the user's emotions and adjusting the generated model answer; means for displaying the generated model answer on an operator display; and means for automatically displaying a related information viewing screen so that the operator can quickly access related information. This makes it possible to reduce the burden on operators, shorten response times, and improve customer satisfaction.

[0741] "Past customer interaction records" refer to data that includes details of previous interactions with customers.

[0742] An "operator's manual" is a document that explains how to use a system or product.

[0743] A "generative artificial intelligence model" is a machine learning model that has the ability to generate natural language based on a large amount of data.

[0744] "Converting to text" is the process of converting non-text data, such as audio or images, into written text.

[0745] "Speech recognition technology" is a technology that converts speech data into text in real time.

[0746] A "model answer" is a standard or recommended response provided to a customer inquiry.

[0747] "Emotional analysis" is the process of identifying a customer's emotional state from their voice or text.

[0748] An "operator display" is a display device used by operators to view and manipulate information.

[0749] The "Related Information Viewing Screen" is a screen that displays the necessary information in response to customer inquiries.

[0750] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then combines it with an emotion engine to recognize the user's emotions, generates a model answer using the generative artificial intelligence model, and finally displays it on an operator display.

[0751] System Configuration

[0752] Hardware and software

[0753] The server manages the database, trains and deploys generative artificial intelligence models, and performs speech recognition and sentiment analysis. Software used includes Python, SQL, OpenAI GPT-3, IBM Watson Tone Analyzer, and Google Cloud Speech-to-Text.

[0754] Terminal: Captures user inquiries and exchanges data with the server. The terminal is equipped with voice recognition software, a browser, and an operator display.

[0755] User: Contact us by phone or chat.

[0756] Processing of the invention

[0757] Initial setup and learning phase

[0758] 1. The server retrieves past customer interaction records (in audio and text format) from the database and converts them to text using speech recognition software (Google Cloud Speech-to-Text).

[0759] 2. The server cleans up the acquired text and performs preprocessing to train a generative artificial intelligence model (OpenAI GPT-3).

[0760] 3. The server trains a generative artificial intelligence model using preprocessed data, and saves and deploys the model after training is complete. An API endpoint is configured to allow access from the operator's display terminal.

[0761] Real-time response support phase

[0762] 1. The user makes an inquiry via phone or chat.

[0763] 2. If the device is handling a phone inquiry, it captures the audio in real time and converts it to text using Google Cloud Speech-to-Text.

[0764] 3. The terminal sends the converted text to the server.

[0765] 4. The server inputs the received text into a generative artificial intelligence model and generates a model answer.

[0766] 5. The server uses an emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotions and generate an appropriate model answer.

[0767] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays them on the operator's display.

[0768] 7. The operator will respond to the user based on the displayed information and help them quickly access relevant information.

[0769] Specific example

[0770] Example 1: Telephone inquiry

[0771] 1. The user inquires about the return procedure for the product.

[0772] 2. The device converts the user's voice into text in real time (e.g., "I would like to return this item, how do I do that?").

[0773] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[0774] 4. The server uses an emotion engine to analyze the user's voice and recognizes that the user is dissatisfied.

[0775] 5. The server adjusts the model answer based on the analysis results and generates a more polite instruction message (e.g., "We sincerely apologize for the inconvenience. For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[0776] 6. The server sends this model answer and sentiment data back to the terminal, which then displays it on the operator's display.

[0777] Example 2: Chat support

[0778] 1. The user sends a question via chat about a network connection problem.

[0779] 2. The terminal forwards this text to the server as is.

[0780] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[0781] 4. The server uses an emotion engine to analyze the user's emotions from their text and recognizes that the user is anxious.

[0782] 5. The server adjusts the model answer based on the analysis results and generates an answer that includes calming language (e.g., "First, please stay calm. Try restarting your router. If that doesn't solve the problem, contact your provider to check your internet connection.").

[0783] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays it on the operator's chat screen.

[0784] This will reduce the burden on operators, shorten response times, and improve customer satisfaction.

[0785] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0786] Step 1:

[0787] The server retrieves past customer interaction records (in audio and text format) from the database. Specifically, it uses SQL queries to retrieve customer interaction records from the database and saves them as audio and text files. The input is the customer interaction records from the database, and the output is the audio and text files stored on the server.

[0788] Step 2:

[0789] The server calls speech recognition software (Google Cloud Speech-to-Text) to convert the audio recording into text. The input is an audio file, and the output is the converted text data. The server sends the audio file to the API and receives and temporarily stores the converted text data.

[0790] Step 3:

[0791] The server preprocesses the acquired text data. Preprocessing includes noise reduction, formatting standardization, and spell checking. Specifically, it removes unwanted characters using regular expressions and cleanses the text using Python's "nltk" library. The input is the converted text data, and the output is the preprocessed text data.

[0792] Step 4:

[0793] The server trains a generative artificial intelligence model (such as OpenAI's GPT-3) using preprocessed text data. The input is preprocessed text data, and the output is the trained generative AI model. The server inputs the data into the model and optimizes the model's parameters.

[0794] Step 5:

[0795] The server stores the trained generative AI model and deploys it as an API endpoint. The input is the trained generative AI model, and the output is the API endpoint. This allows the model to be accessed from the operator display terminal.

[0796] Step 6:

[0797] Users contact customer support via phone or chat. In the case of phone inquiries, the device captures the user's voice in real time and converts it to text using speech recognition software such as Google Cloud Speech-to-Text. The input is the user's voice, and the output is the converted text.

[0798] Step 7:

[0799] The terminal sends the converted text to the server. The input is the converted text, and the output is an HTTP request sent to the server. This includes the query content and metadata.

[0800] Step 8:

[0801] The server inputs the received text into a generative artificial intelligence model and generates a model answer based on the inquiry. The input is the received text data, and the output is the model answer. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0802] Step 9:

[0803] The server analyzes the user's emotions using an emotion engine (such as IBM Watson Tone Analyzer). The input is received text or audio data, and the output is the analyzed emotion data. Based on the analysis results, the server adjusts the generated model answer.

[0804] Step 10:

[0805] The server sends the final model answer and sentiment data back to the terminal. The input is the model answer and sentiment data, and the output is an HTTP response to the terminal, which includes the model answer and any associated additional information.

[0806] Step 11:

[0807] The terminal displays the received model answer and sentiment data on the operator's display. The input is an HTTP response from the server, and the output is the displayed model answer and sentiment information. Based on the displayed information, the operator provides an answer to the customer.

[0808] Step 12:

[0809] The terminal automatically displays a related information viewing screen to support operator operations and allow for quick access to relevant information. Input is the inquiry content or model answer, and output is the related information viewing screen. For example, if there is an inquiry about the return procedure, the detailed return procedure page will be automatically displayed.

[0810] Through these steps, the system of the present invention can reduce the burden on operators, shorten response times, and improve customer satisfaction.

[0811] (Application Example 2)

[0812] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0813] Traditional customer service systems have made it difficult to grasp customer emotions in real time, hindering the ability to provide appropriate responses quickly. To improve customer satisfaction, it is necessary to provide timely and appropriate answers to customer inquiries while simultaneously responding in a way that respects the customer's emotions.

[0814] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry voice received in real time into text; means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model; means for displaying the model answer on an operator display; means for analyzing the customer's emotions together with the generated model answer and adjusting the model answer based on the emotion data; and means for displaying the generated model answer and the adjusted model answer on an operator display and providing links to related information. This makes it possible to quickly provide appropriate responses that correspond to the customer's emotions.

[0815] "Past customer service records" refer to data that records the history of customer inquiries and the status of those responses.

[0816] An "operation manual" is a set of instructions that describes how to operate a specific task or system.

[0817] A "generative artificial intelligence model" is an algorithm that uses artificial intelligence technology to learn from data and generate predictions and responses.

[0818] "Real-time" means processing or reacting immediately in accordance with the current timeline.

[0819] "Converting to text" is the process of converting audio data into written information.

[0820] "Chat text" refers to inquiries made using text messages.

[0821] A "model answer" refers to an ideal response or answer that has been generated.

[0822] An "operator display" is a display device used by an operator.

[0823] "Analyzing emotions" is the process of extracting emotional elements from text or audio and interpreting them.

[0824] A "link to related information" is a reference URL that provides access to necessary information sources or additional materials.

[0825] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then combines this with an emotion engine that recognizes the user's emotions, generates a model answer using the generative artificial intelligence model, and finally displays it on an operator display.

[0826] Initial setup and learning phase

[0827] 1. The server retrieves past customer interaction records in audio and text format from the database and converts these records to text. For retrieved audio records, it uses speech recognition software (e.g., SpeechRecognition) to convert them to text.

[0828] 2. The server preprocesses this text data. Preprocessing includes text cleanup (noise removal, formatting standardization, etc.).

[0829] 3. The server trains a generative artificial intelligence model (e.g., the Transformers library) using pre-processed data. During the training process, the model is input with customer interaction records and manual datasets, and natural language processing techniques are applied to optimize the model's parameters.

[0830] 4. The server stores the trained generative artificial intelligence model and deploys it. It configures an API endpoint to make it available from the operator display terminal.

[0831] Real-time response support phase

[0832] 1. The user contacts customer support. In the case of a phone inquiry, the device captures the user's voice in real time. In the case of a chat inquiry, the device receives the text message directly.

[0833] 2. In the case of a phone inquiry, the device uses speech recognition software (e.g., Google Speech-to-Text API) to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[0834] 3. The terminal sends the transcribed query to the server. This transmission is in the form of an HTTP request and includes the query content and metadata (e.g., query type, time, etc.).

[0835] 4. The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0836] 5. The server analyzes the user's emotions using an emotion engine (e.g., an EmotionRecognizer engine). It recognizes the user's emotions from the voice data in the case of voice queries, and from the text data in the case of text queries.

[0837] 6. The server uses the analyzed sentiment data to refine the model response using a generative artificial intelligence model. For example, if the user is angry, it will generate a model response that includes a more courteous response.

[0838] 7. The server sends the generated model answer and sentiment data back to the terminal. The response includes the model answer and any additional relevant information (e.g., links to relevant manual pages).

[0839] 8. The terminal displays the received model answer and sentiment data on the operator display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and relevant documents.

[0840] 9. The operator provides the user with a response, referring to the displayed model answer and sentiment data. The operator provides appropriate instructions to the user, supplementing relevant information as needed.

[0841] 10. The terminal controls the information viewing screen and helps operators quickly access relevant information. For example, if there is an inquiry about the return procedure, it will automatically display the detailed page for the return procedure.

[0842] Specific example

[0843] Example 1: Telephone inquiry

[0844] 1. The user inquires about the return procedure for the product.

[0845] 2. The device converts the user's voice into text in real time (e.g., "I would like to return the item, how do I do that?").

[0846] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[0847] 4. The server uses an emotion engine to analyze the user's voice and recognize that the user is dissatisfied.

[0848] 5. The server adjusts the model answer based on the analysis results and generates a more detailed explanation.

[0849] 6. The server sends this model answer and sentiment data back to the terminal, which then displays it on the operator's display.

[0850] 7. The operator will provide appropriate instructions to the user based on the displayed answer and adjust their response based on the displayed sentiment data.

[0851] Example 2: Chat support

[0852] 1. The user sends a question via chat about a network connection problem.

[0853] 2. The terminal forwards this text to the server as is.

[0854] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[0855] 4. The server uses an emotion engine to analyze the user's emotions from their text and recognizes that the user is anxious.

[0856] 5. The server adjusts the model answer based on the analysis results and generates an answer that includes calming language.

[0857] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays it on the operator's chat screen.

[0858] 7. Operators will respond to users quickly and appropriately, taking into account the displayed information. In doing so, they will strive to reassure users by considering emotional data.

[0859] The system of the present invention can provide a quick and emotionally responsive response to prompts such as "I want to return this item, what should I do?" or "I bought this, but it doesn't work, what should I do?", thereby improving customer satisfaction.

[0860] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0861] Step 1:

[0862] A user contacts customer support. Input is either voice or text. Output is either audio data or a text message.

[0863] Step 2:

[0864] For phone inquiries, the device uses speech recognition software (e.g., Google Speech-to-Text API) to convert the user's voice into text in real time. Input is audio data. Output is text data.

[0865] Step 3:

[0866] The terminal sends the converted text or chat text to the server. This transmission is in the form of an HTTP request and includes the inquiry content and metadata (e.g., inquiry type, time, etc.). The input is text data and metadata. The output is an HTTP request.

[0867] Step 4:

[0868] The server inputs the received text data into a generative artificial intelligence model and generates a model answer based on the query. The input is text data. The output is the model answer.

[0869] Step 5:

[0870] The server analyzes the user's emotions using an emotion engine (e.g., an EmotionRecognizer engine). It recognizes the user's emotions from audio data in the case of voice queries, and from text data in the case of text queries. Input is either text data or audio data. Output is emotion data.

[0871] Step 6:

[0872] The server uses a generative artificial intelligence model to adjust the model answer based on the analyzed emotion data. For example, if the user is angry, it will generate a model answer that includes a more courteous response. The input is emotion data and a model answer. The output is the adjusted model answer.

[0873] Step 7:

[0874] The server sends the generated model answer and sentiment data back to the terminal. The response includes the model answer and related additional information (e.g., links to relevant manual pages). The input is the adjusted model answer and sentiment data. The output is an HTTP response.

[0875] Step 8:

[0876] The terminal displays the received model answer and sentiment data on the operator's display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and related documentation. Input is an HTTP response. Output is the display.

[0877] Step 9:

[0878] The operator provides answers to the user, referring to the displayed model answers and sentiment data. As needed, the operator provides appropriate instructions to the user, supplementing them with relevant information. Input is the information displayed on the screen. Output is the response to the user.

[0879] Step 10:

[0880] The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, it automatically displays the detailed return procedure page. The input is the user's inquiry. The output is an automatically displayed information.

[0881] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0882] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0883] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0884] [Third Embodiment]

[0885] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0886] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0887] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0888] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0889] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0890] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0891] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0892] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0893] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0894] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0895] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0896] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0897] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then uses the generative artificial intelligence model to generate a model answer, and finally displays it on an operator display.

[0898] Initial setup and learning phase

[0899] 1. The server retrieves past customer interaction records in both audio and text format from the database and converts these records to text. In the case of audio records, speech recognition technology is used to convert them to text.

[0900] 2. The server uses the converted text data to train a generative artificial intelligence model (e.g., a natural language processing model). This training process includes data preprocessing (noise removal, formatting standardization, etc.).

[0901] 3. The server deploys the trained generative artificial intelligence model and configures API endpoints so that they can be used on each operator display terminal.

[0902] Real-time response support phase

[0903] 1. The user contacts customer support by phone or chat.

[0904] In the case of phone inquiries, the terminal captures the user's voice in real time.

[0905] In the case of chat support, the device directly receives the text message sent by the user.

[0906] 2. In the case of a phone inquiry, the terminal uses speech recognition software to convert the user's voice into text.

[0907] The converted text is stored in the device's temporary memory.

[0908] 3. The device sends the text converted from the voice or the received chat text to the server.

[0909] This transmission is in the form of an HTTP request and includes the query content and metadata (query type, time, etc.).

[0910] 4. The server inputs the received text into a generative artificial intelligence model and generates a model answer.

[0911] For example, in response to an inquiry such as "Please tell me about returns," a generative artificial intelligence model will generate a model answer such as, "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0912] 5. The server sends the generated model answer back to the terminal.

[0913] This response is also sent in HTTP response format and includes not only the model answer but also related information (for example, a link to the relevant page in the operation manual).

[0914] 6. The terminal displays the model answer received from the server on the operator's display.

[0915] The information displayed includes the answers and related documents the user is looking for.

[0916] 7. The operator will provide appropriate assistance to the user, referring to the provided model answer.

[0917] If necessary, the operator will assist the customer while supplementing the information with relevant details.

[0918] 8. The terminal also controls the information viewing screen, assisting operators in quickly accessing relevant information.

[0919] For example, if an inquiry is received regarding the return process, the detailed return procedure page will be automatically displayed.

[0920] Specific example

[0921] Example 1: Telephone inquiry

[0922] 1. The user inquires about the return procedure for the product.

[0923] 2. The device converts the user's voice into text in real time (e.g., "I would like to return the item, how do I do that?").

[0924] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[0925] 4. The server sends this model answer back to the terminal, which then displays it on the operator's display.

[0926] 5. The operator provides appropriate instructions to the user based on the displayed answer.

[0927] Example 2: Chat support

[0928] 1. The user sends a question via chat about a network connection problem.

[0929] 2. The terminal forwards this text to the server as is.

[0930] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[0931] 4. The server sends the generated model answer back to the terminal, which then displays it on the operator's chat screen.

[0932] 5. The operator replies to the user with the displayed information.

[0933] Thus, the system of the present invention can reduce the burden on operators, shorten response times, and improve customer satisfaction.

[0934] The following describes the processing flow.

[0935] Step 1:

[0936] The server retrieves past customer interaction records in both audio and text format from the database and converts these records to text. For retrieved audio records, speech recognition software is used to convert them to text.

[0937] Step 2:

[0938] The server preprocesses this text data. Preprocessing includes text cleanup (noise removal, formatting standardization, etc.).

[0939] Step 3:

[0940] The server trains a generative artificial intelligence model using pre-processed data. During the training process, customer interaction records and manual datasets are input into the model, and natural language processing techniques are applied to optimize the model's parameters.

[0941] Step 4:

[0942] The server stores the trained generative artificial intelligence model and deploys it. It then configures an API endpoint to make it available from the operator display terminal.

[0943] Step 5:

[0944] A user contacts customer support. For phone inquiries, the device captures the user's voice in real time. For chat inquiries, the device receives text messages directly.

[0945] Step 6:

[0946] When a user makes a voice inquiry, the device uses speech recognition software to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[0947] Step 7:

[0948] The terminal sends the transcribed query content to the server. This transmission is in the form of an HTTP request and includes the query content and metadata (e.g., query type, time, etc.).

[0949] Step 8:

[0950] The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0951] Step 9:

[0952] The server sends the generated model answer back to the terminal in HTTP response format. The response includes the model answer and any related additional information (e.g., links to relevant manual pages).

[0953] Step 10:

[0954] The terminal displays the received model answer on the operator's display. The displayed information includes the appropriate answer to the user's inquiry and related documents.

[0955] Step 11:

[0956] The operator provides the user with an answer, referring to the displayed model answer. If necessary, the operator provides the user with appropriate instructions, supplementing them with relevant information.

[0957] Step 12:

[0958] The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, the terminal automatically displays the detailed return procedure page.

[0959] This series of steps reduces the burden on operators and shortens response times.

[0960] (Example 1)

[0961] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0962] In modern customer support operations, responding quickly and accurately to customer inquiries is crucial. However, customer inquiries cover a wide range of topics, requiring operators to provide appropriate responses each time. This is time-consuming and labor-intensive, and the quality of service can vary from operator to operator. Furthermore, a lack of systems that allow for quick access to information related to inquiries leads to longer response times and a decline in customer satisfaction.

[0963] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0964] In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry audio received in real time into text; means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model; means for automatically displaying an information viewing screen related to the inquiry; means for constructing prompt sentences for generating a model answer using the generative artificial intelligence model; and means for providing additional information related to the model answer (such as a link to the relevant page in the operation manual). This enables operators to respond to customer inquiries quickly and accurately, reduces variability in the quality of responses, and improves customer satisfaction.

[0965] "Past customer interaction records" refer to all communications and interaction history recorded between customers and support operators in the past.

[0966] An "operator's manual" is a document that describes how to use a product or service, troubleshooting procedures, and other related information.

[0967] A "generative artificial intelligence model" is a machine learning model that has the ability to learn from large amounts of text data and perform natural language generation tasks.

[0968] "Inquiry voice messages" refer to audio information transmitted by customers via telephone or voice message.

[0969] "Converting to text" refers to the process of converting audio data into written text.

[0970] "Chat text" refers to the text information that customers send via text chat.

[0971] A "model answer" is the optimal response to a customer inquiry generated by a generative artificial intelligence model.

[0972] An "operator display" refers to a display device or interface used by an operator.

[0973] "Speech recognition technology" is a technology that analyzes speech data and converts it into text data.

[0974] An "information viewing screen" is a display screen used by operators to check relevant information.

[0975] A "prompt statement" is a guide statement that is input to a generative artificial intelligence model.

[0976] This invention relates to a system that trains a generative artificial intelligence model using past customer service records and operation manuals, converts customer inquiries into text in real time, then uses the generative artificial intelligence model to generate model answers, and finally displays them on the operator's display. This system enables operators to provide customer service quickly and effectively.

[0977] First, I will explain the system's initial setup and learning phase.

[0978] The server retrieves past customer interaction records in both audio and text format from a database. Common databases used for this purpose include MySQL and PostgreSQL. Next, the server converts the retrieved audio records into text using speech recognition technologies such as Google Cloud Speech-to-Text or Amazon Transcribe. This conversion process extracts the audio data as text information.

[0979] The server then preprocesses this digitized data. Preprocessing involves removing noise from the data and standardizing its format. The Python pandas library can be used for preprocessing. The preprocessed data is then used to train a generative artificial intelligence model (e.g., OpenAI's GPT-4 or Google's BERT). Deep learning frameworks such as TensorFlow or PyTorch are commonly used for this training process.

[0980] Trained generative artificial intelligence models are deployed to cloud platforms such as AWS EC2 or Google Cloud. After deployment, API endpoints are configured so that they can be used on each operator display terminal. These APIs are often built using web frameworks such as Flask or Django.

[0981] Next, I will explain the real-time response support phase.

[0982] When a user contacts customer support via phone or chat, the device captures the user's voice in real time using a digital microphone in the case of a phone call. In the case of a chat, the device directly receives the text message sent by the user. Slack API and Zendesk Chat are used to receive chat messages.

[0983] The device then uses Google Cloud Speech-to-Text to convert the user's speech to text, in the case of a phone call. The converted text is stored in temporary memory. The device then sends the converted text or chat text to the server. This transmission is in the form of an HTTP POST request and includes metadata such as the content of the inquiry, the type of inquiry, and the time.

[0984] The server inputs the received text into a generative artificial intelligence model. A prompt is used in this process. For example, by inputting a prompt such as "The user asked: Please tell me about the return procedure," the generative AI model generates a model answer. This model answer might be something like, "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[0985] The generated model answer is sent back to the terminal in HTTP response format. In addition to the model answer, it also includes relevant information (e.g., a link to the relevant page in the operation manual). The terminal displays this model answer on the operator display. The display includes the answer the user is looking for and related documents. The operator uses the provided model answer as a reference to respond appropriately to the user. If necessary, the operator will provide additional information to assist the customer.

[0986] Furthermore, the terminal also controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry about a return is received, it automatically displays the detailed return procedure page. In this way, the system can reduce the burden on operators, shorten response times, and improve customer satisfaction.

[0987] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0988] Step 1:

[0989] The server retrieves past customer interaction records in audio and text format from a database. MySQL is used as the database for retrieval. The input data consists of customer interaction records, and the output is the retrieved record data.

[0990] Step 2:

[0991] The server converts audio data from acquired customer interaction records into text using Google Cloud Speech-to-Text. The input data is the audio recording, and the output is the text data converted from the audio. Specifically, the audio file is sent to the API, and a reply is received in text format.

[0992] Step 3:

[0993] The server preprocesses the text data. Here, the Python pandas library is used to remove noise and standardize the text format. The input data is the converted text data, and the output is the preprocessed text data.

[0994] Step 4:

[0995] The server trains a generative artificial intelligence model (e.g., GPT-4) using preprocessed text data. TensorFlow is used for this training. The input data is preprocessed text data, and the output is the trained generative artificial intelligence model.

[0996] Step 5:

[0997] The server deploys the trained generative AI model to a cloud platform such as AWS EC2. Furthermore, it configures API endpoints for use on each operator display terminal. The input data is the trained generative AI model, and the output is the deployed model and the available API endpoints.

[0998] Step 6:

[0999] Users contact customer support via phone or chat. For phone inquiries, the device uses a digital microphone to capture the user's voice in real time. Input data is the user's voice inquiry or chat text, and output is the captured audio or directly received chat messages.

[1000] Step 7:

[1001] For voice inquiries, the device uses Google Cloud Speech-to-Text to convert speech to text. The input data is the captured audio, and the output is the converted text.

[1002] Step 8:

[1003] The terminal sends the converted text or received chat text to the server. The transmission is done via an HTTP POST request. The input data is the converted text or chat text, and the output is the text sent to the server.

[1004] Step 9:

[1005] The server inputs the received text into a generative artificial intelligence model. In doing so, it constructs and inputs a prompt. The input data consists of the received text and the prompt, and the output is the generated model answer. A concrete example of a prompt is: "The user asked: Please tell me about the return procedure."

[1006] Step 10:

[1007] The server sends the generated model answer back to the terminal. The return is also done in HTTP response format. The input data is the generated model answer and related information (such as a link to the operation manual), and the output is the model answer sent to the terminal.

[1008] Step 11:

[1009] The terminal displays the model answer received from the server on the operator's display. The input data is the model answer and related documents, and the output is the displayed information.

[1010] Step 12:

[1011] The operator responds to the user while referring to the provided model answer. Additional information is provided as needed to assist the customer. The input data consists of the displayed model answer and related information, while the output is the operator's response action.

[1012] Step 13:

[1013] The terminal automatically controls the information viewing screen so that operators can quickly access relevant information. For example, if there is an inquiry about the return procedure, it will automatically display the page detailing the return procedure. The input data is the inquiry content, and the output is the automatically displayed page of relevant information.

[1014] (Application Example 1)

[1015] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1016] A support system is needed in logistics centers to enable workers to efficiently process customer inquiries and work instructions. Existing systems require workers to manually verify information, which is time-consuming and can lead to misunderstandings and delays in responses. Furthermore, providing accurate instructions to workers in real time is difficult, resulting in insufficient improvements in work efficiency and reduction of errors. To address these issues, a real-time support system utilizing speech recognition technology and generative artificial intelligence is required.

[1017] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1018] In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry voices received in real time into text; means for receiving the converted text or chat text and generating model answers using the generative artificial intelligence model; means for displaying the generated model answers on a visual display device; and means for acquiring voice instructions from workers in the logistics center in real time using a basic speech recognition library and an external API, and displaying the generated model answers on smart glasses. This enables improved work efficiency, reduced errors, and faster customer response in the logistics center.

[1019] "Past customer service records" refers to data that records past customer inquiries and the details of the responses received.

[1020] An "operation manual" refers to a document or guide that describes specific procedures or operating methods.

[1021] A "generative artificial intelligence model" refers to an AI model that performs natural language processing based on a large amount of data to generate model answers or recommended responses.

[1022] "Real-time customer inquiry audio" refers to the immediate reception of audio data sent by customers via telephone or voice chat.

[1023] "Means of converting to text" refers to technologies and devices for converting audio data into text data.

[1024] "Chat text" refers to text data sent in text chat format.

[1025] A "visual display device" refers to a device used to visually display the generated model answer.

[1026] A "speech recognition library" refers to a software component used to convert speech data into text.

[1027] An "external API" refers to an interface used to connect with other systems or services.

[1028] "Smart glasses" refer to glasses-type wearable devices with a built-in display that can show information in real time.

[1029] A "logistics center" refers to a facility that stores, manages, and ships goods.

[1030] A "worker" refers to a person who is responsible for tasks such as receiving, managing, and shipping goods at a logistics center.

[1031] The system that implements this application example includes the following components.

[1032] Components

[1033] 1. Server

[1034] Database: Stores past customer service records and operation manuals.

[1035] Generative artificial intelligence models: Trained using past customer interaction records and operation manuals.

[1036] API endpoint: Provides an interface for accessing trained generative artificial intelligence models.

[1037] 2. Terminal

[1038] Smart glasses: Used by workers in logistics centers to display information in real time.

[1039] Speech recognition library: Converts worker speech into text in real time.

[1040] Data transmission / reception function: Exchanges text and model answers with the server.

[1041] 3. User

[1042] Worker: At the logistics center, they use the system to handle customer inquiries, receive work instructions, and respond to them.

[1043] Explanation of the process

[1044] 1. Preparing the database

[1045] The server retrieves past customer service records and operation manuals from a database and uses this data to train a generative artificial intelligence model.

[1046] 2. Training and Deployment

[1047] The server converts audio recordings into text using speech recognition technology, preprocesses the data using a generative artificial intelligence model, and trains the model. The trained generative AI model becomes accessible through an API endpoint.

[1048] 3. Real-time processing

[1049] Users wear smart glasses within the logistics center and make inquiries and give instructions using voice commands.

[1050] The speech recognition library converts the speech to text, and the device sends the text to the server.

[1051] The server inputs the received text into a generative artificial intelligence model and generates a model answer in real time.

[1052] The model answer is sent to the device and displayed on the smart glasses' screen.

[1053] Hardware and software to be used

[1054] 1. Speech recognition library

[1055] Libraries used: Python's speech_recognition

[1056] Purpose: Converts audio data to text.

[1057] 2. External API

[1058] How to use: Send a query to a generative artificial intelligence model using an HTTP request and obtain a model answer.

[1059] 3. Smart Glasses

[1060] Device used: A wearable device in the form of glasses with a built-in display, worn by workers at a logistics center.

[1061] Specific example

[1062] In a logistics center, when a worker wears smart glasses to retrieve a specific item from a storage shelf, they are asked aloud, "What is the next item?" A speech recognition library captures this audio and converts it to text. The converted text is sent to a server, where a generative artificial intelligence model generates the optimal response. This response is displayed on the smart glasses' screen in the form of, "Please retrieve the item on shelf B5."

[1063] Example of a prompt

[1064] The following is an example of a prompt message that describes the specific operation of this system.

[1065] "Please explain how to create a system that uses speech recognition technology to convert a user's voice into text in real time, inputs that text into a generative AI model to generate a model answer, and displays the result on smart glasses."

[1066] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1067] Step 1:

[1068] Users wear smart glasses and make inquiries and give instructions by voice within the logistics center.

[1069] Input: User's voice (e.g., "What is the next product?")

[1070] Operation:

[1071] The user speaks into the microphone of the smart glasses. The smart glasses capture the audio.

[1072] Step 2:

[1073] The device uses a speech recognition library to convert the user's speech into text.

[1074] Input: User's voice captured in Step 1

[1075] Data processing: A speech recognition library is used to convert audio data into text.

[1076] Output: Text (Example: "What is the next product?")

[1077] Operation:

[1078] The speech recognition library (Python's speech_recognition) converts the speech into text data and stores it in temporary memory.

[1079] Step 3:

[1080] The terminal sends the converted text to the server in the form of an HTTP request.

[1081] Input: Text generated in Step 2

[1082] Data processing: Converting text data into HTTP request format.

[1083] Output: HTTP request sent to the server

[1084] Operation:

[1085] The text data is sent as an HTTP request to the server's API endpoint.

[1086] Step 4:

[1087] The server inputs the received text into a generative artificial intelligence model and generates a model answer.

[1088] Input: Text received as an HTTP request

[1089] Data processing: Generative artificial intelligence models generate the optimal model answer from text.

[1090] Output: Generated model answer (Example: "Please take the item from shelf B5.")

[1091] Operation:

[1092] A generative artificial intelligence model on the server generates a model answer based on the received text.

[1093] Step 5:

[1094] The server sends the generated model answer back to the terminal in HTTP response format.

[1095] Input: Generated model answer

[1096] Data processing: Convert model answers into HTTP response format.

[1097] Output: HTTP response sent to the terminal

[1098] Operation:

[1099] The server sends an HTTP response containing the model answer back to the terminal.

[1100] Step 6:

[1101] The device displays the received model answer on the smart glasses' screen.

[1102] Input: Model answer included in the HTTP response received in Step 5

[1103] Data processing: Convert model answers to a display format.

[1104] Output: Model answer displayed on the smart glasses screen (Example: "Please take the item from shelf B5.")

[1105] Operation:

[1106] The device converts the model answer data into an appropriate display format and displays it on the smart glasses' screen.

[1107] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1108] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then combines this with an emotion engine that recognizes the user's emotions, generates a model answer using the generative artificial intelligence model, and finally displays it on an operator display.

[1109] Initial setup and learning phase

[1110] 1. The server retrieves past customer interaction records in audio and text format from the database and converts these records to text. For retrieved audio records, speech recognition software is used to convert them to text.

[1111] 2. The server preprocesses this text data. Preprocessing includes text cleanup (noise removal, formatting standardization, etc.).

[1112] 3. The server trains a generative artificial intelligence model using pre-processed data. During the training process, the model is fed a dataset of customer interaction records and manuals, and natural language processing techniques are applied to optimize the model's parameters.

[1113] 4. The server saves the trained generative artificial intelligence model and deploys it. It configures an API endpoint to make it available from the operator display terminal.

[1114] Real-time response support phase

[1115] 1. The user contacts customer support. In the case of a phone inquiry, the device captures the user's voice in real time. In the case of a chat inquiry, the device receives the text message directly.

[1116] 2. When the device receives a phone inquiry, it uses speech recognition software to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[1117] 3. The terminal sends the text-based query to the server. This transmission is in the form of an HTTP request and includes the query content and metadata (e.g., query type, time, etc.).

[1118] 4. The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[1119] 5. The server uses an emotion engine to analyze the user's emotions. In the case of voice queries, the server recognizes the user's emotions from the voice data; in the case of text queries, it recognizes them from the text data.

[1120] 6. Based on the sentiment data analyzed by the server, the generative artificial intelligence model adjusts the model answer. For example, if the user is angry, it will generate a model answer that includes a more courteous response.

[1121] 7. The server sends the generated model answer and sentiment data back to the terminal. The response includes the model answer and any additional relevant information (e.g., links to relevant manual pages).

[1122] 8. The terminal displays the received model answer and sentiment data on the operator display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and relevant documents.

[1123] 9. The operator provides the user with a response, referring to the displayed model answer and sentiment data. The operator provides appropriate instructions to the user, supplementing relevant information as needed.

[1124] 10. The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, the terminal automatically displays the detailed page for the return procedure.

[1125] Specific example

[1126] Example 1: Telephone inquiry

[1127] 1. The user inquires about the return procedure for the product.

[1128] 2. The device converts the user's voice into text in real time (e.g., "I would like to return the item, how do I do that?").

[1129] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[1130] 4. The server uses an emotion engine to analyze the user's voice and recognize that the user is dissatisfied.

[1131] 5. The server adjusts the model answer based on the analysis results and generates a more detailed explanation.

[1132] 6. The server sends this model answer and sentiment data back to the terminal, which then displays it on the operator's display.

[1133] 7. The operator will provide appropriate instructions to the user based on the displayed answer and adjust their response based on the displayed sentiment data.

[1134] Example 2: Chat support

[1135] 1. The user sends a question via chat about a network connection problem.

[1136] 2. The terminal forwards this text to the server as is.

[1137] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[1138] 4. The server uses an emotion engine to analyze the user's emotions from their text and recognizes that the user is anxious.

[1139] 5. The server adjusts the model answer based on the analysis results and generates an answer that includes calming language.

[1140] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays it on the operator's chat screen.

[1141] 7. Operators will respond to users quickly and appropriately, taking into account the displayed information. In doing so, they will strive to reassure users by considering emotional data.

[1142] Thus, the system of the present invention reduces the burden on operators, shortens response times, improves customer satisfaction, and enables appropriate responses that respond to the user's emotions.

[1143] The following describes the processing flow.

[1144] Step 1:

[1145] The server retrieves past customer interaction records in both audio and text format from the database, and converts the audio records to text using speech recognition software. The retrieved text data is preprocessed to remove noise and standardize the format.

[1146] Step 2:

[1147] The server trains a generative artificial intelligence model using pre-processed data. Natural language processing techniques are used to optimize the model's parameters during training. Once the training is complete, the model is saved, and an API endpoint is configured for use on each operator's display terminal.

[1148] Step 3:

[1149] A user contacts customer support. For phone inquiries, the device captures the user's voice in real time. For chat inquiries, the device receives text messages directly.

[1150] Step 4:

[1151] When a call is made, the device uses speech recognition software to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[1152] Step 5:

[1153] The terminal sends the text-based query content to the server. The transmission is in HTTP request format and includes the query content and metadata (e.g., query type, time, etc.).

[1154] Step 6:

[1155] The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[1156] Step 7:

[1157] The server uses an emotion engine to analyze the user's emotions. If audio data is available, it recognizes emotions from the audio; if text data is available, it analyzes emotions from the text. For example, it analyzes tone and speed of voice from audio, and word choice and expression from text.

[1158] Step 8:

[1159] The server adjusts the model response based on the analyzed sentiment data. If the user is clearly dissatisfied, it generates a more polite and helpful message. For example, it might add phrases like, "We apologize for your dissatisfaction with the return process..."

[1160] Step 9:

[1161] The server returns the generated model answer and sentiment data to the terminal in HTTP response format. The response includes the model answer and related additional information (e.g., links to relevant manual pages).

[1162] Step 10:

[1163] The terminal displays the received model answer and sentiment data on the operator's display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and relevant documents.

[1164] Step 11:

[1165] The operator provides answers to the user, referring to the displayed model answers and sentiment data. If necessary, they adjust their response based on the displayed sentiment data and provide appropriate instructions to the user.

[1166] Step 12:

[1167] The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, the system automatically displays the detailed return procedure page. This series of steps reduces the burden on operators and shortens response times. It also enables appropriate responses that are tailored to the user's emotions.

[1168] (Example 2)

[1169] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1170] Traditional customer support systems have several drawbacks, including the significant burden placed on operators when responding to customer inquiries in real time. Furthermore, the difficulty in responding to customer emotions and the resulting decrease in customer satisfaction are concerns. Additionally, the difficulty in quickly accessing relevant information leads to longer response times and inefficient support.

[1171] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry voice received in real time into text; means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model; means for analyzing the user's emotions and adjusting the generated model answer; means for displaying the generated model answer on an operator display; and means for automatically displaying a related information viewing screen so that the operator can quickly access related information. This makes it possible to reduce the burden on operators, shorten response times, and improve customer satisfaction.

[1172] "Past customer interaction records" refer to data that includes details of previous interactions with customers.

[1173] An "operator's manual" is a document that explains how to use a system or product.

[1174] A "generative artificial intelligence model" is a machine learning model that has the ability to generate natural language based on a large amount of data.

[1175] "Converting to text" is the process of converting non-text data, such as audio or images, into written text.

[1176] "Speech recognition technology" is a technology that converts speech data into text in real time.

[1177] A "model answer" is a standard or recommended response provided to a customer inquiry.

[1178] "Emotional analysis" is the process of identifying a customer's emotional state from their voice or text.

[1179] An "operator display" is a display device used by operators to view and manipulate information.

[1180] The "Related Information Viewing Screen" is a screen that displays the necessary information in response to customer inquiries.

[1181] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then combines it with an emotion engine to recognize the user's emotions, generates a model answer using the generative artificial intelligence model, and finally displays it on an operator display.

[1182] System Configuration

[1183] Hardware and software

[1184] The server manages the database, trains and deploys generative artificial intelligence models, and performs speech recognition and sentiment analysis. Software used includes Python, SQL, OpenAI GPT-3, IBM Watson Tone Analyzer, and Google Cloud Speech-to-Text.

[1185] Terminal: Captures user inquiries and exchanges data with the server. The terminal is equipped with voice recognition software, a browser, and an operator display.

[1186] User: Contact us by phone or chat.

[1187] Processing of the invention

[1188] Initial setup and learning phase

[1189] 1. The server retrieves past customer interaction records (in audio and text format) from the database and converts them to text using speech recognition software (Google Cloud Speech-to-Text).

[1190] 2. The server cleans up the acquired text and performs preprocessing to train a generative artificial intelligence model (OpenAI GPT-3).

[1191] 3. The server trains a generative artificial intelligence model using preprocessed data, and saves and deploys the model after training is complete. An API endpoint is configured to allow access from the operator's display terminal.

[1192] Real-time response support phase

[1193] 1. The user makes an inquiry via phone or chat.

[1194] 2. If the device is handling a phone inquiry, it captures the audio in real time and converts it to text using Google Cloud Speech-to-Text.

[1195] 3. The terminal sends the converted text to the server.

[1196] 4. The server inputs the received text into a generative artificial intelligence model and generates a model answer.

[1197] 5. The server uses an emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotions and generate an appropriate model answer.

[1198] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays them on the operator's display.

[1199] 7. The operator will respond to the user based on the displayed information and help them quickly access relevant information.

[1200] Specific example

[1201] Example 1: Telephone inquiry

[1202] 1. The user inquires about the return procedure for the product.

[1203] 2. The device converts the user's voice into text in real time (e.g., "I would like to return this item, how do I do that?").

[1204] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[1205] 4. The server uses an emotion engine to analyze the user's voice and recognizes that the user is dissatisfied.

[1206] 5. The server adjusts the model answer based on the analysis results and generates a more polite instruction message (e.g., "We sincerely apologize for the inconvenience. For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[1207] 6. The server sends this model answer and sentiment data back to the terminal, which then displays it on the operator's display.

[1208] Example 2: Chat support

[1209] 1. The user sends a question via chat about a network connection problem.

[1210] 2. The terminal forwards this text to the server as is.

[1211] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[1212] 4. The server uses an emotion engine to analyze the user's emotions from their text and recognizes that the user is anxious.

[1213] 5. The server adjusts the model answer based on the analysis results and generates an answer that includes calming language (e.g., "First, please stay calm. Try restarting your router. If that doesn't solve the problem, contact your provider to check your internet connection.").

[1214] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays it on the operator's chat screen.

[1215] This will reduce the burden on operators, shorten response times, and improve customer satisfaction.

[1216] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1217] Step 1:

[1218] The server retrieves past customer interaction records (in audio and text format) from the database. Specifically, it uses SQL queries to retrieve customer interaction records from the database and saves them as audio and text files. The input is the customer interaction records from the database, and the output is the audio and text files stored on the server.

[1219] Step 2:

[1220] The server calls speech recognition software (Google Cloud Speech-to-Text) to convert the audio recording into text. The input is an audio file, and the output is the converted text data. The server sends the audio file to the API and receives and temporarily stores the converted text data.

[1221] Step 3:

[1222] The server preprocesses the acquired text data. Preprocessing includes noise reduction, formatting standardization, and spell checking. Specifically, it removes unwanted characters using regular expressions and cleanses the text using Python's "nltk" library. The input is the converted text data, and the output is the preprocessed text data.

[1223] Step 4:

[1224] The server trains a generative artificial intelligence model (such as OpenAI's GPT-3) using preprocessed text data. The input is preprocessed text data, and the output is the trained generative AI model. The server inputs the data into the model and optimizes the model's parameters.

[1225] Step 5:

[1226] The server stores the trained generative AI model and deploys it as an API endpoint. The input is the trained generative AI model, and the output is the API endpoint. This allows the model to be accessed from the operator display terminal.

[1227] Step 6:

[1228] Users contact customer support via phone or chat. In the case of phone inquiries, the device captures the user's voice in real time and converts it to text using speech recognition software such as Google Cloud Speech-to-Text. The input is the user's voice, and the output is the converted text.

[1229] Step 7:

[1230] The terminal sends the converted text to the server. The input is the converted text, and the output is an HTTP request sent to the server. This includes the query content and metadata.

[1231] Step 8:

[1232] The server inputs the received text into a generative artificial intelligence model and generates a model answer based on the inquiry. The input is the received text data, and the output is the model answer. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[1233] Step 9:

[1234] The server analyzes the user's emotions using an emotion engine (such as IBM Watson Tone Analyzer). The input is received text or audio data, and the output is the analyzed emotion data. Based on the analysis results, the server adjusts the generated model answer.

[1235] Step 10:

[1236] The server sends the final model answer and sentiment data back to the terminal. The input is the model answer and sentiment data, and the output is an HTTP response to the terminal, which includes the model answer and any associated additional information.

[1237] Step 11:

[1238] The terminal displays the received model answer and sentiment data on the operator's display. The input is an HTTP response from the server, and the output is the displayed model answer and sentiment information. Based on the displayed information, the operator provides an answer to the customer.

[1239] Step 12:

[1240] The terminal automatically displays a related information viewing screen to support operator operations and allow for quick access to relevant information. Input is the inquiry content or model answer, and output is the related information viewing screen. For example, if there is an inquiry about the return procedure, the detailed return procedure page will be automatically displayed.

[1241] Through these steps, the system of the present invention can reduce the burden on operators, shorten response times, and improve customer satisfaction.

[1242] (Application Example 2)

[1243] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1244] Traditional customer service systems have made it difficult to grasp customer emotions in real time, hindering the ability to provide appropriate responses quickly. To improve customer satisfaction, it is necessary to provide timely and appropriate answers to customer inquiries while simultaneously responding in a way that respects the customer's emotions.

[1245] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry voice received in real time into text; means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model; means for displaying the model answer on an operator display; means for analyzing the customer's emotions together with the generated model answer and adjusting the model answer based on the emotion data; and means for displaying the generated model answer and the adjusted model answer on an operator display and providing links to related information. This makes it possible to quickly provide appropriate responses that correspond to the customer's emotions.

[1246] "Past customer service records" refer to data that records the history of customer inquiries and the status of those responses.

[1247] An "operation manual" is a set of instructions that describes how to operate a specific task or system.

[1248] A "generative artificial intelligence model" is an algorithm that uses artificial intelligence technology to learn from data and generate predictions and responses.

[1249] "Real-time" means processing or reacting immediately in accordance with the current timeline.

[1250] "Converting to text" is the process of converting audio data into written information.

[1251] "Chat text" refers to inquiries made using text messages.

[1252] A "model answer" refers to an ideal response or answer that has been generated.

[1253] An "operator display" is a display device used by an operator.

[1254] "Analyzing emotions" is the process of extracting emotional elements from text or audio and interpreting them.

[1255] A "link to related information" is a reference URL that provides access to necessary information sources or additional materials.

[1256] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then combines this with an emotion engine that recognizes the user's emotions, generates a model answer using the generative artificial intelligence model, and finally displays it on an operator display.

[1257] Initial setup and learning phase

[1258] 1. The server retrieves past customer interaction records in audio and text format from the database and converts these records to text. For retrieved audio records, it uses speech recognition software (e.g., SpeechRecognition) to convert them to text.

[1259] 2. The server preprocesses this text data. Preprocessing includes text cleanup (noise removal, formatting standardization, etc.).

[1260] 3. The server trains a generative artificial intelligence model (e.g., the Transformers library) using pre-processed data. During the training process, the model is input with customer interaction records and manual datasets, and natural language processing techniques are applied to optimize the model's parameters.

[1261] 4. The server stores the trained generative artificial intelligence model and deploys it. It configures an API endpoint to make it available from the operator display terminal.

[1262] Real-time response support phase

[1263] 1. The user contacts customer support. In the case of a phone inquiry, the device captures the user's voice in real time. In the case of a chat inquiry, the device receives the text message directly.

[1264] 2. In the case of a phone inquiry, the device uses speech recognition software (e.g., Google Speech-to-Text API) to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[1265] 3. The terminal sends the transcribed query to the server. This transmission is in the form of an HTTP request and includes the query content and metadata (e.g., query type, time, etc.).

[1266] 4. The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[1267] 5. The server analyzes the user's emotions using an emotion engine (e.g., an EmotionRecognizer engine). It recognizes the user's emotions from the voice data in the case of voice queries, and from the text data in the case of text queries.

[1268] 6. The server uses the analyzed sentiment data to refine the model response using a generative artificial intelligence model. For example, if the user is angry, it will generate a model response that includes a more courteous response.

[1269] 7. The server sends the generated model answer and sentiment data back to the terminal. The response includes the model answer and any additional relevant information (e.g., links to relevant manual pages).

[1270] 8. The terminal displays the received model answer and sentiment data on the operator display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and relevant documents.

[1271] 9. The operator provides the user with a response, referring to the displayed model answer and sentiment data. The operator provides appropriate instructions to the user, supplementing relevant information as needed.

[1272] 10. The terminal controls the information viewing screen and helps operators quickly access relevant information. For example, if there is an inquiry about the return procedure, it will automatically display the detailed page for the return procedure.

[1273] Specific example

[1274] Example 1: Telephone inquiry

[1275] 1. The user inquires about the return procedure for the product.

[1276] 2. The device converts the user's voice into text in real time (e.g., "I would like to return the item, how do I do that?").

[1277] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[1278] 4. The server uses an emotion engine to analyze the user's voice and recognize that the user is dissatisfied.

[1279] 5. The server adjusts the model answer based on the analysis results and generates a more detailed explanation.

[1280] 6. The server sends this model answer and sentiment data back to the terminal, which then displays it on the operator's display.

[1281] 7. The operator will provide appropriate instructions to the user based on the displayed answer and adjust their response based on the displayed sentiment data.

[1282] Example 2: Chat support

[1283] 1. The user sends a question via chat about a network connection problem.

[1284] 2. The terminal forwards this text to the server as is.

[1285] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[1286] 4. The server uses an emotion engine to analyze the user's emotions from their text and recognizes that the user is anxious.

[1287] 5. The server adjusts the model answer based on the analysis results and generates an answer that includes calming language.

[1288] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays it on the operator's chat screen.

[1289] 7. Operators will respond to users quickly and appropriately, taking into account the displayed information. In doing so, they will strive to reassure users by considering emotional data.

[1290] The system of the present invention can provide a quick and emotionally responsive response to prompts such as "I want to return this item, what should I do?" or "I bought this, but it doesn't work, what should I do?", thereby improving customer satisfaction.

[1291] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1292] Step 1:

[1293] A user contacts customer support. Input is either voice or text. Output is either audio data or a text message.

[1294] Step 2:

[1295] For phone inquiries, the device uses speech recognition software (e.g., Google Speech-to-Text API) to convert the user's voice into text in real time. Input is audio data. Output is text data.

[1296] Step 3:

[1297] The terminal sends the converted text or chat text to the server. This transmission is in the form of an HTTP request and includes the inquiry content and metadata (e.g., inquiry type, time, etc.). The input is text data and metadata. The output is an HTTP request.

[1298] Step 4:

[1299] The server inputs the received text data into a generative artificial intelligence model and generates a model answer based on the query. The input is text data. The output is the model answer.

[1300] Step 5:

[1301] The server analyzes the user's emotions using an emotion engine (e.g., an EmotionRecognizer engine). It recognizes the user's emotions from audio data in the case of voice queries, and from text data in the case of text queries. Input is either text data or audio data. Output is emotion data.

[1302] Step 6:

[1303] The server uses a generative artificial intelligence model to adjust the model answer based on the analyzed emotion data. For example, if the user is angry, it will generate a model answer that includes a more courteous response. The input is emotion data and a model answer. The output is the adjusted model answer.

[1304] Step 7:

[1305] The server sends the generated model answer and sentiment data back to the terminal. The response includes the model answer and related additional information (e.g., links to relevant manual pages). The input is the adjusted model answer and sentiment data. The output is an HTTP response.

[1306] Step 8:

[1307] The terminal displays the received model answer and sentiment data on the operator's display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and related documentation. Input is an HTTP response. Output is the display.

[1308] Step 9:

[1309] The operator provides answers to the user, referring to the displayed model answers and sentiment data. As needed, the operator provides appropriate instructions to the user, supplementing them with relevant information. Input is the information displayed on the screen. Output is the response to the user.

[1310] Step 10:

[1311] The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, it automatically displays the detailed return procedure page. The input is the user's inquiry. The output is an automatically displayed information.

[1312] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1313] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1314] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1315] [Fourth Embodiment]

[1316] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1317] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1318] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1319] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1320] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1321] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1322] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1323] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1324] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1325] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1326] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1327] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1328] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1329] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then uses the generative artificial intelligence model to generate a model answer, and finally displays it on an operator display.

[1330] Initial setup and learning phase

[1331] 1. The server retrieves past customer interaction records in both audio and text format from the database and converts these records to text. In the case of audio records, speech recognition technology is used to convert them to text.

[1332] 2. The server uses the converted text data to train a generative artificial intelligence model (e.g., a natural language processing model). This training process includes data preprocessing (noise removal, formatting standardization, etc.).

[1333] 3. The server deploys the trained generative artificial intelligence model and configures API endpoints so that they can be used on each operator display terminal.

[1334] Real-time response support phase

[1335] 1. The user contacts customer support by phone or chat.

[1336] In the case of phone inquiries, the terminal captures the user's voice in real time.

[1337] In the case of chat support, the device directly receives the text message sent by the user.

[1338] 2. In the case of a phone inquiry, the terminal uses speech recognition software to convert the user's voice into text.

[1339] The converted text is stored in the device's temporary memory.

[1340] 3. The device sends the text converted from the voice or the received chat text to the server.

[1341] This transmission is in the form of an HTTP request and includes the query content and metadata (query type, time, etc.).

[1342] 4. The server inputs the received text into a generative artificial intelligence model and generates a model answer.

[1343] For example, in response to an inquiry such as "Please tell me about returns," a generative artificial intelligence model will generate a model answer such as, "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[1344] 5. The server sends the generated model answer back to the terminal.

[1345] This response is also sent in HTTP response format and includes not only the model answer but also related information (for example, a link to the relevant page in the operation manual).

[1346] 6. The terminal displays the model answer received from the server on the operator's display.

[1347] The information displayed includes the answers and related documents the user is looking for.

[1348] 7. The operator will provide appropriate assistance to the user, referring to the provided model answer.

[1349] If necessary, the operator will assist the customer while supplementing the information with relevant details.

[1350] 8. The terminal also controls the information viewing screen, assisting operators in quickly accessing relevant information.

[1351] For example, if an inquiry is received regarding the return process, the detailed return procedure page will be automatically displayed.

[1352] Specific example

[1353] Example 1: Telephone inquiry

[1354] 1. The user inquires about the return procedure for the product.

[1355] 2. The device converts the user's voice into text in real time (e.g., "I would like to return the item, how do I do that?").

[1356] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[1357] 4. The server sends this model answer back to the terminal, which then displays it on the operator's display.

[1358] 5. The operator provides appropriate instructions to the user based on the displayed answer.

[1359] Example 2: Chat support

[1360] 1. The user sends a question via chat about a network connection problem.

[1361] 2. The terminal forwards this text to the server as is.

[1362] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[1363] 4. The server sends the generated model answer back to the terminal, which then displays it on the operator's chat screen.

[1364] 5. The operator replies to the user with the displayed information.

[1365] Thus, the system of the present invention can reduce the burden on operators, shorten response times, and improve customer satisfaction.

[1366] The following describes the processing flow.

[1367] Step 1:

[1368] The server retrieves past customer interaction records in both audio and text format from the database and converts these records to text. For retrieved audio records, speech recognition software is used to convert them to text.

[1369] Step 2:

[1370] The server preprocesses this text data. Preprocessing includes text cleanup (noise removal, formatting standardization, etc.).

[1371] Step 3:

[1372] The server trains a generative artificial intelligence model using pre-processed data. During the training process, customer interaction records and manual datasets are input into the model, and natural language processing techniques are applied to optimize the model's parameters.

[1373] Step 4:

[1374] The server stores the trained generative artificial intelligence model and deploys it. It then configures an API endpoint to make it available from the operator display terminal.

[1375] Step 5:

[1376] A user contacts customer support. For phone inquiries, the device captures the user's voice in real time. For chat inquiries, the device receives text messages directly.

[1377] Step 6:

[1378] When a user makes a voice inquiry, the device uses speech recognition software to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[1379] Step 7:

[1380] The terminal sends the transcribed query content to the server. This transmission is in the form of an HTTP request and includes the query content and metadata (e.g., query type, time, etc.).

[1381] Step 8:

[1382] The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[1383] Step 9:

[1384] The server sends the generated model answer back to the terminal in HTTP response format. The response includes the model answer and any related additional information (e.g., links to relevant manual pages).

[1385] Step 10:

[1386] The terminal displays the received model answer on the operator's display. The displayed information includes the appropriate answer to the user's inquiry and related documents.

[1387] Step 11:

[1388] The operator provides the user with an answer, referring to the displayed model answer. If necessary, the operator provides the user with appropriate instructions, supplementing them with relevant information.

[1389] Step 12:

[1390] The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, the terminal automatically displays the detailed return procedure page.

[1391] This series of steps reduces the burden on operators and shortens response times.

[1392] (Example 1)

[1393] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1394] In modern customer support operations, responding quickly and accurately to customer inquiries is crucial. However, customer inquiries cover a wide range of topics, requiring operators to provide appropriate responses each time. This is time-consuming and labor-intensive, and the quality of service can vary from operator to operator. Furthermore, a lack of systems that allow for quick access to information related to inquiries leads to longer response times and a decline in customer satisfaction.

[1395] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1396] In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry audio received in real time into text; means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model; means for automatically displaying an information viewing screen related to the inquiry; means for constructing prompt sentences for generating a model answer using the generative artificial intelligence model; and means for providing additional information related to the model answer (such as a link to the relevant page in the operation manual). This enables operators to respond to customer inquiries quickly and accurately, reduces variability in the quality of responses, and improves customer satisfaction.

[1397] "Past customer interaction records" refer to all communications and interaction history recorded between customers and support operators in the past.

[1398] An "operator's manual" is a document that describes how to use a product or service, troubleshooting procedures, and other related information.

[1399] A "generative artificial intelligence model" is a machine learning model that has the ability to learn from large amounts of text data and perform natural language generation tasks.

[1400] "Inquiry voice messages" refer to audio information transmitted by customers via telephone or voice message.

[1401] "Converting to text" refers to the process of converting audio data into written text.

[1402] "Chat text" refers to the text information that customers send via text chat.

[1403] A "model answer" is the optimal response to a customer inquiry generated by a generative artificial intelligence model.

[1404] An "operator display" refers to a display device or interface used by an operator.

[1405] "Speech recognition technology" is a technology that analyzes speech data and converts it into text data.

[1406] An "information viewing screen" is a display screen used by operators to check relevant information.

[1407] A "prompt statement" is a guide statement that is input to a generative artificial intelligence model.

[1408] This invention relates to a system that trains a generative artificial intelligence model using past customer service records and operation manuals, converts customer inquiries into text in real time, then uses the generative artificial intelligence model to generate model answers, and finally displays them on the operator's display. This system enables operators to provide customer service quickly and effectively.

[1409] First, I will explain the system's initial setup and learning phase.

[1410] The server retrieves past customer interaction records in both audio and text format from a database. Common databases used for this purpose include MySQL and PostgreSQL. Next, the server converts the retrieved audio records into text using speech recognition technologies such as Google Cloud Speech-to-Text or Amazon Transcribe. This conversion process extracts the audio data as text information.

[1411] The server then preprocesses this digitized data. Preprocessing involves removing noise from the data and standardizing its format. The Python pandas library can be used for preprocessing. The preprocessed data is then used to train a generative artificial intelligence model (e.g., OpenAI's GPT-4 or Google's BERT). Deep learning frameworks such as TensorFlow or PyTorch are commonly used for this training process.

[1412] Trained generative artificial intelligence models are deployed to cloud platforms such as AWS EC2 or Google Cloud. After deployment, API endpoints are configured so that they can be used on each operator display terminal. These APIs are often built using web frameworks such as Flask or Django.

[1413] Next, I will explain the real-time response support phase.

[1414] When a user contacts customer support via phone or chat, the device captures the user's voice in real time using a digital microphone in the case of a phone call. In the case of a chat, the device directly receives the text message sent by the user. Slack API and Zendesk Chat are used to receive chat messages.

[1415] The device then uses Google Cloud Speech-to-Text to convert the user's speech to text, in the case of a phone call. The converted text is stored in temporary memory. The device then sends the converted text or chat text to the server. This transmission is in the form of an HTTP POST request and includes metadata such as the content of the inquiry, the type of inquiry, and the time.

[1416] The server inputs the received text into a generative artificial intelligence model. A prompt is used in this process. For example, by inputting a prompt such as "The user asked: Please tell me about the return procedure," the generative AI model generates a model answer. This model answer might be something like, "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[1417] The generated model answer is sent back to the terminal in HTTP response format. In addition to the model answer, it also includes relevant information (e.g., a link to the relevant page in the operation manual). The terminal displays this model answer on the operator display. The display includes the answer the user is looking for and related documents. The operator uses the provided model answer as a reference to respond appropriately to the user. If necessary, the operator will provide additional information to assist the customer.

[1418] Furthermore, the terminal also controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry about a return is received, it automatically displays the detailed return procedure page. In this way, the system can reduce the burden on operators, shorten response times, and improve customer satisfaction.

[1419] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1420] Step 1:

[1421] The server retrieves past customer interaction records in audio and text format from a database. MySQL is used as the database for retrieval. The input data consists of customer interaction records, and the output is the retrieved record data.

[1422] Step 2:

[1423] The server converts audio data from acquired customer interaction records into text using Google Cloud Speech-to-Text. The input data is the audio recording, and the output is the text data converted from the audio. Specifically, the audio file is sent to the API, and a reply is received in text format.

[1424] Step 3:

[1425] The server preprocesses the text data. Here, the Python pandas library is used to remove noise and standardize the text format. The input data is the converted text data, and the output is the preprocessed text data.

[1426] Step 4:

[1427] The server trains a generative artificial intelligence model (e.g., GPT-4) using preprocessed text data. TensorFlow is used for this training. The input data is preprocessed text data, and the output is the trained generative artificial intelligence model.

[1428] Step 5:

[1429] The server deploys the trained generative AI model to a cloud platform such as AWS EC2. Furthermore, it configures API endpoints for use on each operator display terminal. The input data is the trained generative AI model, and the output is the deployed model and the available API endpoints.

[1430] Step 6:

[1431] Users contact customer support via phone or chat. For phone inquiries, the device uses a digital microphone to capture the user's voice in real time. Input data is the user's voice inquiry or chat text, and output is the captured audio or directly received chat messages.

[1432] Step 7:

[1433] For voice inquiries, the device uses Google Cloud Speech-to-Text to convert speech to text. The input data is the captured audio, and the output is the converted text.

[1434] Step 8:

[1435] The terminal sends the converted text or received chat text to the server. The transmission is done via an HTTP POST request. The input data is the converted text or chat text, and the output is the text sent to the server.

[1436] Step 9:

[1437] The server inputs the received text into a generative artificial intelligence model. In doing so, it constructs and inputs a prompt. The input data consists of the received text and the prompt, and the output is the generated model answer. A concrete example of a prompt is: "The user asked: Please tell me about the return procedure."

[1438] Step 10:

[1439] The server sends the generated model answer back to the terminal. The return is also done in HTTP response format. The input data is the generated model answer and related information (such as a link to the operation manual), and the output is the model answer sent to the terminal.

[1440] Step 11:

[1441] The terminal displays the model answer received from the server on the operator's display. The input data is the model answer and related documents, and the output is the displayed information.

[1442] Step 12:

[1443] The operator responds to the user while referring to the provided model answer. Additional information is provided as needed to assist the customer. The input data consists of the displayed model answer and related information, while the output is the operator's response action.

[1444] Step 13:

[1445] The terminal automatically controls the information viewing screen so that operators can quickly access relevant information. For example, if there is an inquiry about the return procedure, it will automatically display the page detailing the return procedure. The input data is the inquiry content, and the output is the automatically displayed page of relevant information.

[1446] (Application Example 1)

[1447] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1448] A support system is needed in logistics centers to enable workers to efficiently process customer inquiries and work instructions. Existing systems require workers to manually verify information, which is time-consuming and can lead to misunderstandings and delays in responses. Furthermore, providing accurate instructions to workers in real time is difficult, resulting in insufficient improvements in work efficiency and reduction of errors. To address these issues, a real-time support system utilizing speech recognition technology and generative artificial intelligence is required.

[1449] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1450] In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry voices received in real time into text; means for receiving the converted text or chat text and generating model answers using the generative artificial intelligence model; means for displaying the generated model answers on a visual display device; and means for acquiring voice instructions from workers in the logistics center in real time using a basic speech recognition library and an external API, and displaying the generated model answers on smart glasses. This enables improved work efficiency, reduced errors, and faster customer response in the logistics center.

[1451] "Past customer service records" refers to data that records past customer inquiries and the details of the responses received.

[1452] An "operation manual" refers to a document or guide that describes specific procedures or operating methods.

[1453] A "generative artificial intelligence model" refers to an AI model that performs natural language processing based on a large amount of data to generate model answers or recommended responses.

[1454] "Real-time customer inquiry audio" refers to the immediate reception of audio data sent by customers via telephone or voice chat.

[1455] "Means of converting to text" refers to technologies and devices for converting audio data into text data.

[1456] "Chat text" refers to text data sent in text chat format.

[1457] A "visual display device" refers to a device used to visually display the generated model answer.

[1458] A "speech recognition library" refers to a software component used to convert speech data into text.

[1459] An "external API" refers to an interface used to connect with other systems or services.

[1460] "Smart glasses" refer to glasses-type wearable devices with a built-in display that can show information in real time.

[1461] A "logistics center" refers to a facility that stores, manages, and ships goods.

[1462] A "worker" refers to a person who is responsible for tasks such as receiving, managing, and shipping goods at a logistics center.

[1463] The system that implements this application example includes the following components.

[1464] Components

[1465] 1. Server

[1466] Database: Stores past customer service records and operation manuals.

[1467] Generative artificial intelligence models: Trained using past customer interaction records and operation manuals.

[1468] API endpoint: Provides an interface for accessing trained generative artificial intelligence models.

[1469] 2. Terminal

[1470] Smart glasses: Used by workers in logistics centers to display information in real time.

[1471] Speech recognition library: Converts worker speech into text in real time.

[1472] Data transmission / reception function: Exchanges text and model answers with the server.

[1473] 3. User

[1474] Worker: At the logistics center, they use the system to handle customer inquiries, receive work instructions, and respond to them.

[1475] Explanation of the process

[1476] 1. Preparing the database

[1477] The server retrieves past customer service records and operation manuals from a database and uses this data to train a generative artificial intelligence model.

[1478] 2. Training and Deployment

[1479] The server converts audio recordings into text using speech recognition technology, preprocesses the data using a generative artificial intelligence model, and trains the model. The trained generative AI model becomes accessible through an API endpoint.

[1480] 3. Real-time processing

[1481] Users wear smart glasses within the logistics center and make inquiries and give instructions using voice commands.

[1482] The speech recognition library converts the speech to text, and the device sends the text to the server.

[1483] The server inputs the received text into a generative artificial intelligence model and generates a model answer in real time.

[1484] The model answer is sent to the device and displayed on the smart glasses' screen.

[1485] Hardware and software to be used

[1486] 1. Speech recognition library

[1487] Libraries used: Python's speech_recognition

[1488] Purpose: Converts audio data to text.

[1489] 2. External API

[1490] How to use: Send a query to a generative artificial intelligence model using an HTTP request and obtain a model answer.

[1491] 3. Smart Glasses

[1492] Device used: A wearable device in the form of glasses with a built-in display, worn by workers at a logistics center.

[1493] Specific example

[1494] In a logistics center, when a worker wears smart glasses to retrieve a specific item from a storage shelf, they are asked aloud, "What is the next item?" A speech recognition library captures this audio and converts it to text. The converted text is sent to a server, where a generative artificial intelligence model generates the optimal response. This response is displayed on the smart glasses' screen in the form of, "Please retrieve the item on shelf B5."

[1495] Example of a prompt

[1496] The following is an example of a prompt message that describes the specific operation of this system.

[1497] "Please explain how to create a system that uses speech recognition technology to convert a user's voice into text in real time, inputs that text into a generative AI model to generate a model answer, and displays the result on smart glasses."

[1498] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1499] Step 1:

[1500] Users wear smart glasses and make inquiries and give instructions by voice within the logistics center.

[1501] Input: User's voice (e.g., "What is the next product?")

[1502] Operation:

[1503] The user speaks into the microphone of the smart glasses. The smart glasses capture the audio.

[1504] Step 2:

[1505] The device uses a speech recognition library to convert the user's speech into text.

[1506] Input: User's voice captured in Step 1

[1507] Data processing: A speech recognition library is used to convert audio data into text.

[1508] Output: Text (Example: "What is the next product?")

[1509] Operation:

[1510] The speech recognition library (Python's speech_recognition) converts the speech into text data and stores it in temporary memory.

[1511] Step 3:

[1512] The terminal sends the converted text to the server in the form of an HTTP request.

[1513] Input: Text generated in Step 2

[1514] Data processing: Converting text data into HTTP request format.

[1515] Output: HTTP request sent to the server

[1516] Operation:

[1517] The text data is sent as an HTTP request to the server's API endpoint.

[1518] Step 4:

[1519] The server inputs the received text into a generative artificial intelligence model and generates a model answer.

[1520] Input: Text received as an HTTP request

[1521] Data processing: Generative artificial intelligence models generate the optimal model answer from text.

[1522] Output: Generated model answer (Example: "Please take the item from shelf B5.")

[1523] Operation:

[1524] A generative artificial intelligence model on the server generates a model answer based on the received text.

[1525] Step 5:

[1526] The server sends the generated model answer back to the terminal in HTTP response format.

[1527] Input: Generated model answer

[1528] Data processing: Convert model answers into HTTP response format.

[1529] Output: HTTP response sent to the terminal

[1530] Operation:

[1531] The server sends an HTTP response containing the model answer back to the terminal.

[1532] Step 6:

[1533] The device displays the received model answer on the smart glasses' screen.

[1534] Input: Model answer included in the HTTP response received in Step 5

[1535] Data processing: Convert model answers to a display format.

[1536] Output: Model answer displayed on the smart glasses screen (Example: "Please take the item from shelf B5.")

[1537] Operation:

[1538] The device converts the model answer data into an appropriate display format and displays it on the smart glasses' screen.

[1539] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1540] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then combines this with an emotion engine that recognizes the user's emotions, generates a model answer using the generative artificial intelligence model, and finally displays it on an operator display.

[1541] Initial setup and learning phase

[1542] 1. The server retrieves past customer interaction records in audio and text format from the database and converts these records to text. For retrieved audio records, speech recognition software is used to convert them to text.

[1543] 2. The server preprocesses this text data. Preprocessing includes text cleanup (noise removal, formatting standardization, etc.).

[1544] 3. The server trains a generative artificial intelligence model using pre-processed data. During the training process, the model is fed a dataset of customer interaction records and manuals, and natural language processing techniques are applied to optimize the model's parameters.

[1545] 4. The server saves the trained generative artificial intelligence model and deploys it. It configures an API endpoint to make it available from the operator display terminal.

[1546] Real-time response support phase

[1547] 1. The user contacts customer support. In the case of a phone inquiry, the device captures the user's voice in real time. In the case of a chat inquiry, the device receives the text message directly.

[1548] 2. When the device receives a phone inquiry, it uses speech recognition software to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[1549] 3. The terminal sends the text-based query to the server. This transmission is in the form of an HTTP request and includes the query content and metadata (e.g., query type, time, etc.).

[1550] 4. The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[1551] 5. The server uses an emotion engine to analyze the user's emotions. In the case of voice queries, the server recognizes the user's emotions from the voice data; in the case of text queries, it recognizes them from the text data.

[1552] 6. Based on the sentiment data analyzed by the server, the generative artificial intelligence model adjusts the model answer. For example, if the user is angry, it will generate a model answer that includes a more courteous response.

[1553] 7. The server sends the generated model answer and sentiment data back to the terminal. The response includes the model answer and any additional relevant information (e.g., links to relevant manual pages).

[1554] 8. The terminal displays the received model answer and sentiment data on the operator display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and relevant documents.

[1555] 9. The operator provides the user with a response, referring to the displayed model answer and sentiment data. The operator provides appropriate instructions to the user, supplementing relevant information as needed.

[1556] 10. The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, the terminal automatically displays the detailed page for the return procedure.

[1557] Specific example

[1558] Example 1: Telephone inquiry

[1559] 1. The user inquires about the return procedure for the product.

[1560] 2. The device converts the user's voice into text in real time (e.g., "I would like to return the item, how do I do that?").

[1561] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[1562] 4. The server uses an emotion engine to analyze the user's voice and recognize that the user is dissatisfied.

[1563] 5. The server adjusts the model answer based on the analysis results and generates a more detailed explanation.

[1564] 6. The server sends this model answer and sentiment data back to the terminal, which then displays it on the operator's display.

[1565] 7. The operator will provide appropriate instructions to the user based on the displayed answer and adjust their response based on the displayed sentiment data.

[1566] Example 2: Chat support

[1567] 1. The user sends a question via chat about a network connection problem.

[1568] 2. The terminal forwards this text to the server as is.

[1569] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[1570] 4. The server uses an emotion engine to analyze the user's emotions from their text and recognizes that the user is anxious.

[1571] 5. The server adjusts the model answer based on the analysis results and generates an answer that includes calming language.

[1572] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays it on the operator's chat screen.

[1573] 7. Operators will respond to users quickly and appropriately, taking into account the displayed information. In doing so, they will strive to reassure users by considering emotional data.

[1574] Thus, the system of the present invention reduces the burden on operators, shortens response times, improves customer satisfaction, and enables appropriate responses that respond to the user's emotions.

[1575] The following describes the processing flow.

[1576] Step 1:

[1577] The server retrieves past customer interaction records in both audio and text format from the database, and converts the audio records to text using speech recognition software. The retrieved text data is preprocessed to remove noise and standardize the format.

[1578] Step 2:

[1579] The server trains a generative artificial intelligence model using pre-processed data. Natural language processing techniques are used to optimize the model's parameters during training. Once the training is complete, the model is saved, and an API endpoint is configured for use on each operator's display terminal.

[1580] Step 3:

[1581] A user contacts customer support. For phone inquiries, the device captures the user's voice in real time. For chat inquiries, the device receives text messages directly.

[1582] Step 4:

[1583] When a call is made, the device uses speech recognition software to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[1584] Step 5:

[1585] The terminal sends the text-based query content to the server. The transmission is in HTTP request format and includes the query content and metadata (e.g., query type, time, etc.).

[1586] Step 6:

[1587] The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[1588] Step 7:

[1589] The server uses an emotion engine to analyze the user's emotions. If audio data is available, it recognizes emotions from the audio; if text data is available, it analyzes emotions from the text. For example, it analyzes tone and speed of voice from audio, and word choice and expression from text.

[1590] Step 8:

[1591] The server adjusts the model response based on the analyzed sentiment data. If the user is clearly dissatisfied, it generates a more polite and helpful message. For example, it might add phrases like, "We apologize for your dissatisfaction with the return process..."

[1592] Step 9:

[1593] The server returns the generated model answer and sentiment data to the terminal in HTTP response format. The response includes the model answer and related additional information (e.g., links to relevant manual pages).

[1594] Step 10:

[1595] The terminal displays the received model answer and sentiment data on the operator's display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and relevant documents.

[1596] Step 11:

[1597] The operator provides answers to the user, referring to the displayed model answers and sentiment data. If necessary, they adjust their response based on the displayed sentiment data and provide appropriate instructions to the user.

[1598] Step 12:

[1599] The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, the system automatically displays the detailed return procedure page. This series of steps reduces the burden on operators and shortens response times. It also enables appropriate responses that are tailored to the user's emotions.

[1600] (Example 2)

[1601] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1602] Traditional customer support systems have several drawbacks, including the significant burden placed on operators when responding to customer inquiries in real time. Furthermore, the difficulty in responding to customer emotions and the resulting decrease in customer satisfaction are concerns. Additionally, the difficulty in quickly accessing relevant information leads to longer response times and inefficient support.

[1603] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry voice received in real time into text; means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model; means for analyzing the user's emotions and adjusting the generated model answer; means for displaying the generated model answer on an operator display; and means for automatically displaying a related information viewing screen so that the operator can quickly access related information. This makes it possible to reduce the burden on operators, shorten response times, and improve customer satisfaction.

[1604] "Past customer interaction records" refer to data that includes details of previous interactions with customers.

[1605] An "operator's manual" is a document that explains how to use a system or product.

[1606] A "generative artificial intelligence model" is a machine learning model that has the ability to generate natural language based on a large amount of data.

[1607] "Converting to text" is the process of converting non-text data, such as audio or images, into written text.

[1608] "Speech recognition technology" is a technology that converts speech data into text in real time.

[1609] A "model answer" is a standard or recommended response provided to a customer inquiry.

[1610] "Emotional analysis" is the process of identifying a customer's emotional state from their voice or text.

[1611] An "operator display" is a display device used by operators to view and manipulate information.

[1612] The "Related Information Viewing Screen" is a screen that displays the necessary information in response to customer inquiries.

[1613] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then combines it with an emotion engine to recognize the user's emotions, generates a model answer using the generative artificial intelligence model, and finally displays it on an operator display.

[1614] System Configuration

[1615] Hardware and software

[1616] The server manages the database, trains and deploys generative artificial intelligence models, and performs speech recognition and sentiment analysis. Software used includes Python, SQL, OpenAI GPT-3, IBM Watson Tone Analyzer, and Google Cloud Speech-to-Text.

[1617] Terminal: Captures user inquiries and exchanges data with the server. The terminal is equipped with voice recognition software, a browser, and an operator display.

[1618] User: Contact us by phone or chat.

[1619] Processing of the invention

[1620] Initial setup and learning phase

[1621] 1. The server retrieves past customer interaction records (in audio and text format) from the database and converts them to text using speech recognition software (Google Cloud Speech-to-Text).

[1622] 2. The server cleans up the acquired text and performs preprocessing to train a generative artificial intelligence model (OpenAI GPT-3).

[1623] 3. The server trains a generative artificial intelligence model using preprocessed data, and saves and deploys the model after training is complete. An API endpoint is configured to allow access from the operator's display terminal.

[1624] Real-time response support phase

[1625] 1. The user makes an inquiry via phone or chat.

[1626] 2. If the device is handling a phone inquiry, it captures the audio in real time and converts it to text using Google Cloud Speech-to-Text.

[1627] 3. The terminal sends the converted text to the server.

[1628] 4. The server inputs the received text into a generative artificial intelligence model and generates a model answer.

[1629] 5. The server uses an emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotions and generate an appropriate model answer.

[1630] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays them on the operator's display.

[1631] 7. The operator will respond to the user based on the displayed information and help them quickly access relevant information.

[1632] Specific example

[1633] Example 1: Telephone inquiry

[1634] 1. The user inquires about the return procedure for the product.

[1635] 2. The device converts the user's voice into text in real time (e.g., "I would like to return this item, how do I do that?").

[1636] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[1637] 4. The server uses an emotion engine to analyze the user's voice and recognizes that the user is dissatisfied.

[1638] 5. The server adjusts the model answer based on the analysis results and generates a more polite instruction message (e.g., "We sincerely apologize for the inconvenience. For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[1639] 6. The server sends this model answer and sentiment data back to the terminal, which then displays it on the operator's display.

[1640] Example 2: Chat support

[1641] 1. The user sends a question via chat about a network connection problem.

[1642] 2. The terminal forwards this text to the server as is.

[1643] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[1644] 4. The server uses an emotion engine to analyze the user's emotions from their text and recognizes that the user is anxious.

[1645] 5. The server adjusts the model answer based on the analysis results and generates an answer that includes calming language (e.g., "First, please stay calm. Try restarting your router. If that doesn't solve the problem, contact your provider to check your internet connection.").

[1646] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays it on the operator's chat screen.

[1647] This will reduce the burden on operators, shorten response times, and improve customer satisfaction.

[1648] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1649] Step 1:

[1650] The server retrieves past customer interaction records (in audio and text format) from the database. Specifically, it uses SQL queries to retrieve customer interaction records from the database and saves them as audio and text files. The input is the customer interaction records from the database, and the output is the audio and text files stored on the server.

[1651] Step 2:

[1652] The server calls speech recognition software (Google Cloud Speech-to-Text) to convert the audio recording into text. The input is an audio file, and the output is the converted text data. The server sends the audio file to the API and receives and temporarily stores the converted text data.

[1653] Step 3:

[1654] The server preprocesses the acquired text data. Preprocessing includes noise reduction, formatting standardization, and spell checking. Specifically, it removes unwanted characters using regular expressions and cleanses the text using Python's "nltk" library. The input is the converted text data, and the output is the preprocessed text data.

[1655] Step 4:

[1656] The server trains a generative artificial intelligence model (such as OpenAI's GPT-3) using preprocessed text data. The input is preprocessed text data, and the output is the trained generative AI model. The server inputs the data into the model and optimizes the model's parameters.

[1657] Step 5:

[1658] The server stores the trained generative AI model and deploys it as an API endpoint. The input is the trained generative AI model, and the output is the API endpoint. This allows the model to be accessed from the operator display terminal.

[1659] Step 6:

[1660] Users contact customer support via phone or chat. In the case of phone inquiries, the device captures the user's voice in real time and converts it to text using speech recognition software such as Google Cloud Speech-to-Text. The input is the user's voice, and the output is the converted text.

[1661] Step 7:

[1662] The terminal sends the converted text to the server. The input is the converted text, and the output is an HTTP request sent to the server. This includes the query content and metadata.

[1663] Step 8:

[1664] The server inputs the received text into a generative artificial intelligence model and generates a model answer based on the inquiry. The input is the received text data, and the output is the model answer. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[1665] Step 9:

[1666] The server analyzes the user's emotions using an emotion engine (such as IBM Watson Tone Analyzer). The input is received text or audio data, and the output is the analyzed emotion data. Based on the analysis results, the server adjusts the generated model answer.

[1667] Step 10:

[1668] The server sends the final model answer and sentiment data back to the terminal. The input is the model answer and sentiment data, and the output is an HTTP response to the terminal, which includes the model answer and any associated additional information.

[1669] Step 11:

[1670] The terminal displays the received model answer and sentiment data on the operator's display. The input is an HTTP response from the server, and the output is the displayed model answer and sentiment information. Based on the displayed information, the operator provides an answer to the customer.

[1671] Step 12:

[1672] The terminal automatically displays a related information viewing screen to support operator operations and allow for quick access to relevant information. Input is the inquiry content or model answer, and output is the related information viewing screen. For example, if there is an inquiry about the return procedure, the detailed return procedure page will be automatically displayed.

[1673] Through these steps, the system of the present invention can reduce the burden on operators, shorten response times, and improve customer satisfaction.

[1674] (Application Example 2)

[1675] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1676] Traditional customer service systems have made it difficult to grasp customer emotions in real time, hindering the ability to provide appropriate responses quickly. To improve customer satisfaction, it is necessary to provide timely and appropriate answers to customer inquiries while simultaneously responding in a way that respects the customer's emotions.

[1677] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for training a generative artificial intelligence model using past customer interaction records and operation manuals; means for converting customer inquiry voice received in real time into text; means for receiving the converted text or chat text and generating a model answer using the generative artificial intelligence model; means for displaying the model answer on an operator display; means for analyzing the customer's emotions together with the generated model answer and adjusting the model answer based on the emotion data; and means for displaying the generated model answer and the adjusted model answer on an operator display and providing links to related information. This makes it possible to quickly provide appropriate responses that correspond to the customer's emotions.

[1678] "Past customer service records" refer to data that records the history of customer inquiries and the status of those responses.

[1679] An "operation manual" is a set of instructions that describes how to operate a specific task or system.

[1680] A "generative artificial intelligence model" is an algorithm that uses artificial intelligence technology to learn from data and generate predictions and responses.

[1681] "Real-time" means processing or reacting immediately in accordance with the current timeline.

[1682] "Converting to text" is the process of converting audio data into written information.

[1683] "Chat text" refers to inquiries made using text messages.

[1684] A "model answer" refers to an ideal response or answer that has been generated.

[1685] An "operator display" is a display device used by an operator.

[1686] "Analyzing emotions" is the process of extracting emotional elements from text or audio and interpreting them.

[1687] A "link to related information" is a reference URL that provides access to necessary information sources or additional materials.

[1688] This invention relates to a system that trains a generative artificial intelligence model using past customer interaction records and operation manuals, converts customer inquiry audio received in real time into text, then combines this with an emotion engine that recognizes the user's emotions, generates a model answer using the generative artificial intelligence model, and finally displays it on an operator display.

[1689] Initial setup and learning phase

[1690] 1. The server retrieves past customer interaction records in audio and text format from the database and converts these records to text. For retrieved audio records, it uses speech recognition software (e.g., SpeechRecognition) to convert them to text.

[1691] 2. The server preprocesses this text data. Preprocessing includes text cleanup (noise removal, formatting standardization, etc.).

[1692] 3. The server trains a generative artificial intelligence model (e.g., the Transformers library) using pre-processed data. During the training process, the model is input with customer interaction records and manual datasets, and natural language processing techniques are applied to optimize the model's parameters.

[1693] 4. The server stores the trained generative artificial intelligence model and deploys it. It configures an API endpoint to make it available from the operator display terminal.

[1694] Real-time response support phase

[1695] 1. The user contacts customer support. In the case of a phone inquiry, the device captures the user's voice in real time. In the case of a chat inquiry, the device receives the text message directly.

[1696] 2. In the case of a phone inquiry, the device uses speech recognition software (e.g., Google Speech-to-Text API) to convert the user's voice into text in real time. The converted text is temporarily stored in the device's memory.

[1697] 3. The terminal sends the transcribed query to the server. This transmission is in the form of an HTTP request and includes the query content and metadata (e.g., query type, time, etc.).

[1698] 4. The server inputs the received text into a generative artificial intelligence model to generate a model answer based on the inquiry. For example, in response to the inquiry "Please tell me about returns," it generates a model answer such as "For details on the return procedure, please return the product along with your proof of purchase and the attached label."

[1699] 5. The server analyzes the user's emotions using an emotion engine (e.g., an EmotionRecognizer engine). It recognizes the user's emotions from the voice data in the case of voice queries, and from the text data in the case of text queries.

[1700] 6. The server uses the analyzed sentiment data to refine the model response using a generative artificial intelligence model. For example, if the user is angry, it will generate a model response that includes a more courteous response.

[1701] 7. The server sends the generated model answer and sentiment data back to the terminal. The response includes the model answer and any additional relevant information (e.g., links to relevant manual pages).

[1702] 8. The terminal displays the received model answer and sentiment data on the operator display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and relevant documents.

[1703] 9. The operator provides the user with a response, referring to the displayed model answer and sentiment data. The operator provides appropriate instructions to the user, supplementing relevant information as needed.

[1704] 10. The terminal controls the information viewing screen and helps operators quickly access relevant information. For example, if there is an inquiry about the return procedure, it will automatically display the detailed page for the return procedure.

[1705] Specific example

[1706] Example 1: Telephone inquiry

[1707] 1. The user inquires about the return procedure for the product.

[1708] 2. The device converts the user's voice into text in real time (e.g., "I would like to return the item, how do I do that?").

[1709] 3. The server inputs the converted text into a generative artificial intelligence model to generate a model answer (e.g., "For details on the return procedure, please return the product along with your proof of purchase and the attached label.").

[1710] 4. The server uses an emotion engine to analyze the user's voice and recognize that the user is dissatisfied.

[1711] 5. The server adjusts the model answer based on the analysis results and generates a more detailed explanation.

[1712] 6. The server sends this model answer and sentiment data back to the terminal, which then displays it on the operator's display.

[1713] 7. The operator will provide appropriate instructions to the user based on the displayed answer and adjust their response based on the displayed sentiment data.

[1714] Example 2: Chat support

[1715] 1. The user sends a question via chat about a network connection problem.

[1716] 2. The terminal forwards this text to the server as is.

[1717] 3. The server inputs the received text into a generative artificial intelligence model and generates a model answer (e.g., "First, try restarting your router. If that doesn't work, contact your internet provider to check your connection status.").

[1718] 4. The server uses an emotion engine to analyze the user's emotions from their text and recognizes that the user is anxious.

[1719] 5. The server adjusts the model answer based on the analysis results and generates an answer that includes calming language.

[1720] 6. The server sends the generated model answer and sentiment data back to the terminal, which then displays it on the operator's chat screen.

[1721] 7. Operators will respond to users quickly and appropriately, taking into account the displayed information. In doing so, they will strive to reassure users by considering emotional data.

[1722] The system of the present invention can provide a quick and emotionally responsive response to prompts such as "I want to return this item, what should I do?" or "I bought this, but it doesn't work, what should I do?", thereby improving customer satisfaction.

[1723] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1724] Step 1:

[1725] A user contacts customer support. Input is either voice or text. Output is either audio data or a text message.

[1726] Step 2:

[1727] For phone inquiries, the device uses speech recognition software (e.g., Google Speech-to-Text API) to convert the user's voice into text in real time. Input is audio data. Output is text data.

[1728] Step 3:

[1729] The terminal sends the converted text or chat text to the server. This transmission is in the form of an HTTP request and includes the inquiry content and metadata (e.g., inquiry type, time, etc.). The input is text data and metadata. The output is an HTTP request.

[1730] Step 4:

[1731] The server inputs the received text data into a generative artificial intelligence model and generates a model answer based on the query. The input is text data. The output is the model answer.

[1732] Step 5:

[1733] The server analyzes the user's emotions using an emotion engine (e.g., an EmotionRecognizer engine). It recognizes the user's emotions from audio data in the case of voice queries, and from text data in the case of text queries. Input is either text data or audio data. Output is emotion data.

[1734] Step 6:

[1735] The server uses a generative artificial intelligence model to adjust the model answer based on the analyzed emotion data. For example, if the user is angry, it will generate a model answer that includes a more courteous response. The input is emotion data and a model answer. The output is the adjusted model answer.

[1736] Step 7:

[1737] The server sends the generated model answer and sentiment data back to the terminal. The response includes the model answer and related additional information (e.g., links to relevant manual pages). The input is the adjusted model answer and sentiment data. The output is an HTTP response.

[1738] Step 8:

[1739] The terminal displays the received model answer and sentiment data on the operator's display. The displayed information includes the appropriate answer to the user's inquiry, sentiment data, and related documentation. Input is an HTTP response. Output is the display.

[1740] Step 9:

[1741] The operator provides answers to the user, referring to the displayed model answers and sentiment data. As needed, the operator provides appropriate instructions to the user, supplementing them with relevant information. Input is the information displayed on the screen. Output is the response to the user.

[1742] Step 10:

[1743] The terminal controls the information viewing screen, helping operators quickly access relevant information. For example, if an inquiry is made regarding the return procedure, it automatically displays the detailed return procedure page. The input is the user's inquiry. The output is an automatically displayed information.

[1744] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1745] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1746] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1747] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1748] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1749] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1750] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1751] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1752] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1753] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1754] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1755] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1756] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1757] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1758] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1759] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1760] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1761] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1762] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1763] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1764] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1765] The following is further disclosed regarding the embodiments described above.

[1766] (Claim 1)

[1767] A means of training a generative artificial intelligence model using past customer service records and operation manuals,

[1768] A means of converting customer inquiry audio received in real time into text,

[1769] A means for receiving the converted text or chat text and generating a model answer using a generative artificial intelligence model,

[1770] A system including means for displaying the generated model answer on an operator display.

[1771] (Claim 2)

[1772] The system according to claim 1, wherein the means for converting the aforementioned speech into text in real time utilizes speech recognition technology.

[1773] (Claim 3)

[1774] The system according to claim 1, further comprising means for displaying a screen for viewing relevant information based on the customer's inquiry when generating the aforementioned model answer.

[1775] "Example 1"

[1776] (Claim 1)

[1777] A means of training a generative artificial intelligence model using past customer service records and operation manuals,

[1778] A means of converting customer inquiry audio received in real time into text,

[1779] A means for receiving the converted text or chat text and generating a model answer using a generative artificial intelligence model,

[1780] Means for displaying the generated model answer on an operator display,

[1781] A means to automatically display a screen for viewing information related to the inquiry,

[1782] Means for constructing a prompt sentence for generating a model answer using the aforementioned generative artificial intelligence model,

[1783] A system that includes means of providing additional information related to the model answer (such as a link to the relevant page in the operation manual).

[1784] (Claim 2)

[1785] The system according to claim 1, wherein the means for converting the aforementioned speech into text in real time utilizes speech recognition technology.

[1786] (Claim 3)

[1787] The system according to claim 1, further comprising means for displaying a screen for viewing relevant information based on the customer's inquiry when generating the aforementioned model answer.

[1788] "Application Example 1"

[1789] (Claim 1)

[1790] A means of training a generative artificial intelligence model using past customer service records and operation manuals,

[1791] A means of converting customer inquiry audio received in real time into text,

[1792] A means for receiving the converted text or chat text and generating a model answer using a generative artificial intelligence model,

[1793] Means for displaying the generated model answer on a visual display device,

[1794] A system that uses a basic speech recognitio...

Claims

1. A means of training a generative artificial intelligence model using past customer service records and operation manuals, A means of converting customer inquiry audio received in real time into text, A means for receiving the converted text or chat text and generating a model answer using a generative artificial intelligence model, A system including means for displaying the generated model answer on an operator display.

2. The system according to claim 1, wherein the means for converting the aforementioned speech into text in real time utilizes speech recognition technology.

3. The system according to claim 1, further comprising means for displaying a screen for viewing relevant information based on the customer's inquiry when generating the aforementioned model answer.

Images