System

Abstract

A system is provided.SOLUTION: AI translation means for translating text in real time, AI emotion analysis means for neutralizing emotions from the translated text by performing emotion analysis, means for displaying the content of the query processed by the translation means and the emotion analysis means on an operator terminal, and means for translating replies input by the operator into the user's language by the translation means and the emotion analysis means and transmitting the neutralized text to the user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to solve the problem of globally deployed call centers, where it is difficult to respond quickly and appropriately when multiple languages ​​are required or when emotional complaints are made by users. Specifically, the purpose is to alleviate the shortage of multilingual personnel, reduce operator stress, and realize effective responses to emotional complaints. [Means for solving the problem]

[0005] The present invention provides a system that includes an AI translation means for translating text in real time, an AI sentiment analysis means for performing sentiment analysis and neutralizing the sentiment of the translated text, a means for displaying the inquiry content processed by the translation means and the sentiment analysis means on an operator terminal, and a means for translating a reply entered by the operator into the user's language using the translation means and the sentiment analysis means and sending the neutralized text to the user terminal. This system eliminates the need for operators to have specific language skills, facilitates multilingual support, and enables neutral and calm responses to emotional complaints. Furthermore, the system also includes a means for receiving the inquiry content and extracting its linguistic information, and a means for detecting emotional expressions in the user's inquiry content and converting them into neutral, less emotional expressions, thereby achieving a smoother and less stressful response.

[0006] "AI translation means that translates text in real time" is a system that uses artificial intelligence to instantly convert inquiry content and operator replies into different languages.

[0007] The "AI sentiment analysis method that performs sentiment analysis and neutralizes emotions from translated text" is a system that uses artificial intelligence to analyze text containing emotional expressions and convert them into neutral expressions that suppress the emotions.

[0008] The "means for displaying the inquiry content on the operator terminal" is a system that has the function of displaying the translated and sentiment-analyzed inquiry content on the terminal used by the operator.

[0009] "Means for translating replies entered by an operator into the user's language and sending the neutralized text to the user's terminal" refers to a system that has the function of converting replies created by an operator into the user's language using translation and sentiment analysis means, and then neutralizing emotional expressions before sending them to the user's terminal.

[0010] "Means for receiving the inquiry content and extracting its language information" refers to a system that has the function of receiving the inquiry content from a user and identifying and extracting the language used.

[0011] The "means for detecting emotional expressions and converting them into neutral expressions with reduced emotional impact" refers to a system that has the function of analyzing emotional expressions contained in received text and converting them into neutral expressions with minimal emotional impact. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] The present invention relates to a system for multilingual and emotional complaint handling, which includes a real-time translation means and a sentiment analysis means, enabling fast and effective responses in global call centers.

[0034] Program processing

[0035] The main components of this system are the server, terminals, and users (operators and users). The roles and operations of each are explained below.

[0036] 1. Receiving inquiries

[0037] The user sends a support request from their device, which includes the content of the inquiry and the language they are using. The device then sends this information to the server.

[0038] 2. Translation and Sentiment Analysis

[0039] The server then passes the received inquiry to a real-time translation unit, which translates the user's language into a language the operator can understand. The emotion analysis unit then analyzes the translated content and converts emotional expressions into neutral ones, allowing the operator to respond calmly based on the information.

[0040] 3. Notification to Operator

[0041] The translated and neutralized query content is sent from the server to the operator's terminal, where it is displayed in an easy-to-understand format.

[0042] 4. Operator response and reply

[0043] The user (operator) responds based on the displayed content and enters a reply, which is then sent from the terminal to the server.

[0044] 5. Reply translation and notification

[0045] The server then passes the operator's reply to the real-time translation tool, which translates it into the user's language. The sentiment analysis tool then converts the translated reply into a neutral expression and sends the final answer to the user's device, ensuring the user receives a calm and clear response.

[0046] Specific examples

[0047] Example 1: When a user makes an inquiry in Japanese

[0048] The user makes an inquiry in Japanese, saying, "The product delivery is delayed. What's going on?" The terminal sends this inquiry to the server, which uses real-time translation means to translate it into English as, "The product delivery is delayed, what is happening?" The emotion analysis means then analyzes this content and neutralizes the emotional expression, "It's delayed, what's going on?" The operator's terminal then displays the neutralized message, "The product delivery is delayed, please advise [current status]."

[0049] The operator replies, "We will investigate, please wait a moment," and sends the reply from the device to the server. The server translates this reply from Japanese to English, saying, "We will investigate, please wait a moment," and then uses sentiment analysis to confirm that it is neutral. The final reply is sent to the user's device.

[0050] Example 2: When a customer makes a complaint in English

[0051] If a user complains in English, "I'm extremely upset with your service. My order was not delivered on time," the device sends this content to the server, which then uses a translation tool to translate it into Japanese as "I'm extremely upset with your service. My order was not delivered on time." The sentiment analysis tool neutralizes the strong expression "I'm dissatisfied," and displays to the operator, "A customer has inquired about their dissatisfaction with the service and the delay in their order."

[0052] The operator replies, "We apologize for the inconvenience, we will check immediately." The server then translates this into English and sends it to the user's device as, "We apologize for the inconvenience, we will check immediately."

[0053] The system of the present invention achieves multilingual support through real-time translation and sentiment analysis, providing a stress-free environment for both operators and users.

[0054] The processing flow will be explained below.

[0055] Step 1:

[0056] A user sends a support request from their own device. This request contains the inquiry content and the user's language information. For example, a user enters a message in Japanese saying, "My product is late. What's going on?" and clicks the send button.

[0057] Step 2:

[0058] The terminal receives the user's input information, sends the query content and the user's language information to the server, packages the user's language and text content, and makes a request to the server.

[0059] Step 3:

[0060] The server passes the received inquiry to an AI translation engine, which translates the user's language into a language the operator can understand. For example, the Japanese phrase "The product delivery is delayed, what's happening?" is translated into English as "The product delivery is delayed, what is happening?"

[0061] Step 4:

[0062] The server then sends the translation results to an AI sentiment analysis engine, which analyzes the translated query, detects any emotional expressions, and converts them into neutral phrases. For example, it neutralizes the emotional expression "what is happening?" to "please advise [current status]."

[0063] Step 5:

[0064] The server sends the translated and neutralized inquiry to the operator terminal. The operator terminal displays the translated and neutralized inquiry. The operator is shown a message saying, "A customer has inquired about a delay in the delivery of their product."

[0065] Step 6:

[0066] The user (operator) responds based on the displayed inquiry and enters a reply. For example, they might type, "I will look into this, so could you please wait a moment?" and click the send button.

[0067] Step 7:

[0068] The terminal sends the operator's input to the server and requests the operator's reply text from the server.

[0069] Step 8:

[0070] The server then passes the operator's reply to the AI ​​translation engine again, which translates it from the operator's language to the user's language. For example, the Japanese phrase "We will investigate, so please wait a moment." is translated into English as "We will investigate, please wait a moment."

[0071] Step 9:

[0072] The server then sends the translated reply back to the AI ​​sentiment analysis engine to neutralize the emotional content. If the content is already neutral, no changes are made.

[0073] Step 10:

[0074] The server then sends the translated and neutralized reply to the user's device. The server then sends the translated reply, "We'll look into this, so could you please wait a moment?"

[0075] Step 11:

[0076] The terminal (user) displays the translated reply received from the server on its screen. The user's screen displays "We will look into this, could you please wait a moment?", and the response is complete.

[0077] The above steps constitute a system that enables multilingual and emotion-neutral communication between users and operators.

[0078] Example 1

[0079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0080] Today's global call centers are required to handle inquiries from users quickly and accurately. However, due to issues such as diverse languages ​​and emotional expressions, it is often difficult for operators to respond efficiently. It is also not easy to respond calmly and neutrally to emotional complaints. This increases the burden on operators and risks a decline in response quality. There is a need for a system that can solve these issues, improve the efficiency of inquiry responses at global call centers, and improve response quality.

[0081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0082] In this invention, the server includes means for receiving inquiries from user terminals, AI translation means for translating text in real time, AI sentiment analysis means for performing sentiment analysis and neutralizing the sentiment of the translated text, means for displaying the inquiry content processed by the translation means and the sentiment analysis means on an operator terminal, and means for translating a reply entered by an operator into the user's language by the translation means and the sentiment analysis means and sending the neutralized text to the user terminal. This enables inquiries to be processed quickly and accurately in a global call center, providing a stress-free environment for both operators and users.

[0083] "User" refers to an individual or company that uses the system to make an inquiry.

[0084] A "terminal" is a device (e.g., PC, smartphone, tablet) used by a user or operator to access the system.

[0085] A "server" is a central computer that manages the processing of the entire system and sends and receives various data.

[0086] "AI translation mechanism" refers to a mechanism that utilizes artificial intelligence technology to translate text into another language in real time.

[0087] An "AI sentiment analysis tool" is a mechanism that uses artificial intelligence technology to analyze and neutralize emotional expressions in text.

[0088] "Inquiry content" refers to the text information of questions or complaints sent by users through the system.

[0089] An "operator" is a system user who responds to inquiries from users and provides necessary replies.

[0090] A "neutralized text" is a text that has been transformed into a calm and objective expression, with emotional expressions suppressed.

[0091] This invention is a system that realizes multilingual support and emotional complaint handling in global call centers. This system is mainly composed of three components: a server, terminals, and users (operators and customers).

[0092] System Overview

[0093] Receiving inquiries

[0094] A user sends a support request from their device (PC, smartphone, etc.). This request includes the inquiry and the language used by the user. The device then sends this information to the server.

[0095] Translation and Sentiment Analysis

[0096] The server passes the received inquiry to a real-time translation tool (e.g., Google Translate API) to translate the user's language into a language the operator can understand. The translated content is then analyzed using a sentiment analysis tool (e.g., IBM Watson Natural Language Understanding) to convert emotional expressions into neutral ones, allowing the operator to respond calmly based on the information.

[0097] Operator notification

[0098] The translated and neutralized inquiry is sent from the server to the operator's terminal, where the inquiry is displayed in an easy-to-understand format through dedicated customer support software.

[0099] Operator response and reply

[0100] The user (operator) responds based on the displayed information and enters a reply, which is then sent from the terminal to the server.

[0101] Reply translation and notification

[0102] The server then passes the operator's reply to the real-time translation unit, which translates it into the user's language. The sentiment analysis unit then converts the translated reply into a neutral expression, and the final answer is sent to the user's device. This allows the user to receive a calm and clear response.

[0103] Specific examples

[0104] Example 1: When a user makes an inquiry in Japanese

[0105] The user makes an inquiry in Japanese, saying, "The product delivery is delayed. What's going on?" The device sends this inquiry to the server. The server uses real-time translation means to translate this into English as, "The product delivery is delayed, what is happening?" The sentiment analysis means then analyzes this content and displays a neutralized response, "The product delivery is delayed, please advise [current status]." on the operator's device. The operator replies, "We will investigate, could you please wait a moment?" and sends this response from the device to the server. The server then translates this response from Japanese to English, saying, "We will investigate, please wait a moment." The sentiment analysis means then confirms the neutral expression, and the final reply is sent to the user's device.

[0106] Example 2: When a customer makes a complaint in English

[0107] A user complains in English, saying, "I'm extremely upset with your service. My order was not delivered on time." The device sends this content to the server. The server uses a translation tool to translate this into Japanese as "I'm extremely unhappy with your service. My order was not delivered on time." The sentiment analysis tool neutralizes the strong expression "unhappy" and displays to the operator, "A customer has inquired about their dissatisfaction with the service and a delayed order." The operator replies, "We apologize, we will check immediately." The server translates this content into English and sends it to the user's device as "We apologize for the inconvenience, we will check immediately."

[0108] Prompt Sentence Examples

[0109] Here are some examples of prompts for generative AI models:

[0110] "Please explain a scenario in which a user makes an inquiry in Japanese and an operator responds. Please also include the specific process flow and the functions used (translation methods, sentiment analysis methods)."

[0111] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0112] Step 1:

[0113] A user sends a support request using their own device. As input, they enter text data containing the inquiry and the user's language information into the device. The device then sends this data to the server as an HTTP request. The output is the inquiry data transferred to the server.

[0114] Step 2:

[0115] The server passes the received query data to a real-time translation method (e.g., Google Translate API). The input is the query text sent by the user and its language information. The server passes this information to the API for translation processing. The output is the translated text returned to the server.

[0116] Step 3:

[0117] The server passes the translated text to a sentiment analyzer (e.g., IBM Watson Natural Language Understanding). The input is the translated query text. The server sends the text to a sentiment analysis API, which analyzes the sentiment in the text. The output is text with neutralized emotional expressions.

[0118] Step 4:

[0119] The server sends the neutralized text to the operator's terminal. The input is the query data processed by the sentiment analysis means. The server transfers this data to the operator's terminal using REST API or WebSocket. The output is the neutralized query content displayed on the operator's terminal.

[0120] Step 5:

[0121] The user (operator) considers how to respond based on the displayed content and enters a reply. As input, the operator enters the proposed response as text into the terminal. The terminal then sends this reply data back to the server. The output is the reply data transferred to the server.

[0122] Step 6:

[0123] The server passes the operator's reply to a real-time translation means. The input is the reply text sent by the operator. The server sends the reply text to a translation means, which translates the reply into the user's language. The output is the translated reply text.

[0124] Step 7:

[0125] The server passes the translated reply to the sentiment analyzer again to confirm the neutralized expression. The input is the translated reply text. The server passes this text to the sentiment analysis API for verification. The output is the reply text with the neutralized expression confirmed.

[0126] Step 8:

[0127] The server sends the final answer to the user's device. The input is the reply text that has been checked for neutrality. The server transfers this text to the user's device using a REST API or WebSocket. The output is the final reply that is displayed on the user's device.

[0128] (Application example 1)

[0129] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] Conventional customer service systems have difficulty dealing with customers who speak different languages ​​and have different cultural backgrounds, and have had particular problems with incomplete multilingual support and emotional complaint handling. Furthermore, in brick-and-mortar stores, staff were not equipped with an environment where they could instantly provide multilingual support, making it difficult to provide prompt and appropriate service.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0132] In this invention, the server includes an artificial intelligence translation means for translating text in real time, an artificial intelligence emotion analysis means for performing emotion analysis and neutralizing the emotion from the translated text, a means for displaying the inquiry content processed by the translation means and the emotion analysis means on a staff terminal, a means for translating a reply entered by a staff member into the user's language by the translation means and the emotion analysis means and sending the neutralized text to the user terminal, and a means for displaying the inquiry content and the reply content on an augmented reality device used as a staff terminal. This enables multilingual and emotional complaint handling to be performed quickly and effectively even in physical stores, thereby improving customer satisfaction.

[0133] "Artificial intelligence translation means for real-time text translation" refers to a device that uses artificial intelligence to instantly convert the content of an inquiry into a different language.

[0134] The "artificial intelligence emotion analysis means for performing emotion analysis and neutralizing emotions from translated text" is a device that uses artificial intelligence to analyze emotional expressions in translated text and convert them into calm expressions.

[0135] "Staff terminal" means an electronic device used by customer service staff to receive and enter information.

[0136] "User terminal" refers to an electronic device used by a service user to receive support or enter inquiries.

[0137] An "augmented reality device" is a device worn by staff that can overlay virtual information onto the actual environment.

[0138] "Inquiry content" refers to the content of questions, requests, and complaints made by users to customer service.

[0139] A "neutralized text" is text that has been transformed from emotional expression into calm, objective expression.

[0140] A "translation tool" is a device or software for converting text from one language to another.

[0141] A "sentiment analyzer" is a device or software for analyzing and identifying emotional elements in text.

[0142] The present invention is a system for multilingual support and emotional complaint handling in brick-and-mortar stores. This system performs real-time translation and sentiment analysis between customers and customer service staff (hereinafter referred to as staff), enabling staff to respond appropriately and quickly. Specific embodiments are described below.

[0143] Overall system configuration

[0144] 1. Server and Cloud Infrastructure

[0145] The server includes the hardware and software necessary to implement real-time translation and sentiment analysis. The main software used is the Google Translator API, which provides translation functionality, the TextBlob library for sentiment analysis, and the SpeechRecognition library for voice recognition.

[0146] 2. Staff terminals and augmented reality devices

[0147] Smart glasses (e.g., Google Glass) are used as staff terminals, and these devices display inquiries and replies to staff in real time.

[0148] 3. User Device

[0149] A user (customer) makes an inquiry using a personal device such as a smartphone.

[0150] Data processing and calculation

[0151] The server processes and calculates data in the following procedure.

[0152] 1. Voice Recognition

[0153] When a user makes a voice inquiry using their smartphone, the voice data is transmitted to the staff member's smart glasses, which then send the voice data to the server, which then converts the voice data into text using the SpeechRecognition library.

[0154] 2. Real-time translation

[0155] The server translates the text obtained by speech recognition in real time using the Google Translator API. For example, it converts a Japanese inquiry into English.

[0156] 3. Sentiment Analysis and Neutralization

[0157] The server performs sentiment analysis on the translated text using the "TextBlob" library. If the sentiment is strong, it converts certain keywords (e.g., "very dissatisfied") into neutral expressions (e.g., "concerned").

[0158] 4. Staff Display

[0159] The neutralized text is processed on the server and displayed on the staff's smart glasses, allowing them to respond appropriately to the customer.

[0160] 5. Reply Translation and Notification

[0161] The reply entered by the staff is sent back to the server, where it is translated and sentiment-analyzed. The translated text is neutralized and then sent to the customer's smartphone.

[0162] Specific use cases

[0163] If a customer asks in Japanese, "The product delivery is delayed, what's going on?", this speech is converted into text on the server and translated into English. The translated text, "The product delivery is delayed, what is happening?", undergoes sentiment analysis and neutralization (e.g., "The product delivery is delayed, please advise [current status].") and is displayed on the staff member's smart glasses. The staff member responds, "We will investigate, please wait a moment," and the reply is also translated into English and notifies the customer, "We will investigate, please wait a moment."

[0164] Example prompt sentence:

[0165] Analyze the following sentences and convert them into neutral expressions.

[0166] Text: "I'm extremely upset with your service. My order was not delivered on time."

[0167] Neutral: "I'm concerned about the delay in my order delivery. Please advise on the current status."

[0168] As described above, the present invention aims to realize multilingual support and emotional complaint handling, thereby improving the quality of customer service in physical stores.

[0169] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0170] Step 1:

[0171] A user makes a voice inquiry using a smartphone. This voice data is collected through the smartphone's microphone and saved as digital data. The input is voice data, and the output is digital voice data.

[0172] Step 2:

[0173] The user's smartphone sends voice data to the server, which converts it into text using the "SpeechRecognition" library. The input is digital voice data, and the output is text data. This conversion transforms the voice data into concrete text information.

[0174] Step 3:

[0175] The server translates the text data in real time using the Google Translator API. For example, Japanese text is converted into English text. The input is Japanese text data, and the output is English text data. This translation converts the data into a language that the staff can understand.

[0176] Step 4:

[0177] The server performs sentiment analysis on the translated English text data using the "TextBlob" library. The sentiment analysis results in positive, negative, and neutral sentiment. The input is the English text data, and the output is the sentiment analysis results and corrected text data. Sentiment analysis neutralizes expressions as needed.

[0178] Step 5:

[0179] The server sends neutralized text data with reduced emotion to the augmented reality device (smart glasses). The staff member's smart glasses receive this data and display it on their screen. The input is the neutralized text data, and the output is the display on the smart glasses.

[0180] Step 6:

[0181] The staff member checks the neutralized text through the smart glasses and types an appropriate reply to the customer. The reply is recorded as audio by the smart glasses' microphone and sent to the server as text data. The input is audio data and the output is text data.

[0182] Step 7:

[0183] The server again translates the staff member's reply text data into the customer's language (e.g., Japanese) using the "Google Translator API." It then performs sentiment analysis on the translated text using the "TextBlob" library to confirm neutralization. The input is the staff member's English text data, and the output is neutralized Japanese text data.

[0184] Step 8:

[0185] The server sends the neutralized Japanese text data to the user's smartphone. The user's smartphone receives this data and displays it on its screen. The input is the neutralized Japanese text data, and the output is what is displayed on the smartphone's screen.

[0186] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0187] The present invention relates to a system for multilingual and emotional complaint handling, which includes a real-time translation means, a sentiment analysis means, and an emotion engine, enabling fast and effective responses in global call centers.

[0188] Program processing

[0189] The main components of this system are the server, terminals, and users (operators and users). The roles and operations of each are explained below.

[0190] 1. Receiving inquiries

[0191] A user sends a support request from their own device. This request contains the inquiry content and the user's language information. For example, a user enters a message in Japanese saying, "My product is late. What's going on?" and clicks the send button.

[0192] 2. Translation and Sentiment Analysis

[0193] The server passes the received inquiry to a real-time translation means, which translates it from the user's language into a language the operator can understand. For example, the Japanese phrase "The product delivery is delayed, what's going on?" is translated into English as "The product delivery is delayed, what is happening?". Next, the sentiment analysis means analyzes the translated content and converts emotional expressions into neutral expressions. For example, the emotional expression "what is happening?" is neutralized to "please advise [current status]."

[0194] 3. Notification to Operator

[0195] The translated and neutralized inquiry is sent from the server to the operator's terminal. The operator's terminal displays the translated and neutralized inquiry. For example, the operator might see, "A customer has inquired about a delay in the arrival of their product."

[0196] 4. Operator response and reply

[0197] The user (operator) responds based on the displayed inquiry and enters a reply. For example, they might type, "I will look into this, so could you please wait a moment?" and click the send button.

[0198] 5. Reply translation and notification

[0199] The server then passes the operator's reply back to the real-time translation means, which translates it from the operator's language into the user's language. For example, the Japanese phrase "We will investigate, so please wait a moment." is translated into English as "We will investigate, please wait a moment." The emotion analysis means then checks for neutral expressions. The emotion engine monitors the user's emotional state in real time, neutralizes the emotional content, and sends it to the user's device.

[0200] 6. Use of Emotion Engines

[0201] The emotion engine recognizes the user's emotional state and suggests how the operator should respond based on that. For example, if the emotion engine recognizes that the user is angry, it will warn the operator that "the customer is angry, so a polite response is required." In addition, based on emotion analysis, the frequency and patterns of emotional complaints are analyzed and data is accumulated. This data is used for future service improvements and training.

[0202] Specific examples

[0203] Example 1: When a user makes an inquiry in Japanese

[0204] The user makes an inquiry in Japanese, saying, "The product delivery is delayed. What's going on?" The device sends this inquiry to the server, which uses real-time translation to translate it into English as, "The product delivery is delayed, what is happening?" The emotion analysis device analyzes this content and neutralizes the emotional expression, "It's delayed, what's going on?" The neutralized message, "The product delivery is delayed, please advise [current status]," is displayed on the operator's device. The emotion engine recognizes the user's emotions and alerts the operator.

[0205] The operator replies, "We will investigate, so please wait a moment," and sends the reply from the device to the server. The server translates this reply from Japanese to English and sends it to the user's device as "We will investigate, please wait a moment." The emotion engine then rechecks the emotion of the reply, confirming that it is neutral before sending it.

[0206] Example 2: When a customer makes a complaint in English

[0207] If a user complains in English, "I'm extremely upset with your service. My order was not delivered on time," the device sends this content to the server, which then uses a translation tool to translate it into Japanese as "I'm extremely upset with your service. My order was not delivered on time." The emotion analysis tool neutralizes the strong expression "dissatisfied" and displays to the operator, "A customer has inquired about their dissatisfaction with the service and a delayed order." The emotion engine recognizes the user's strong anger and alerts the operator, "The customer is extremely dissatisfied. Please handle this as a priority."

[0208] The operator replies, "We apologize for the inconvenience, we will check immediately," and the server translates this into English and sends it to the user's device as, "We apologize for the inconvenience, we will check immediately." The emotion engine then rechecks the emotion of the reply, making sure it is neutral before sending it.

[0209] The system of the present invention realizes multilingual support through real-time translation, emotion analysis, and an emotion engine, providing a stress-free environment for both operators and users. The emotion engine enables more advanced emotion recognition and suggests ways to respond, improving service quality.

[0210] The processing flow will be explained below.

[0211] Step 1:

[0212] A user sends a support request from their own device. This request contains the inquiry content and the user's language information. For example, a user enters a message in Japanese saying, "My product is late. What's going on?" and clicks the send button.

[0213] Step 2:

[0214] The terminal receives the user's input information and sends the query content and the user's language information to the server. The terminal transfers the text content and language information to the server in packet format.

[0215] Step 3:

[0216] The server passes the received inquiry as a request to the AI ​​translation engine. The AI ​​translation engine translates the user's language into a language the operator can understand. For example, the Japanese phrase "The product delivery is delayed, what's happening?" is translated into English as "The product delivery is delayed, what is happening?"

[0217] Step 4:

[0218] The server sends the translation results to an AI sentiment analysis engine, which analyzes the translated query, detects emotional expressions, and converts them into neutral expressions. For example, it neutralizes the emotional expression "what is happening?" to "please advise [current status]."

[0219] Step 5:

[0220] The server sends the translated and neutralized query content to the operator terminal, which checks the displayed content and starts responding to the user based on it.

[0221] Step 6:

[0222] The emotion engine analyzes the emotional state of the received text and monitors the user's emotional state in real time. The emotion engine determines whether the user is angry or dissatisfied, and if strong emotions are detected, a warning message such as "The customer is angry, so polite response is required" is displayed on the operator terminal.

[0223] Step 7:

[0224] The user (operator) responds based on the displayed inquiry and enters a reply. For example, they might type, "I will look into this, so could you please wait a moment?" and click the send button.

[0225] Step 8:

[0226] The terminal sends the operator's input to the server. The server then passes the reply back to the AI ​​translation engine, which translates it from the operator's language into the user's language. For example, the Japanese phrase "We will investigate, so could you please wait a moment?" is translated into English as "We will investigate, please wait a moment."

[0227] Step 9:

[0228] The server then sends the translated reply back to the AI ​​sentiment analysis engine to neutralize the emotional content. The sentiment analysis engine analyzes the reply and neutralizes it as necessary. Once this process is complete, the server sends the final translated and neutralized reply to the user's device.

[0229] Step 10:

[0230] The terminal (user) displays the translated reply received from the server on its screen and notifies the user. The user confirms the reply displayed on the screen and completes the inquiry.

[0231] Through the above steps, a system is realized in which communication with users is carried out in multiple languages ​​and in an emotion-neutral manner, allowing operators to respond quickly and effectively.

[0232] Example 2

[0233] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0234] To minimize communication delays and misunderstandings caused by language barriers and emotional complaint handling in global call centers, and to achieve effective and neutral communication between users and operators.

[0235] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for accepting inquiry content and translating the text in real time, means for sentiment-analyzing the translated text and neutralizing emotional expressions, means for displaying the translated and neutralized inquiry content on an operator terminal, means for translating a reply entered by the operator in real time, sentiment-analyzing and neutralizing the reply, and sending it to the user's terminal, means for recognizing, analyzing, and monitoring the user's emotional state and suggesting a response method to the operator, and means for accumulating data on the inquiry content and the reply content and storing it for analysis. This minimizes communication delays and misunderstandings due to language barriers and emotional responses to complaints, enabling effective and neutral communication between users and operators.

[0236] The "means for receiving inquiry content" is a function for electronically receiving inquiry content from a user and transferring that information to the server.

[0237] "Means for translating text in real time" refers to the ability to instantly convert received text into another language.

[0238] "Means for sentiment analysis of translated text and neutralizing emotional expressions" is a function that analyzes translated text to detect emotional elements and converts them into neutral expressions.

[0239] The "means for displaying the translated and neutralized inquiry content on the operator terminal" is a function for visually displaying the processed inquiry content on the operator terminal.

[0240] "Means for translating replies entered by operators in real time, analyzing and neutralizing emotions, and sending them to the user's device" refers to a function that instantly translates the operator's response into another language, neutralizes emotional elements, and then sends it to the user.

[0241] "Means for recognizing, analyzing, and monitoring the user's emotional state and suggesting appropriate ways to respond to the operator" refers to a function that detects and analyzes the user's emotions and suggests appropriate ways to respond to the operator based on that information.

[0242] "Means for accumulating data on inquiry and reply content and storing it for analysis" refers to a function that stores the history of all inquiries and replies in a database so that it can be used later for analysis and improvement.

[0243] This invention relates to a system for effectively handling multilingual and emotional complaints in global call centers. This system includes a real-time translation means, a sentiment analysis means, and an emotion engine, and automates the entire process from processing inquiries to replying.

[0244] Hardware and software used

[0245] The main components of the system are the server, the terminal, and the user (operator and user). This system uses the following hardware and software:

[0246] Real-time translation means: As an example, we will use Google Translate API, which is a software function for instantly translating inquiries and replies into different languages.

[0247] Sentiment analysis tools: As an example, we will use IBM Watson Emotion Analysis, which is a function for analyzing and neutralizing the emotional components of received text.

[0248] Emotion engine: Using the Affectiva SDK as an example, this is an engine that monitors the user's emotional state in real time and suggests ways to respond to the operator based on the analysis results.

[0249] The entire process

[0250] 1. Receiving inquiries

[0251] The user (consumer) inputs the inquiry content from their own device and sends it. For example, a user might input "The arrival of the product is delayed. What's going on?" in Japanese and click the send button.

[0252] 2. Translation and Sentiment Analysis

[0253] The server translates the received query in real time using the Google Translate API, while neutralizing emotional expressions using IBM Watson Emotion Analysis.

[0254] 3. Notification to Operator

[0255] The translated and neutralized inquiry is sent from the server to the operator's terminal, where the translated inquiry is displayed and the operator can respond appropriately.

[0256] 4. Operator response and reply

[0257] The operator types a reply based on the displayed inquiry and sends it to the server. For example, they might type, "I'll look into it, so could you please wait a moment?"

[0258] 5. Reply translation and notification

[0259] The server then translates the operator's reply using the Google Translate API, adapting it to the user's language, and uses sentiment analysis tools to recheck and neutralize the emotional content of the reply.

[0260] 6. Use of Emotion Engines

[0261] The emotion engine accumulates the contents of inquiries and replies, suggests appropriate responses to operators, and analyzes the user's emotional state and issues warnings to operators based on the analysis results.

[0262] Specific examples

[0263] Example 1: Inquiry in Japanese

[0264] When a user makes an inquiry in Japanese, such as "The product delivery is delayed, what's going on?", the device sends this message to the server, which uses the Google Translate API to translate it into English as "The product delivery is delayed, what is happening?" IBM Watson Emotion Analysis then neutralizes the emotional expression, saying "The product delivery is delayed, please advise [current status]." The operator considers how to respond based on this message and replies, "We will investigate, so could you please wait a moment?" This reply is then translated again by the server into English as "We will investigate, please wait a moment," and sent to the user.

[0265] Example 2: Complaint in English

[0266] If a user complains in English, saying, "I'm extremely upset with your service. My order was not delivered on time," the device sends this content to the server, which then uses the Google Translate API to translate it into Japanese as, "I'm extremely unhappy with your service. My order was not delivered on time." The emotion analysis means neutralizes strong emotional expressions and notifies the operator, "A customer has inquired about their dissatisfaction with the service and a delayed order." The emotion engine analyzes the user's emotional state and alerts the operator, saying, "The customer is extremely unhappy. Please prioritize their response."

[0267] Example prompts for generative AI models

[0268] Below are some example prompts that explain the operation of programs on this system:

[0269] Please describe in detail the processing steps of your system for multilingual support and emotional complaint handling using specific hardware and software. Include specific behavior for each processing step. For example, please include the processing for when a customer inquires in Japanese, "My item is late. What's going on?", and when a customer in English complains, "I'm extremely upset with your service. My order was not delivered on time."

[0270] The system of the present invention provides an efficient and stress-free environment for both users and operators by centrally utilizing real-time translation, emotion analysis, and an emotion engine. Utilizing the emotion engine enables more advanced emotion recognition and the proposal of response methods, thereby achieving improved service quality.

[0271] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0272] Step 1:

[0273] A user sends a query from their own terminal. Specifically, they enter the query content in their own language and click the send button. At this time, the terminal sends the query content and the user's language information to the server. The input is the query text entered by the user, and the output is an HTTP request sent to the server.

[0274] Step 2:

[0275] The server passes the received inquiry to a real-time translation tool and translates it into the specified language. Specifically, it uses the Google Translate API to translate the received Japanese text into English. The input is the user's inquiry text, and the output is the translated text. At this stage, data processing involves converting the text into a different language.

[0276] Step 3:

[0277] The server passes the translated text to a sentiment analyzer to detect and neutralize emotional expressions. Specifically, it uses IBM Watson Emotion Analysis to convert strong emotional expressions into neutral ones. The input is the translated text, and the output is text with neutralized emotional elements. At this stage, the data operations are sentiment analysis and neutralization.

[0278] Step 4:

[0279] The server sends the neutralized query content to the operator's terminal. Specifically, the translated and neutralized query content is displayed on the operator's terminal. The input is the neutralized text, and the output is the content displayed on the operator's terminal. The server sends the data as an HTTP response, and the operator's terminal receives and displays it.

[0280] Step 5:

[0281] The user (operator) responds based on the displayed inquiry. Specifically, the operator enters a reply and presses the send button, which sends the reply from the terminal to the server. The input is the reply entered by the operator, and the output is an HTTP request sent to the server.

[0282] Step 6:

[0283] The server passes the operator's reply to a real-time translation tool, which translates it into the user's language. Specifically, it uses the Google Translate API to translate replies entered in Japanese into English. The input is the operator's reply text, and the output is the translated text. Data processing at this stage involves another language conversion.

[0284] Step 7:

[0285] The server then rechecks the translated reply content using sentiment analysis and neutralizes it. Specifically, it uses IBM Watson Emotion Analysis to check for emotional expressions in the reply content and neutralize it. The input is the translated reply text, and the output is the neutralized reply text. The data operations at this stage are sentiment analysis and neutralization.

[0286] Step 8:

[0287] The server sends the neutralized reply content to the user's terminal. The server sends the reply content with the emotional elements removed to the user's terminal as an HTTP response. The input is the neutralized reply text, and the output is the content displayed on the user's terminal.

[0288] Step 9:

[0289] The emotion engine monitors the user's emotional state and proposes the analysis results to the operator. Specifically, it uses the Affectiva SDK to detect the user's emotional state in real time and proposes a response. The input is the inquiry and reply, and the output is a response proposal to the operator.

[0290] Step 10:

[0291] The server accumulates data on inquiries and replies and stores it for analysis. The server stores all communication history in a database for later analysis and service improvement. The input is the inquiry and reply data, and the output is the accumulated database.

[0292] (Application example 2)

[0293] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0294] Conventional call centers and user support systems have limited multilingual support and emotion analysis capabilities, making it difficult to provide fast and effective global customer support. Furthermore, they lack the ability to properly recognize users' emotions and propose neutral responses, which can easily cause stress for both operators and users. If these issues continue for a long period of time, they could lead to a decline in service quality and customer satisfaction.

[0295] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0296] In this invention, the server includes knowledge processing means for translating text in real time, sentiment analysis means for performing sentiment analysis and neutralizing sentiment from the translated text, means for displaying on an operation device the inquiry content processed by the translation means and the sentiment analysis means, means for translating a reply entered on the operation device into the language of the terminal by the translation means and the sentiment analysis means and transmitting the neutralized text to the terminal device, means for monitoring the emotional state of the user in real time using a sentiment analysis function and proposing a response method, and means for quickly and effectively processing user feedback based on the response method. This enables advanced user support that combines multilingual real-time translation and sentiment analysis.

[0297] The "knowledge processing means for real-time text translation" is a function for instantly converting the content of a user's inquiry into another language.

[0298] The "sentiment analysis means for performing sentiment analysis and neutralizing emotions from translated text" is a function for analyzing emotional expressions in translated text and converting them into neutral expressions in order to reduce the intensity of the emotions.

[0299] The "means for displaying on an operating device" is a function for displaying the translated and sentiment-analyzed inquiry content on a device operated by an operator.

[0300] The "means for transmitting to the terminal device" is a function that translates and neutralizes the emotion again in order to transmit the reply entered by the operator to the user's terminal.

[0301] "Means of monitoring the user's emotional state in real time using an emotion analysis function and suggesting ways to respond" is a function that detects the user's emotions in real time and suggests the most appropriate way to respond to the operator based on that emotional state.

[0302] "Means for quickly and effectively processing feedback" refers to a function that instantly translates and analyzes sentiment on user feedback and quickly responds appropriately.

[0303] The present invention relates to a system for handling complaints in multiple languages ​​and emotionally. This system has the following specific configuration and operation.

[0304] System configuration

[0305] 1. Server:

[0306] The server is equipped with a knowledge processing means for translating text in real time, a sentiment analysis means for performing sentiment analysis to neutralize emotions from the translated text, and a sentiment analysis function. This server receives, processes, and transmits inquiries from users.

[0307] 2. Terminal:

[0308] The terminal consists of a device used by the user (e.g., a smartphone or smart glasses) and an operating device used by the operator. The terminal displays the translated and sentiment-analyzed content from the server and receives input from the operator.

[0309] 3. User:

[0310] The users include the users who make inquiries and the operators who respond to those inquiries.

[0311] Program processing

[0312] 1. Receiving inquiries

[0313] A user sends a query from their own device to the server, which includes the query content and the user's language information.

[0314] 2. Translation and Sentiment Analysis

[0315] The server translates the received query using a knowledge processing means that translates in real time.Then, a sentiment analysis means analyzes the translated content and neutralizes emotional expressions.

[0316] 3. Notification to operator and response

[0317] The processed inquiry is sent from the server to the operator's operating device. The operator responds based on the displayed information. The reply entered by the operator is translated and neutralized again before being sent to the user's device.

[0318] Hardware and software used

[0319] The system of the present invention mainly uses the following hardware and software.

[0320] Hardware: Server, smartphone, smart glasses, operator control device

[0321] Software: Python, Translators API, TextBlob

[0322] Data processing and calculation

[0323] 1. Real-time translation

[0324] The server translates the received text in real time using the Translators API.

[0325] 2. Sentiment analysis

[0326] Analyze the sentiment of text using TextBlob and neutralize the emotional expressions.

[0327] 3. Monitoring your emotional state

[0328] The server uses emotion analysis to monitor the user's emotional state in real time and present the operator with the most appropriate response.

[0329] Specific examples

[0330] Example 1: User gives feedback in English

[0331] The user sends feedback in English, saying, "I'm extremely upset with your service. My order was not delivered on time." The server receives this feedback and translates it in real time to "I'm extremely upset. My order was not delivered on time." It then performs sentiment analysis, which parses it as an expression of strong anger. The server then neutralizes the emotion and displays on the operator's control device, "A customer has inquired about their dissatisfaction with the service and a delayed order." The sentiment analysis function also issues a warning, such as, "The customer is extremely dissatisfied. Please prioritize their response." The operator enters an appropriate response, which the server translates and sends to the user's device, "We apologize for the inconvenience, we will check immediately."

[0332] Example of input prompt for generative AI model

[0333] "I'm extremely upset with your service. My order was not delivered on time."

[0334] Prompt: "Translate the above English sentence into Japanese and perform sentiment analysis to neutralize the emotional content. Then, suggest how to respond to the operator."

[0335] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0336] Step 1:

[0337] This is the action in which a user (consumer) inputs the content of a query from their own device and sends it to the server. The input is the user's query and its language information, and the output is the raw query data received by the server. The server then performs preliminary preparations to translate this query in real time. Specifically, the query and language information are saved in a database and used in the next step.

[0338] Step 2:

[0339] This is the operation in which the server translates the input inquiry content in real time. The input is the inquiry content and language information received in step 1, and the output is the translated text. The server calls the Translators API to translate from the original language into a common language (such as English) that the operator can understand. Specifically, the Translators API is used to translate the English sentence, "I'm extremely upset with your service. My order was not delivered on time." into Japanese.

[0340] Step 3:

[0341] The server performs sentiment analysis on the translated text and neutralizes emotional expressions as necessary. The input is the translated text obtained in step 2, and the output is the neutralized text. The server uses TextBlob to perform sentiment analysis and detect emotional expressions. For example, it converts a sentence that expresses strong anger into a neutral expression such as "We apologize for the inconvenience."

[0342] Step 4:

[0343] This is the operation in which the server notifies the operator based on emotion analysis. The input is the neutralized text obtained in step 3, and the output is a notification message displayed on the operator's operating device. The server uses its emotion analysis function to evaluate the user's emotional state and sends a notification to the operator, such as "The customer is very dissatisfied. Please handle this as a priority."

[0344] Step 5:

[0345] This is the operation in which the operator inputs a response on the operating device and sends the content to the server. The input is the text of the operator's response, and the output is the response data from the operator received by the server. The operator inputs an appropriate response based on the neutralized inquiry content and clicks the send button.

[0346] Step 6:

[0347] The server then translates the operator's response again and performs sentiment analysis. The input is the operator's response text received in step 5, and the output is the translated and neutralized reply text. The server again uses the Translators API and TextBlob to translate the operator's reply into the user's language and neutralize it to avoid emotional overload.

[0348] Step 7:

[0349] This is the operation in which the server finally sends the translated and sentiment-analyzed reply text to the user's device. The input is the neutralized reply text obtained in step 6, and the output is the reply message displayed on the user's device. The server sends the translated and neutralized text to the user's device, providing a response in a way that is less stressful for the user.

[0350] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0351] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0352] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0353] [Second embodiment]

[0354] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0355] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0356] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0357] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0358] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0359] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0360] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0361] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0362] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0363] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0364] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0365] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0366] The present invention relates to a system for multilingual and emotional complaint handling, which includes a real-time translation means and a sentiment analysis means, enabling fast and effective responses in global call centers.

[0367] Program processing

[0368] The main components of this system are the server, terminals, and users (operators and users). The roles and operations of each are explained below.

[0369] 1. Receiving inquiries

[0370] The user sends a support request from their device, which includes the content of the inquiry and the language they are using. The device then sends this information to the server.

[0371] 2. Translation and Sentiment Analysis

[0372] The server then passes the received inquiry to a real-time translation unit, which translates the user's language into a language the operator can understand. The emotion analysis unit then analyzes the translated content and converts emotional expressions into neutral ones, allowing the operator to respond calmly based on the information.

[0373] 3. Notification to Operator

[0374] The translated and neutralized query content is sent from the server to the operator's terminal, where it is displayed in an easy-to-understand format.

[0375] 4. Operator response and reply

[0376] The user (operator) responds based on the displayed content and enters a reply, which is then sent from the terminal to the server.

[0377] 5. Reply translation and notification

[0378] The server then passes the operator's reply to the real-time translation tool, which translates it into the user's language. The sentiment analysis tool then converts the translated reply into a neutral expression and sends the final answer to the user's device, ensuring the user receives a calm and clear response.

[0379] Specific examples

[0380] Example 1: When a user makes an inquiry in Japanese

[0381] The user makes an inquiry in Japanese, saying, "The product delivery is delayed. What's going on?" The terminal sends this inquiry to the server, which uses real-time translation means to translate it into English as, "The product delivery is delayed, what is happening?" The emotion analysis means then analyzes this content and neutralizes the emotional expression, "It's delayed, what's going on?" The operator's terminal then displays the neutralized message, "The product delivery is delayed, please advise [current status]."

[0382] The operator replies, "We will investigate, please wait a moment," and sends the reply from the device to the server. The server translates this reply from Japanese to English, saying, "We will investigate, please wait a moment," and then uses sentiment analysis to confirm that it is neutral. The final reply is sent to the user's device.

[0383] Example 2: When a customer makes a complaint in English

[0384] If a user complains in English, "I'm extremely upset with your service. My order was not delivered on time," the device sends this content to the server, which then uses a translation tool to translate it into Japanese as "I'm extremely upset with your service. My order was not delivered on time." The sentiment analysis tool neutralizes the strong expression "I'm dissatisfied," and displays to the operator, "A customer has inquired about their dissatisfaction with the service and the delay in their order."

[0385] The operator replies, "We apologize for the inconvenience, we will check immediately." The server then translates this into English and sends it to the user's device as, "We apologize for the inconvenience, we will check immediately."

[0386] The system of the present invention achieves multilingual support through real-time translation and sentiment analysis, providing a stress-free environment for both operators and users.

[0387] The processing flow will be explained below.

[0388] Step 1:

[0389] A user sends a support request from their own device. This request contains the inquiry content and the user's language information. For example, a user enters a message in Japanese saying, "My product is late. What's going on?" and clicks the send button.

[0390] Step 2:

[0391] The terminal receives the user's input information, sends the query content and the user's language information to the server, packages the user's language and text content, and makes a request to the server.

[0392] Step 3:

[0393] The server passes the received inquiry to an AI translation engine, which translates the user's language into a language the operator can understand. For example, the Japanese phrase "The product delivery is delayed, what's happening?" is translated into English as "The product delivery is delayed, what is happening?"

[0394] Step 4:

[0395] The server then sends the translation results to an AI sentiment analysis engine, which analyzes the translated query, detects any emotional expressions, and converts them into neutral phrases. For example, it neutralizes the emotional expression "what is happening?" to "please advise [current status]."

[0396] Step 5:

[0397] The server sends the translated and neutralized inquiry to the operator terminal. The operator terminal displays the translated and neutralized inquiry. The operator is shown a message saying, "A customer has inquired about a delay in the delivery of their product."

[0398] Step 6:

[0399] The user (operator) responds based on the displayed inquiry and enters a reply. For example, they might type, "I will look into this, so could you please wait a moment?" and click the send button.

[0400] Step 7:

[0401] The terminal sends the operator's input to the server and requests the operator's reply text from the server.

[0402] Step 8:

[0403] The server then passes the operator's reply to the AI ​​translation engine again, which translates it from the operator's language to the user's language. For example, the Japanese phrase "We will investigate, so please wait a moment." is translated into English as "We will investigate, please wait a moment."

[0404] Step 9:

[0405] The server then sends the translated reply back to the AI ​​sentiment analysis engine to neutralize the emotional content. If the content is already neutral, no changes are made.

[0406] Step 10:

[0407] The server then sends the translated and neutralized reply to the user's device. The server then sends the translated reply, "We'll look into this, so could you please wait a moment?"

[0408] Step 11:

[0409] The terminal (user) displays the translated reply received from the server on its screen. The user's screen displays "We will look into this, could you please wait a moment?", and the response is complete.

[0410] The above steps constitute a system that enables multilingual and emotion-neutral communication between users and operators.

[0411] Example 1

[0412] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0413] Today's global call centers are required to handle inquiries from users quickly and accurately. However, due to issues such as diverse languages ​​and emotional expressions, it is often difficult for operators to respond efficiently. It is also not easy to respond calmly and neutrally to emotional complaints. This increases the burden on operators and risks a decline in response quality. There is a need for a system that can solve these issues, improve the efficiency of inquiry responses at global call centers, and improve response quality.

[0414] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0415] In this invention, the server includes means for receiving inquiries from user terminals, AI translation means for translating text in real time, AI sentiment analysis means for performing sentiment analysis and neutralizing the sentiment of the translated text, means for displaying the inquiry content processed by the translation means and the sentiment analysis means on an operator terminal, and means for translating a reply entered by an operator into the user's language by the translation means and the sentiment analysis means and sending the neutralized text to the user terminal. This enables inquiries to be processed quickly and accurately in a global call center, providing a stress-free environment for both operators and users.

[0416] "User" refers to an individual or company that uses the system to make an inquiry.

[0417] A "terminal" is a device (e.g., PC, smartphone, tablet) used by a user or operator to access the system.

[0418] A "server" is a central computer that manages the processing of the entire system and sends and receives various data.

[0419] "AI translation mechanism" refers to a mechanism that utilizes artificial intelligence technology to translate text into another language in real time.

[0420] An "AI sentiment analysis tool" is a mechanism that uses artificial intelligence technology to analyze and neutralize emotional expressions in text.

[0421] "Inquiry content" refers to the text information of questions or complaints sent by users through the system.

[0422] An "operator" is a system user who responds to inquiries from users and provides necessary replies.

[0423] A "neutralized text" is a text that has been transformed into a calm and objective expression, with emotional expressions suppressed.

[0424] This invention is a system that realizes multilingual support and emotional complaint handling in global call centers. This system is mainly composed of three components: a server, terminals, and users (operators and customers).

[0425] System Overview

[0426] Receiving inquiries

[0427] A user sends a support request from their device (PC, smartphone, etc.). This request includes the inquiry and the language used by the user. The device then sends this information to the server.

[0428] Translation and Sentiment Analysis

[0429] The server passes the received inquiry to a real-time translation tool (e.g., Google Translate API) to translate the user's language into a language the operator can understand. The translated content is then analyzed using a sentiment analysis tool (e.g., IBM Watson Natural Language Understanding) to convert emotional expressions into neutral ones, allowing the operator to respond calmly based on the information.

[0430] Operator notification

[0431] The translated and neutralized inquiry is sent from the server to the operator's terminal, where the inquiry is displayed in an easy-to-understand format through dedicated customer support software.

[0432] Operator response and reply

[0433] The user (operator) responds based on the displayed information and enters a reply, which is then sent from the terminal to the server.

[0434] Reply translation and notification

[0435] The server then passes the operator's reply to the real-time translation unit, which translates it into the user's language. The sentiment analysis unit then converts the translated reply into a neutral expression, and the final answer is sent to the user's device. This allows the user to receive a calm and clear response.

[0436] Specific examples

[0437] Example 1: When a user makes an inquiry in Japanese

[0438] The user makes an inquiry in Japanese, saying, "The product delivery is delayed. What's going on?" The device sends this inquiry to the server. The server uses real-time translation means to translate this into English as, "The product delivery is delayed, what is happening?" The sentiment analysis means then analyzes this content and displays a neutralized response, "The product delivery is delayed, please advise [current status]." on the operator's device. The operator replies, "We will investigate, could you please wait a moment?" and sends this response from the device to the server. The server then translates this response from Japanese to English, saying, "We will investigate, please wait a moment." The sentiment analysis means then confirms the neutral expression, and the final reply is sent to the user's device.

[0439] Example 2: When a customer makes a complaint in English

[0440] A user complains in English, saying, "I'm extremely upset with your service. My order was not delivered on time." The device sends this content to the server. The server uses a translation tool to translate this into Japanese as "I'm extremely unhappy with your service. My order was not delivered on time." The sentiment analysis tool neutralizes the strong expression "unhappy" and displays to the operator, "A customer has inquired about their dissatisfaction with the service and a delayed order." The operator replies, "We apologize, we will check immediately." The server translates this content into English and sends it to the user's device as "We apologize for the inconvenience, we will check immediately."

[0441] Prompt Sentence Examples

[0442] Here are some examples of prompts for generative AI models:

[0443] "Please explain a scenario in which a user makes an inquiry in Japanese and an operator responds. Please also include the specific process flow and the functions used (translation methods, sentiment analysis methods)."

[0444] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0445] Step 1:

[0446] A user sends a support request using their own device. As input, they enter text data containing the inquiry and the user's language information into the device. The device then sends this data to the server as an HTTP request. The output is the inquiry data transferred to the server.

[0447] Step 2:

[0448] The server passes the received query data to a real-time translation method (e.g., Google Translate API). The input is the query text sent by the user and its language information. The server passes this information to the API for translation processing. The output is the translated text returned to the server.

[0449] Step 3:

[0450] The server passes the translated text to a sentiment analyzer (e.g., IBM Watson Natural Language Understanding). The input is the translated query text. The server sends the text to a sentiment analysis API, which analyzes the sentiment in the text. The output is text with neutralized emotional expressions.

[0451] Step 4:

[0452] The server sends the neutralized text to the operator's terminal. The input is the query data processed by the sentiment analysis means. The server transfers this data to the operator's terminal using REST API or WebSocket. The output is the neutralized query content displayed on the operator's terminal.

[0453] Step 5:

[0454] The user (operator) considers how to respond based on the displayed content and enters a reply. As input, the operator enters the proposed response as text into the terminal. The terminal then sends this reply data back to the server. The output is the reply data transferred to the server.

[0455] Step 6:

[0456] The server passes the operator's reply to a real-time translation means. The input is the reply text sent by the operator. The server sends the reply text to a translation means, which translates the reply into the user's language. The output is the translated reply text.

[0457] Step 7:

[0458] The server passes the translated reply to the sentiment analyzer again to confirm the neutralized expression. The input is the translated reply text. The server passes this text to the sentiment analysis API for verification. The output is the reply text with the neutralized expression confirmed.

[0459] Step 8:

[0460] The server sends the final answer to the user's device. The input is the reply text that has been checked for neutrality. The server transfers this text to the user's device using a REST API or WebSocket. The output is the final reply that is displayed on the user's device.

[0461] (Application example 1)

[0462] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0463] Conventional customer service systems have difficulty dealing with customers who speak different languages ​​and have different cultural backgrounds, and have had particular problems with incomplete multilingual support and emotional complaint handling. Furthermore, in brick-and-mortar stores, staff were not equipped with an environment where they could instantly provide multilingual support, making it difficult to provide prompt and appropriate service.

[0464] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0465] In this invention, the server includes an artificial intelligence translation means for translating text in real time, an artificial intelligence emotion analysis means for performing emotion analysis and neutralizing the emotion from the translated text, a means for displaying the inquiry content processed by the translation means and the emotion analysis means on a staff terminal, a means for translating a reply entered by a staff member into the user's language by the translation means and the emotion analysis means and sending the neutralized text to the user terminal, and a means for displaying the inquiry content and the reply content on an augmented reality device used as a staff terminal. This enables multilingual and emotional complaint handling to be performed quickly and effectively even in physical stores, thereby improving customer satisfaction.

[0466] "Artificial intelligence translation means for real-time text translation" refers to a device that uses artificial intelligence to instantly convert the content of an inquiry into a different language.

[0467] The "artificial intelligence emotion analysis means for performing emotion analysis and neutralizing emotions from translated text" is a device that uses artificial intelligence to analyze emotional expressions in translated text and convert them into calm expressions.

[0468] "Staff terminal" means an electronic device used by customer service staff to receive and enter information.

[0469] "User terminal" refers to an electronic device used by a service user to receive support or enter inquiries.

[0470] An "augmented reality device" is a device worn by staff that can overlay virtual information onto the actual environment.

[0471] "Inquiry content" refers to the content of questions, requests, and complaints made by users to customer service.

[0472] A "neutralized text" is text that has been transformed from emotional expression into calm, objective expression.

[0473] A "translation tool" is a device or software for converting text from one language to another.

[0474] A "sentiment analyzer" is a device or software for analyzing and identifying emotional elements in text.

[0475] The present invention is a system for multilingual support and emotional complaint handling in brick-and-mortar stores. This system performs real-time translation and sentiment analysis between customers and customer service staff (hereinafter referred to as staff), enabling staff to respond appropriately and quickly. Specific embodiments are described below.

[0476] Overall system configuration

[0477] 1. Server and Cloud Infrastructure

[0478] The server includes the hardware and software necessary to implement real-time translation and sentiment analysis. The main software used is the Google Translator API, which provides translation functionality, the TextBlob library for sentiment analysis, and the SpeechRecognition library for voice recognition.

[0479] 2. Staff terminals and augmented reality devices

[0480] Smart glasses (e.g., Google Glass) are used as staff terminals, and these devices display inquiries and replies to staff in real time.

[0481] 3. User Device

[0482] A user (customer) makes an inquiry using a personal device such as a smartphone.

[0483] Data processing and calculation

[0484] The server processes and calculates data in the following procedure.

[0485] 1. Voice Recognition

[0486] When a user makes a voice inquiry using their smartphone, the voice data is transmitted to the staff member's smart glasses, which then send the voice data to the server, which then converts the voice data into text using the SpeechRecognition library.

[0487] 2. Real-time translation

[0488] The server translates the text obtained by speech recognition in real time using the Google Translator API. For example, it converts a Japanese inquiry into English.

[0489] 3. Sentiment Analysis and Neutralization

[0490] The server performs sentiment analysis on the translated text using the "TextBlob" library. If the sentiment is strong, it converts certain keywords (e.g., "very dissatisfied") into neutral expressions (e.g., "concerned").

[0491] 4. Staff Display

[0492] The neutralized text is processed on the server and displayed on the staff's smart glasses, allowing them to respond appropriately to the customer.

[0493] 5. Reply Translation and Notification

[0494] The reply entered by the staff is sent back to the server, where it is translated and sentiment-analyzed. The translated text is neutralized and then sent to the customer's smartphone.

[0495] Specific use cases

[0496] If a customer asks in Japanese, "The product delivery is delayed, what's going on?", this speech is converted into text on the server and translated into English. The translated text, "The product delivery is delayed, what is happening?", undergoes sentiment analysis and neutralization (e.g., "The product delivery is delayed, please advise [current status].") and is displayed on the staff member's smart glasses. The staff member responds, "We will investigate, please wait a moment," and the reply is also translated into English and notifies the customer, "We will investigate, please wait a moment."

[0497] Example prompt sentence:

[0498] Analyze the following sentences and convert them into neutral expressions.

[0499] Text: "I'm extremely upset with your service. My order was not delivered on time."

[0500] Neutral: "I'm concerned about the delay in my order delivery. Please advise on the current status."

[0501] As described above, the present invention aims to realize multilingual support and emotional complaint handling, thereby improving the quality of customer service in physical stores.

[0502] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0503] Step 1:

[0504] A user makes a voice inquiry using a smartphone. This voice data is collected through the smartphone's microphone and saved as digital data. The input is voice data, and the output is digital voice data.

[0505] Step 2:

[0506] The user's smartphone sends voice data to the server, which converts it into text using the "SpeechRecognition" library. The input is digital voice data, and the output is text data. This conversion transforms the voice data into concrete text information.

[0507] Step 3:

[0508] The server translates the text data in real time using the Google Translator API. For example, Japanese text is converted into English text. The input is Japanese text data, and the output is English text data. This translation converts the data into a language that the staff can understand.

[0509] Step 4:

[0510] The server performs sentiment analysis on the translated English text data using the "TextBlob" library. The sentiment analysis results in positive, negative, and neutral sentiment. The input is the English text data, and the output is the sentiment analysis results and corrected text data. Sentiment analysis neutralizes expressions as needed.

[0511] Step 5:

[0512] The server sends neutralized text data with reduced emotion to the augmented reality device (smart glasses). The staff member's smart glasses receive this data and display it on their screen. The input is the neutralized text data, and the output is the display on the smart glasses.

[0513] Step 6:

[0514] The staff member checks the neutralized text through the smart glasses and types an appropriate reply to the customer. The reply is recorded as audio by the smart glasses' microphone and sent to the server as text data. The input is audio data and the output is text data.

[0515] Step 7:

[0516] The server again translates the staff member's reply text data into the customer's language (e.g., Japanese) using the "Google Translator API." It then performs sentiment analysis on the translated text using the "TextBlob" library to confirm neutralization. The input is the staff member's English text data, and the output is neutralized Japanese text data.

[0517] Step 8:

[0518] The server sends the neutralized Japanese text data to the user's smartphone. The user's smartphone receives this data and displays it on its screen. The input is the neutralized Japanese text data, and the output is what is displayed on the smartphone's screen.

[0519] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0520] The present invention relates to a system for multilingual and emotional complaint handling, which includes a real-time translation means, a sentiment analysis means, and an emotion engine, enabling fast and effective responses in global call centers.

[0521] Program processing

[0522] The main components of this system are the server, terminals, and users (operators and users). The roles and operations of each are explained below.

[0523] 1. Receiving inquiries

[0524] A user sends a support request from their own device. This request contains the inquiry content and the user's language information. For example, a user enters a message in Japanese saying, "My product is late. What's going on?" and clicks the send button.

[0525] 2. Translation and Sentiment Analysis

[0526] The server passes the received inquiry to a real-time translation means, which translates it from the user's language into a language the operator can understand. For example, the Japanese phrase "The product delivery is delayed, what's going on?" is translated into English as "The product delivery is delayed, what is happening?". Next, the sentiment analysis means analyzes the translated content and converts emotional expressions into neutral expressions. For example, the emotional expression "what is happening?" is neutralized to "please advise [current status]."

[0527] 3. Notification to Operator

[0528] The translated and neutralized inquiry is sent from the server to the operator's terminal. The operator's terminal displays the translated and neutralized inquiry. For example, the operator might see, "A customer has inquired about a delay in the arrival of their product."

[0529] 4. Operator response and reply

[0530] The user (operator) responds based on the displayed inquiry and enters a reply. For example, they might type, "I will look into this, so could you please wait a moment?" and click the send button.

[0531] 5. Reply translation and notification

[0532] The server then passes the operator's reply back to the real-time translation means, which translates it from the operator's language into the user's language. For example, the Japanese phrase "We will investigate, so please wait a moment." is translated into English as "We will investigate, please wait a moment." The emotion analysis means then checks for neutral expressions. The emotion engine monitors the user's emotional state in real time, neutralizes the emotional content, and sends it to the user's device.

[0533] 6. Use of Emotion Engines

[0534] The emotion engine recognizes the user's emotional state and suggests how the operator should respond based on that. For example, if the emotion engine recognizes that the user is angry, it will warn the operator that "the customer is angry, so a polite response is required." In addition, based on emotion analysis, the frequency and patterns of emotional complaints are analyzed and data is accumulated. This data is used for future service improvements and training.

[0535] Specific examples

[0536] Example 1: When a user makes an inquiry in Japanese

[0537] The user makes an inquiry in Japanese, saying, "The product delivery is delayed. What's going on?" The device sends this inquiry to the server, which uses real-time translation to translate it into English as, "The product delivery is delayed, what is happening?" The emotion analysis device analyzes this content and neutralizes the emotional expression, "It's delayed, what's going on?" The neutralized message, "The product delivery is delayed, please advise [current status]," is displayed on the operator's device. The emotion engine recognizes the user's emotions and alerts the operator.

[0538] The operator replies, "We will investigate, so please wait a moment," and sends the reply from the device to the server. The server translates this reply from Japanese to English and sends it to the user's device as "We will investigate, please wait a moment." The emotion engine then rechecks the emotion of the reply, confirming that it is neutral before sending it.

[0539] Example 2: When a customer makes a complaint in English

[0540] If a user complains in English, "I'm extremely upset with your service. My order was not delivered on time," the device sends this content to the server, which then uses a translation tool to translate it into Japanese as "I'm extremely upset with your service. My order was not delivered on time." The emotion analysis tool neutralizes the strong expression "dissatisfied" and displays to the operator, "A customer has inquired about their dissatisfaction with the service and a delayed order." The emotion engine recognizes the user's strong anger and alerts the operator, "The customer is extremely dissatisfied. Please handle this as a priority."

[0541] The operator replies, "We apologize for the inconvenience, we will check immediately," and the server translates this into English and sends it to the user's device as, "We apologize for the inconvenience, we will check immediately." The emotion engine then rechecks the emotion of the reply, making sure it is neutral before sending it.

[0542] The system of the present invention realizes multilingual support through real-time translation, emotion analysis, and an emotion engine, providing a stress-free environment for both operators and users. The emotion engine enables more advanced emotion recognition and suggests ways to respond, improving service quality.

[0543] The processing flow will be explained below.

[0544] Step 1:

[0545] A user sends a support request from their own device. This request contains the inquiry content and the user's language information. For example, a user enters a message in Japanese saying, "My product is late. What's going on?" and clicks the send button.

[0546] Step 2:

[0547] The terminal receives the user's input information and sends the query content and the user's language information to the server. The terminal transfers the text content and language information to the server in packet format.

[0548] Step 3:

[0549] The server passes the received inquiry as a request to the AI ​​translation engine. The AI ​​translation engine translates the user's language into a language the operator can understand. For example, the Japanese phrase "The product delivery is delayed, what's happening?" is translated into English as "The product delivery is delayed, what is happening?"

[0550] Step 4:

[0551] The server sends the translation results to an AI sentiment analysis engine, which analyzes the translated query, detects emotional expressions, and converts them into neutral expressions. For example, it neutralizes the emotional expression "what is happening?" to "please advise [current status]."

[0552] Step 5:

[0553] The server sends the translated and neutralized query content to the operator terminal, which checks the displayed content and starts responding to the user based on it.

[0554] Step 6:

[0555] The emotion engine analyzes the emotional state of the received text and monitors the user's emotional state in real time. The emotion engine determines whether the user is angry or dissatisfied, and if strong emotions are detected, a warning message such as "The customer is angry, so polite response is required" is displayed on the operator terminal.

[0556] Step 7:

[0557] The user (operator) responds based on the displayed inquiry and enters a reply. For example, they might type, "I will look into this, so could you please wait a moment?" and click the send button.

[0558] Step 8:

[0559] The terminal sends the operator's input to the server. The server then passes the reply back to the AI ​​translation engine, which translates it from the operator's language into the user's language. For example, the Japanese phrase "We will investigate, so could you please wait a moment?" is translated into English as "We will investigate, please wait a moment."

[0560] Step 9:

[0561] The server then sends the translated reply back to the AI ​​sentiment analysis engine to neutralize the emotional content. The sentiment analysis engine analyzes the reply and neutralizes it as necessary. Once this process is complete, the server sends the final translated and neutralized reply to the user's device.

[0562] Step 10:

[0563] The terminal (user) displays the translated reply received from the server on its screen and notifies the user. The user confirms the reply displayed on the screen and completes the inquiry.

[0564] Through the above steps, a system is realized in which communication with users is carried out in multiple languages ​​and in an emotion-neutral manner, allowing operators to respond quickly and effectively.

[0565] Example 2

[0566] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0567] To minimize communication delays and misunderstandings caused by language barriers and emotional complaint handling in global call centers, and to achieve effective and neutral communication between users and operators.

[0568] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for accepting inquiry content and translating the text in real time, means for sentiment-analyzing the translated text and neutralizing emotional expressions, means for displaying the translated and neutralized inquiry content on an operator terminal, means for translating a reply entered by the operator in real time, sentiment-analyzing and neutralizing the reply, and sending it to the user's terminal, means for recognizing, analyzing, and monitoring the user's emotional state and suggesting a response method to the operator, and means for accumulating data on the inquiry content and the reply content and storing it for analysis. This minimizes communication delays and misunderstandings due to language barriers and emotional responses to complaints, enabling effective and neutral communication between users and operators.

[0569] The "means for receiving inquiry content" is a function for electronically receiving inquiry content from a user and transferring that information to the server.

[0570] "Means for translating text in real time" refers to the ability to instantly convert received text into another language.

[0571] "Means for sentiment analysis of translated text and neutralizing emotional expressions" is a function that analyzes translated text to detect emotional elements and converts them into neutral expressions.

[0572] The "means for displaying the translated and neutralized inquiry content on the operator terminal" is a function for visually displaying the processed inquiry content on the operator terminal.

[0573] "Means for translating replies entered by operators in real time, analyzing and neutralizing emotions, and sending them to the user's device" refers to a function that instantly translates the operator's response into another language, neutralizes emotional elements, and then sends it to the user.

[0574] "Means for recognizing, analyzing, and monitoring the user's emotional state and suggesting appropriate ways to respond to the operator" refers to a function that detects and analyzes the user's emotions and suggests appropriate ways to respond to the operator based on that information.

[0575] "Means for accumulating data on inquiry and reply content and storing it for analysis" refers to a function that stores the history of all inquiries and replies in a database so that it can be used later for analysis and improvement.

[0576] This invention relates to a system for effectively handling multilingual and emotional complaints in global call centers. This system includes a real-time translation means, a sentiment analysis means, and an emotion engine, and automates the entire process from processing inquiries to replying.

[0577] Hardware and software used

[0578] The main components of the system are the server, the terminal, and the user (operator and user). This system uses the following hardware and software:

[0579] Real-time translation means: As an example, we will use Google Translate API, which is a software function for instantly translating inquiries and replies into different languages.

[0580] Sentiment analysis tools: As an example, we will use IBM Watson Emotion Analysis, which is a function for analyzing and neutralizing the emotional components of received text.

[0581] Emotion engine: Using the Affectiva SDK as an example, this is an engine that monitors the user's emotional state in real time and suggests ways to respond to the operator based on the analysis results.

[0582] The entire process

[0583] 1. Receiving inquiries

[0584] The user (consumer) inputs the inquiry content from their own device and sends it. For example, a user might input "The arrival of the product is delayed. What's going on?" in Japanese and click the send button.

[0585] 2. Translation and Sentiment Analysis

[0586] The server translates the received query in real time using the Google Translate API, while neutralizing emotional expressions using IBM Watson Emotion Analysis.

[0587] 3. Notification to Operator

[0588] The translated and neutralized inquiry is sent from the server to the operator's terminal, where the translated inquiry is displayed and the operator can respond appropriately.

[0589] 4. Operator response and reply

[0590] The operator types a reply based on the displayed inquiry and sends it to the server. For example, they might type, "I'll look into it, so could you please wait a moment?"

[0591] 5. Reply translation and notification

[0592] The server then translates the operator's reply using the Google Translate API, adapting it to the user's language, and uses sentiment analysis tools to recheck and neutralize the emotional content of the reply.

[0593] 6. Use of Emotion Engines

[0594] The emotion engine accumulates the contents of inquiries and replies, suggests appropriate responses to operators, and analyzes the user's emotional state and issues warnings to operators based on the analysis results.

[0595] Specific examples

[0596] Example 1: Inquiry in Japanese

[0597] When a user makes an inquiry in Japanese, such as "The product delivery is delayed, what's going on?", the device sends this message to the server, which uses the Google Translate API to translate it into English as "The product delivery is delayed, what is happening?" IBM Watson Emotion Analysis then neutralizes the emotional expression, saying "The product delivery is delayed, please advise [current status]." The operator considers how to respond based on this message and replies, "We will investigate, so could you please wait a moment?" This reply is then translated again by the server into English as "We will investigate, please wait a moment," and sent to the user.

[0598] Example 2: Complaint in English

[0599] If a user complains in English, saying, "I'm extremely upset with your service. My order was not delivered on time," the device sends this content to the server, which then uses the Google Translate API to translate it into Japanese as, "I'm extremely unhappy with your service. My order was not delivered on time." The emotion analysis means neutralizes strong emotional expressions and notifies the operator, "A customer has inquired about their dissatisfaction with the service and a delayed order." The emotion engine analyzes the user's emotional state and alerts the operator, saying, "The customer is extremely unhappy. Please prioritize their response."

[0600] Example prompts for generative AI models

[0601] Below are some example prompts that explain the operation of programs on this system:

[0602] Please describe in detail the processing steps of your system for multilingual support and emotional complaint handling using specific hardware and software. Include specific behavior for each processing step. For example, please include the processing for when a customer inquires in Japanese, "My item is late. What's going on?", and when a customer in English complains, "I'm extremely upset with your service. My order was not delivered on time."

[0603] The system of the present invention provides an efficient and stress-free environment for both users and operators by centrally utilizing real-time translation, emotion analysis, and an emotion engine. Utilizing the emotion engine enables more advanced emotion recognition and the proposal of response methods, thereby achieving improved service quality.

[0604] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0605] Step 1:

[0606] A user sends a query from their own terminal. Specifically, they enter the query content in their own language and click the send button. At this time, the terminal sends the query content and the user's language information to the server. The input is the query text entered by the user, and the output is an HTTP request sent to the server.

[0607] Step 2:

[0608] The server passes the received inquiry to a real-time translation tool and translates it into the specified language. Specifically, it uses the Google Translate API to translate the received Japanese text into English. The input is the user's inquiry text, and the output is the translated text. At this stage, data processing involves converting the text into a different language.

[0609] Step 3:

[0610] The server passes the translated text to a sentiment analyzer to detect and neutralize emotional expressions. Specifically, it uses IBM Watson Emotion Analysis to convert strong emotional expressions into neutral ones. The input is the translated text, and the output is text with neutralized emotional elements. At this stage, the data operations are sentiment analysis and neutralization.

[0611] Step 4:

[0612] The server sends the neutralized query content to the operator's terminal. Specifically, the translated and neutralized query content is displayed on the operator's terminal. The input is the neutralized text, and the output is the content displayed on the operator's terminal. The server sends the data as an HTTP response, and the operator's terminal receives and displays it.

[0613] Step 5:

[0614] The user (operator) responds based on the displayed inquiry. Specifically, the operator enters a reply and presses the send button, which sends the reply from the terminal to the server. The input is the reply entered by the operator, and the output is an HTTP request sent to the server.

[0615] Step 6:

[0616] The server passes the operator's reply to a real-time translation tool, which translates it into the user's language. Specifically, it uses the Google Translate API to translate replies entered in Japanese into English. The input is the operator's reply text, and the output is the translated text. Data processing at this stage involves another language conversion.

[0617] Step 7:

[0618] The server then rechecks the translated reply content using sentiment analysis and neutralizes it. Specifically, it uses IBM Watson Emotion Analysis to check for emotional expressions in the reply content and neutralize it. The input is the translated reply text, and the output is the neutralized reply text. The data operations at this stage are sentiment analysis and neutralization.

[0619] Step 8:

[0620] The server sends the neutralized reply content to the user's terminal. The server sends the reply content with the emotional elements removed to the user's terminal as an HTTP response. The input is the neutralized reply text, and the output is the content displayed on the user's terminal.

[0621] Step 9:

[0622] The emotion engine monitors the user's emotional state and proposes the analysis results to the operator. Specifically, it uses the Affectiva SDK to detect the user's emotional state in real time and proposes a response. The input is the inquiry and reply, and the output is a response proposal to the operator.

[0623] Step 10:

[0624] The server accumulates data on inquiries and replies and stores it for analysis. The server stores all communication history in a database for later analysis and service improvement. The input is the inquiry and reply data, and the output is the accumulated database.

[0625] (Application example 2)

[0626] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0627] Conventional call centers and user support systems have limited multilingual support and emotion analysis capabilities, making it difficult to provide fast and effective global customer support. Furthermore, they lack the ability to properly recognize users' emotions and propose neutral responses, which can easily cause stress for both operators and users. If these issues continue for a long period of time, they could lead to a decline in service quality and customer satisfaction.

[0628] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0629] In this invention, the server includes knowledge processing means for translating text in real time, sentiment analysis means for performing sentiment analysis and neutralizing sentiment from the translated text, means for displaying on an operation device the inquiry content processed by the translation means and the sentiment analysis means, means for translating a reply entered on the operation device into the language of the terminal by the translation means and the sentiment analysis means and transmitting the neutralized text to the terminal device, means for monitoring the emotional state of the user in real time using a sentiment analysis function and proposing a response method, and means for quickly and effectively processing user feedback based on the response method. This enables advanced user support that combines multilingual real-time translation and sentiment analysis.

[0630] The "knowledge processing means for real-time text translation" is a function for instantly converting the content of a user's inquiry into another language.

[0631] The "sentiment analysis means for performing sentiment analysis and neutralizing emotions from translated text" is a function for analyzing emotional expressions in translated text and converting them into neutral expressions in order to reduce the intensity of the emotions.

[0632] The "means for displaying on an operating device" is a function for displaying the translated and sentiment-analyzed inquiry content on a device operated by an operator.

[0633] The "means for transmitting to the terminal device" is a function that translates and neutralizes the emotion again in order to transmit the reply entered by the operator to the user's terminal.

[0634] "Means of monitoring the user's emotional state in real time using an emotion analysis function and suggesting ways to respond" is a function that detects the user's emotions in real time and suggests the most appropriate way to respond to the operator based on that emotional state.

[0635] "Means for quickly and effectively processing feedback" refers to a function that instantly translates and analyzes sentiment on user feedback and quickly responds appropriately.

[0636] The present invention relates to a system for handling complaints in multiple languages ​​and emotionally. This system has the following specific configuration and operation.

[0637] System configuration

[0638] 1. Server:

[0639] The server is equipped with a knowledge processing means for translating text in real time, a sentiment analysis means for performing sentiment analysis to neutralize emotions from the translated text, and a sentiment analysis function. This server receives, processes, and transmits inquiries from users.

[0640] 2. Terminal:

[0641] The terminal consists of a device used by the user (e.g., a smartphone or smart glasses) and an operating device used by the operator. The terminal displays the translated and sentiment-analyzed content from the server and receives input from the operator.

[0642] 3. User:

[0643] The users include the users who make inquiries and the operators who respond to those inquiries.

[0644] Program processing

[0645] 1. Receiving inquiries

[0646] A user sends a query from their own device to the server, which includes the query content and the user's language information.

[0647] 2. Translation and Sentiment Analysis

[0648] The server translates the received query using a knowledge processing means that translates in real time.Then, a sentiment analysis means analyzes the translated content and neutralizes emotional expressions.

[0649] 3. Notification to operator and response

[0650] The processed inquiry is sent from the server to the operator's operating device. The operator responds based on the displayed information. The reply entered by the operator is translated and neutralized again before being sent to the user's device.

[0651] Hardware and software used

[0652] The system of the present invention mainly uses the following hardware and software.

[0653] Hardware: Server, smartphone, smart glasses, operator control device

[0654] Software: Python, Translators API, TextBlob

[0655] Data processing and calculation

[0656] 1. Real-time translation

[0657] The server translates the received text in real time using the Translators API.

[0658] 2. Sentiment analysis

[0659] Analyze the sentiment of text using TextBlob and neutralize the emotional expressions.

[0660] 3. Monitoring your emotional state

[0661] The server uses emotion analysis to monitor the user's emotional state in real time and present the operator with the most appropriate response.

[0662] Specific examples

[0663] Example 1: User gives feedback in English

[0664] The user sends feedback in English, saying, "I'm extremely upset with your service. My order was not delivered on time." The server receives this feedback and translates it in real time to "I'm extremely upset. My order was not delivered on time." It then performs sentiment analysis, which parses it as an expression of strong anger. The server then neutralizes the emotion and displays on the operator's control device, "A customer has inquired about their dissatisfaction with the service and a delayed order." The sentiment analysis function also issues a warning, such as, "The customer is extremely dissatisfied. Please prioritize their response." The operator enters an appropriate response, which the server translates and sends to the user's device, "We apologize for the inconvenience, we will check immediately."

[0665] Example of input prompt for generative AI model

[0666] "I'm extremely upset with your service. My order was not delivered on time."

[0667] Prompt: "Translate the above English sentence into Japanese and perform sentiment analysis to neutralize the emotional content. Then, suggest how to respond to the operator."

[0668] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0669] Step 1:

[0670] This is the action in which a user (consumer) inputs the content of a query from their own device and sends it to the server. The input is the user's query and its language information, and the output is the raw query data received by the server. The server then performs preliminary preparations to translate this query in real time. Specifically, the query and language information are saved in a database and used in the next step.

[0671] Step 2:

[0672] This is the operation in which the server translates the input inquiry content in real time. The input is the inquiry content and language information received in step 1, and the output is the translated text. The server calls the Translators API to translate from the original language into a common language (such as English) that the operator can understand. Specifically, the Translators API is used to translate the English sentence, "I'm extremely upset with your service. My order was not delivered on time." into Japanese.

[0673] Step 3:

[0674] The server performs sentiment analysis on the translated text and neutralizes emotional expressions as necessary. The input is the translated text obtained in step 2, and the output is the neutralized text. The server uses TextBlob to perform sentiment analysis and detect emotional expressions. For example, it converts a sentence that expresses strong anger into a neutral expression such as "We apologize for the inconvenience."

[0675] Step 4:

[0676] This is the operation in which the server notifies the operator based on emotion analysis. The input is the neutralized text obtained in step 3, and the output is a notification message displayed on the operator's operating device. The server uses its emotion analysis function to evaluate the user's emotional state and sends a notification to the operator, such as "The customer is very dissatisfied. Please handle this as a priority."

[0677] Step 5:

[0678] This is the operation in which the operator inputs a response on the operating device and sends the content to the server. The input is the text of the operator's response, and the output is the response data from the operator received by the server. The operator inputs an appropriate response based on the neutralized inquiry content and clicks the send button.

[0679] Step 6:

[0680] The server then translates the operator's response again and performs sentiment analysis. The input is the operator's response text received in step 5, and the output is the translated and neutralized reply text. The server again uses the Translators API and TextBlob to translate the operator's reply into the user's language and neutralize it to avoid emotional overload.

[0681] Step 7:

[0682] This is the operation in which the server finally sends the translated and sentiment-analyzed reply text to the user's device. The input is the neutralized reply text obtained in step 6, and the output is the reply message displayed on the user's device. The server sends the translated and neutralized text to the user's device, providing a response in a way that is less stressful for the user.

[0683] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0684] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0685] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0686] [Third embodiment]

[0687] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0688] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0689] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0690] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0691] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0692] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0693] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0694] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0695] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0696] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0697] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0698] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0699] The present invention relates to a system for multilingual and emotional complaint handling, which includes a real-time translation means and a sentiment analysis means, enabling fast and effective responses in global call centers.

[0700] Program processing

[0701] The main components of this system are the server, terminals, and users (operators and users). The roles and operations of each are explained below.

[0702] 1. Receiving inquiries

[0703] The user sends a support request from their device, which includes the content of the inquiry and the language they are using. The device then sends this information to the server.

[0704] 2. Translation and Sentiment Analysis

[0705] The server then passes the received inquiry to a real-time translation unit, which translates the user's language into a language the operator can understand. The emotion analysis unit then analyzes the translated content and converts emotional expressions into neutral ones, allowing the operator to respond calmly based on the information.

[0706] 3. Notification to Operator

[0707] The translated and neutralized query content is sent from the server to the operator's terminal, where it is displayed in an easy-to-understand format.

[0708] 4. Operator response and reply

[0709] The user (operator) responds based on the displayed content and enters a reply, which is then sent from the terminal to the server.

[0710] 5. Reply translation and notification

[0711] The server then passes the operator's reply to the real-time translation tool, which translates it into the user's language. The sentiment analysis tool then converts the translated reply into a neutral expression and sends the final answer to the user's device, ensuring the user receives a calm and clear response.

[0712] Specific examples

[0713] Example 1: When a user makes an inquiry in Japanese

[0714] The user makes an inquiry in Japanese, saying, "The product delivery is delayed. What's going on?" The terminal sends this inquiry to the server, which uses real-time translation means to translate it into English as, "The product delivery is delayed, what is happening?" The emotion analysis means then analyzes this content and neutralizes the emotional expression, "It's delayed, what's going on?" The operator's terminal then displays the neutralized message, "The product delivery is delayed, please advise [current status]."

[0715] The operator replies, "We will investigate, please wait a moment," and sends the reply from the device to the server. The server translates this reply from Japanese to English, saying, "We will investigate, please wait a moment," and then uses sentiment analysis to confirm that it is neutral. The final reply is sent to the user's device.

[0716] Example 2: When a customer makes a complaint in English

[0717] If a user complains in English, "I'm extremely upset with your service. My order was not delivered on time," the device sends this content to the server, which then uses a translation tool to translate it into Japanese as "I'm extremely upset with your service. My order was not delivered on time." The sentiment analysis tool neutralizes the strong expression "I'm dissatisfied," and displays to the operator, "A customer has inquired about their dissatisfaction with the service and the delay in their order."

[0718] The operator replies, "We apologize for the inconvenience, we will check immediately." The server then translates this into English and sends it to the user's device as, "We apologize for the inconvenience, we will check immediately."

[0719] The system of the present invention achieves multilingual support through real-time translation and sentiment analysis, providing a stress-free environment for both operators and users.

[0720] The processing flow will be explained below.

[0721] Step 1:

[0722] A user sends a support request from their own device. This request contains the inquiry content and the user's language information. For example, a user enters a message in Japanese saying, "My product is late. What's going on?" and clicks the send button.

[0723] Step 2:

[0724] The terminal receives the user's input information, sends the query content and the user's language information to the server, packages the user's language and text content, and makes a request to the server.

[0725] Step 3:

[0726] The server passes the received inquiry to an AI translation engine, which translates the user's language into a language the operator can understand. For example, the Japanese phrase "The product delivery is delayed, what's happening?" is translated into English as "The product delivery is delayed, what is happening?"

[0727] Step 4:

[0728] The server then sends the translation results to an AI sentiment analysis engine, which analyzes the translated query, detects any emotional expressions, and converts them into neutral phrases. For example, it neutralizes the emotional expression "what is happening?" to "please advise [current status]."

[0729] Step 5:

[0730] The server sends the translated and neutralized inquiry to the operator terminal. The operator terminal displays the translated and neutralized inquiry. The operator is shown a message saying, "A customer has inquired about a delay in the delivery of their product."

[0731] Step 6:

[0732] The user (operator) responds based on the displayed inquiry and enters a reply. For example, they might type, "I will look into this, so could you please wait a moment?" and click the send button.

[0733] Step 7:

[0734] The terminal sends the operator's input to the server and requests the operator's reply text from the server.

[0735] Step 8:

[0736] The server then passes the operator's reply to the AI ​​translation engine again, which translates it from the operator's language to the user's language. For example, the Japanese phrase "We will investigate, so please wait a moment." is translated into English as "We will investigate, please wait a moment."

[0737] Step 9:

[0738] The server then sends the translated reply back to the AI ​​sentiment analysis engine to neutralize the emotional content. If the content is already neutral, no changes are made.

[0739] Step 10:

[0740] The server then sends the translated and neutralized reply to the user's device. The server then sends the translated reply, "We'll look into this, so could you please wait a moment?"

[0741] Step 11:

[0742] The terminal (user) displays the translated reply received from the server on its screen. The user's screen displays "We will look into this, could you please wait a moment?", and the response is complete.

[0743] The above steps constitute a system that enables multilingual and emotion-neutral communication between users and operators.

[0744] Example 1

[0745] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0746] Today's global call centers are required to handle inquiries from users quickly and accurately. However, due to issues such as diverse languages ​​and emotional expressions, it is often difficult for operators to respond efficiently. It is also not easy to respond calmly and neutrally to emotional complaints. This increases the burden on operators and risks a decline in response quality. There is a need for a system that can solve these issues, improve the efficiency of inquiry responses at global call centers, and improve response quality.

[0747] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0748] In this invention, the server includes means for receiving inquiries from user terminals, AI translation means for translating text in real time, AI sentiment analysis means for performing sentiment analysis and neutralizing the sentiment of the translated text, means for displaying the inquiry content processed by the translation means and the sentiment analysis means on an operator terminal, and means for translating a reply entered by an operator into the user's language by the translation means and the sentiment analysis means and sending the neutralized text to the user terminal. This enables inquiries to be processed quickly and accurately in a global call center, providing a stress-free environment for both operators and users.

[0749] "User" refers to an individual or company that uses the system to make an inquiry.

[0750] A "terminal" is a device (e.g., PC, smartphone, tablet) used by a user or operator to access the system.

[0751] A "server" is a central computer that manages the processing of the entire system and sends and receives various data.

[0752] "AI translation mechanism" refers to a mechanism that utilizes artificial intelligence technology to translate text into another language in real time.

[0753] An "AI sentiment analysis tool" is a mechanism that uses artificial intelligence technology to analyze and neutralize emotional expressions in text.

[0754] "Inquiry content" refers to the text information of questions or complaints sent by users through the system.

[0755] An "operator" is a system user who responds to inquiries from users and provides necessary replies.

[0756] A "neutralized text" is a text that has been transformed into a calm and objective expression, with emotional expressions suppressed.

[0757] This invention is a system that realizes multilingual support and emotional complaint handling in global call centers. This system is mainly composed of three components: a server, terminals, and users (operators and customers).

[0758] System Overview

[0759] Receiving inquiries

[0760] A user sends a support request from their device (PC, smartphone, etc.). This request includes the inquiry and the language used by the user. The device then sends this information to the server.

[0761] Translation and Sentiment Analysis

[0762] The server passes the received inquiry to a real-time translation tool (e.g., Google Translate API) to translate the user's language into a language the operator can understand. The translated content is then analyzed using a sentiment analysis tool (e.g., IBM Watson Natural Language Understanding) to convert emotional expressions into neutral ones, allowing the operator to respond calmly based on the information.

[0763] Operator notification

[0764] The translated and neutralized inquiry is sent from the server to the operator's terminal, where the inquiry is displayed in an easy-to-understand format through dedicated customer support software.

[0765] Operator response and reply

[0766] The user (operator) responds based on the displayed information and enters a reply, which is then sent from the terminal to the server.

[0767] Reply translation and notification

[0768] The server then passes the operator's reply to the real-time translation unit, which translates it into the user's language. The sentiment analysis unit then converts the translated reply into a neutral expression, and the final answer is sent to the user's device. This allows the user to receive a calm and clear response.

[0769] Specific examples

[0770] Example 1: When a user makes an inquiry in Japanese

[0771] The user makes an inquiry in Japanese, saying, "The product delivery is delayed. What's going on?" The device sends this inquiry to the server. The server uses real-time translation means to translate this into English as, "The product delivery is delayed, what is happening?" The sentiment analysis means then analyzes this content and displays a neutralized response, "The product delivery is delayed, please advise [current status]." on the operator's device. The operator replies, "We will investigate, could you please wait a moment?" and sends this response from the device to the server. The server then translates this response from Japanese to English, saying, "We will investigate, please wait a moment." The sentiment analysis means then confirms the neutral expression, and the final reply is sent to the user's device.

[0772] Example 2: When a customer makes a complaint in English

[0773] A user complains in English, saying, "I'm extremely upset with your service. My order was not delivered on time." The device sends this content to the server. The server uses a translation tool to translate this into Japanese as "I'm extremely unhappy with your service. My order was not delivered on time." The sentiment analysis tool neutralizes the strong expression "unhappy" and displays to the operator, "A customer has inquired about their dissatisfaction with the service and a delayed order." The operator replies, "We apologize, we will check immediately." The server translates this content into English and sends it to the user's device as "We apologize for the inconvenience, we will check immediately."

[0774] Prompt Sentence Examples

[0775] Here are some examples of prompts for generative AI models:

[0776] "Please explain a scenario in which a user makes an inquiry in Japanese and an operator responds. Please also include the specific process flow and the functions used (translation methods, sentiment analysis methods)."

[0777] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0778] Step 1:

[0779] A user sends a support request using their own device. As input, they enter text data containing the inquiry and the user's language information into the device. The device then sends this data to the server as an HTTP request. The output is the inquiry data transferred to the server.

[0780] Step 2:

[0781] The server passes the received query data to a real-time translation method (e.g., Google Translate API). The input is the query text sent by the user and its language information. The server passes this information to the API for translation processing. The output is the translated text returned to the server.

[0782] Step 3:

[0783] The server passes the translated text to a sentiment analyzer (e.g., IBM Watson Natural Language Understanding). The input is the translated query text. The server sends the text to a sentiment analysis API, which analyzes the sentiment in the text. The output is text with neutralized emotional expressions.

[0784] Step 4:

[0785] The server sends the neutralized text to the operator's terminal. The input is the query data processed by the sentiment analysis means. The server transfers this data to the operator's terminal using REST API or WebSocket. The output is the neutralized query content displayed on the operator's terminal.

[0786] Step 5:

[0787] The user (operator) considers how to respond based on the displayed content and enters a reply. As input, the operator enters the proposed response as text into the terminal. The terminal then sends this reply data back to the server. The output is the reply data transferred to the server.

[0788] Step 6:

[0789] The server passes the operator's reply to a real-time translation means. The input is the reply text sent by the operator. The server sends the reply text to a translation means, which translates the reply into the user's language. The output is the translated reply text.

[0790] Step 7:

[0791] The server passes the translated reply to the sentiment analyzer again to confirm the neutralized expression. The input is the translated reply text. The server passes this text to the sentiment analysis API for verification. The output is the reply text with the neutralized expression confirmed.

[0792] Step 8:

[0793] The server sends the final answer to the user's device. The input is the reply text that has been checked for neutrality. The server transfers this text to the user's device using a REST API or WebSocket. The output is the final reply that is displayed on the user's device.

[0794] (Application example 1)

[0795] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0796] Conventional customer service systems have difficulty dealing with customers who speak different languages ​​and have different cultural backgrounds, and have had particular problems with incomplete multilingual support and emotional complaint handling. Furthermore, in brick-and-mortar stores, staff were not equipped with an environment where they could instantly provide multilingual support, making it difficult to provide prompt and appropriate service.

[0797] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0798] In this invention, the server includes an artificial intelligence translation means for translating text in real time, an artificial intelligence emotion analysis means for performing emotion analysis and neutralizing the emotion from the translated text, a means for displaying the inquiry content processed by the translation means and the emotion analysis means on a staff terminal, a means for translating a reply entered by a staff member into the user's language by the translation means and the emotion analysis means and sending the neutralized text to the user terminal, and a means for displaying the inquiry content and the reply content on an augmented reality device used as a staff terminal. This enables multilingual and emotional complaint handling to be performed quickly and effectively even in physical stores, thereby improving customer satisfaction.

[0799] "Artificial intelligence translation means for real-time text translation" refers to a device that uses artificial intelligence to instantly convert the content of an inquiry into a different language.

[0800] The "artificial intelligence emotion analysis means for performing emotion analysis and neutralizing emotions from translated text" is a device that uses artificial intelligence to analyze emotional expressions in translated text and convert them into calm expressions.

[0801] "Staff terminal" means an electronic device used by customer service staff to receive and enter information.

[0802] "User terminal" refers to an electronic device used by a service user to receive support or enter inquiries.

[0803] An "augmented reality device" is a device worn by staff that can overlay virtual information onto the actual environment.

[0804] "Inquiry content" refers to the content of questions, requests, and complaints made by users to customer service.

[0805] A "neutralized text" is text that has been transformed from emotional expression into calm, objective expression.

[0806] A "translation tool" is a device or software for converting text from one language to another.

[0807] A "sentiment analyzer" is a device or software for analyzing and identifying emotional elements in text.

[0808] The present invention is a system for multilingual support and emotional complaint handling in brick-and-mortar stores. This system performs real-time translation and sentiment analysis between customers and customer service staff (hereinafter referred to as staff), enabling staff to respond appropriately and quickly. Specific embodiments are described below.

[0809] Overall system configuration

[0810] 1. Server and Cloud Infrastructure

[0811] The server includes the hardware and software necessary to implement real-time translation and sentiment analysis. The main software used is the Google Translator API, which provides translation functionality, the TextBlob library for sentiment analysis, and the SpeechRecognition library for voice recognition.

[0812] 2. Staff terminals and augmented reality devices

[0813] Smart glasses (e.g., Google Glass) are used as staff terminals, and these devices display inquiries and replies to staff in real time.

[0814] 3. User Device

[0815] A user (customer) makes an inquiry using a personal device such as a smartphone.

[0816] Data processing and calculation

[0817] The server processes and calculates data in the following procedure.

[0818] 1. Voice Recognition

[0819] When a user makes a voice inquiry using their smartphone, the voice data is transmitted to the staff member's smart glasses, which then send the voice data to the server, which then converts the voice data into text using the SpeechRecognition library.

[0820] 2. Real-time translation

[0821] The server translates the text obtained by speech recognition in real time using the Google Translator API. For example, it converts a Japanese inquiry into English.

[0822] 3. Sentiment Analysis and Neutralization

[0823] The server performs sentiment analysis on the translated text using the "TextBlob" library. If the sentiment is strong, it converts certain keywords (e.g., "very dissatisfied") into neutral expressions (e.g., "concerned").

[0824] 4. Staff Display

[0825] The neutralized text is processed on the server and displayed on the staff's smart glasses, allowing them to respond appropriately to the customer.

[0826] 5. Reply Translation and Notification

[0827] The reply entered by the staff is sent back to the server, where it is translated and sentiment-analyzed. The translated text is neutralized and then sent to the customer's smartphone.

[0828] Specific use cases

[0829] If a customer asks in Japanese, "The product delivery is delayed, what's going on?", this speech is converted into text on the server and translated into English. The translated text, "The product delivery is delayed, what is happening?", undergoes sentiment analysis and neutralization (e.g., "The product delivery is delayed, please advise [current status].") and is displayed on the staff member's smart glasses. The staff member responds, "We will investigate, please wait a moment," and the reply is also translated into English and notifies the customer, "We will investigate, please wait a moment."

[0830] Example prompt sentence:

[0831] Analyze the following sentences and convert them into neutral expressions.

[0832] Text: "I'm extremely upset with your service. My order was not delivered on time."

[0833] Neutral: "I'm concerned about the delay in my order delivery. Please advise on the current status."

[0834] As described above, the present invention aims to realize multilingual support and emotional complaint handling, thereby improving the quality of customer service in physical stores.

[0835] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0836] Step 1:

[0837] A user makes a voice inquiry using a smartphone. This voice data is collected through the smartphone's microphone and saved as digital data. The input is voice data, and the output is digital voice data.

[0838] Step 2:

[0839] The user's smartphone sends voice data to the server, which converts it into text using the "SpeechRecognition" library. The input is digital voice data, and the output is text data. This conversion transforms the voice data into concrete text information.

[0840] Step 3:

[0841] The server translates the text data in real time using the Google Translator API. For example, Japanese text is converted into English text. The input is Japanese text data, and the output is English text data. This translation converts the data into a language that the staff can understand.

[0842] Step 4:

[0843] The server performs sentiment analysis on the translated English text data using the "TextBlob" library. The sentiment analysis results in positive, negative, and neutral sentiment. The input is the English text data, and the output is the sentiment analysis results and corrected text data. Sentiment analysis neutralizes expressions as needed.

[0844] Step 5:

[0845] The server sends neutralized text data with reduced emotion to the augmented reality device (smart glasses). The staff member's smart glasses receive this data and display it on their screen. The input is the neutralized text data, and the output is the display on the smart glasses.

[0846] Step 6:

[0847] The staff member checks the neutralized text through the smart glasses and types an appropriate reply to the customer. The reply is recorded as audio by the smart glasses' microphone and sent to the server as text data. The input is audio data and the output is text data.

[0848] Step 7:

[0849] The server again translates the staff member's reply text data into the customer's language (e.g., Japanese) using the "Google Translator API." It then performs sentiment analysis on the translated text using the "TextBlob" library to confirm neutralization. The input is the staff member's English text data, and the output is neutralized Japanese text data.

[0850] Step 8:

[0851] The server sends the neutralized Japanese text data to the user's smartphone. The user's smartphone receives this data and displays it on its screen. The input is the neutralized Japanese text data, and the output is what is displayed on the smartphone's screen.

[0852] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0853] The present invention relates to a system for multilingual and emotional complaint handling, which includes a real-time translation means, a sentiment analysis means, and an emotion engine, enabling fast and effective responses in global call centers.

[0854] Program processing

[0855] The main components of this system are the server, terminals, and users (operators and users). The roles and operations of each are explained below.

[0856] 1. Receiving inquiries

[0857] A user sends a support request from their own device. This request contains the inquiry content and the user's language information. For example, a user enters a message in Japanese saying, "My product is late. What's going on?" and clicks the send button.

[0858] 2. Translation and Sentiment Analysis

[0859] The server passes the received inquiry to a real-time translation means, which translates it from the user's language into a language the operator can understand. For example, the Japanese phrase "The product delivery is delayed, what's going on?" is translated into English as "The product delivery is delayed, what is happening?". Next, the sentiment analysis means analyzes the translated content and converts emotional expressions into neutral expressions. For example, the emotional expression "what is happening?" is neutralized to "please advise [current status]."

[0860] 3. Notification to Operator

[0861] The translated and neutralized inquiry is sent from the server to the operator's terminal. The operator's terminal displays the translated and neutralized inquiry. For example, the operator might see, "A customer has inquired about a delay in the arrival of their product."

[0862] 4. Operator response and reply

[0863] The user (operator) responds based on the displayed inquiry and enters a reply. For example, they might type, "I will look into this, so could you please wait a moment?" and click the send button.

[0864] 5. Reply translation and notification

[0865] The server then passes the operator's reply back to the real-time translation means, which translates it from the operator's language into the user's language. For example, the Japanese phrase "We will investigate, so please wait a moment." is translated into English as "We will investigate, please wait a moment." The emotion analysis means then checks for neutral expressions. The emotion engine monitors the user's emotional state in real time, neutralizes the emotional content, and sends it to the user's device.

[0866] 6. Use of Emotion Engines

[0867] The emotion engine recognizes the user's emotional state and suggests how the operator should respond based on that. For example, if the emotion engine recognizes that the user is angry, it will warn the operator that "the customer is angry, so a polite response is required." In addition, based on emotion analysis, the frequency and patterns of emotional complaints are analyzed and data is accumulated. This data is used for future service improvements and training.

[0868] Specific examples

[0869] Example 1: When a user makes an inquiry in Japanese

[0870] The user makes an inquiry in Japanese, saying, "The product delivery is delayed. What's going on?" The device sends this inquiry to the server, which uses real-time translation to translate it into English as, "The product delivery is delayed, what is happening?" The emotion analysis device analyzes this content and neutralizes the emotional expression, "It's delayed, what's going on?" The neutralized message, "The product delivery is delayed, please advise [current status]," is displayed on the operator's device. The emotion engine recognizes the user's emotions and alerts the operator.

[0871] The operator replies, "We will investigate, so please wait a moment," and sends the reply from the device to the server. The server translates this reply from Japanese to English and sends it to the user's device as "We will investigate, please wait a moment." The emotion engine then rechecks the emotion of the reply, confirming that it is neutral before sending it.

[0872] Example 2: When a customer makes a complaint in English

[0873] If a user complains in English, "I'm extremely upset with your service. My order was not delivered on time," the device sends this content to the server, which then uses a translation tool to translate it into Japanese as "I'm extremely upset with your service. My order was not delivered on time." The emotion analysis tool neutralizes the strong expression "dissatisfied" and displays to the operator, "A customer has inquired about their dissatisfaction with the service and a delayed order." The emotion engine recognizes the user's strong anger and alerts the operator, "The customer is extremely dissatisfied. Please handle this as a priority."

[0874] The operator replies, "We apologize for the inconvenience, we will check immediately," and the server translates this into English and sends it to the user's device as, "We apologize for the inconvenience, we will check immediately." The emotion engine then rechecks the emotion of the reply, making sure it is neutral before sending it.

[0875] The system of the present invention realizes multilingual support through real-time translation, emotion analysis, and an emotion engine, providing a stress-free environment for both operators and users. The emotion engine enables more advanced emotion recognition and suggests ways to respond, improving service quality.

[0876] The processing flow will be explained below.

[0877] Step 1:

[0878] A user sends a support request from their own device. This request contains the inquiry content and the user's language information. For example, a user enters a message in Japanese saying, "My product is late. What's going on?" and clicks the send button.

[0879] Step 2:

[0880] The terminal receives the user's input information and sends the query content and the user's language information to the server. The terminal transfers the text content and language information to the server in packet format.

[0881] Step 3:

[0882] The server passes the received inquiry as a request to the AI ​​translation engine. The AI ​​translation engine translates the user's language into a language the operator can understand. For example, the Japanese phrase "The product delivery is delayed, what's happening?" is translated into English as "The product delivery is delayed, what is happening?"

[0883] Step 4:

[0884] The server sends the translation results to an AI sentiment analysis engine, which analyzes the translated query, detects emotional expressions, and converts them into neutral expressions. For example, it neutralizes the emotional expression "what is happening?" to "please advise [current status]."

[0885] Step 5:

[0886] The server sends the translated and neutralized query content to the operator terminal, which checks the displayed content and starts responding to the user based on it.

[0887] Step 6:

[0888] The emotion engine analyzes the emotional state of the received text and monitors the user's emotional state in real time. The emotion engine determines whether the user is angry or dissatisfied, and if strong emotions are detected, a warning message such as "The customer is angry, so polite response is required" is displayed on the operator terminal.

[0889] Step 7:

[0890] The user (operator) responds based on the displayed inquiry and enters a reply. For example, they might type, "I will look into this, so could you please wait a moment?" and click the send button.

[0891] Step 8:

[0892] The terminal sends the operator's input to the server. The server then passes the reply back to the AI ​​translation engine, which translates it from the operator's language into the user's language. For example, the Japanese phrase "We will investigate, so could you please wait a moment?" is translated into English as "We will investigate, please wait a moment."

[0893] Step 9:

[0894] The server then sends the translated reply back to the AI ​​sentiment analysis engine to neutralize the emotional content. The sentiment analysis engine analyzes the reply and neutralizes it as necessary. Once this process is complete, the server sends the final translated and neutralized reply to the user's device.

[0895] Step 10:

[0896] The terminal (user) displays the translated reply received from the server on its screen and notifies the user. The user confirms the reply displayed on the screen and completes the inquiry.

[0897] Through the above steps, a system is realized in which communication with users is carried out in multiple languages ​​and in an emotion-neutral manner, allowing operators to respond quickly and effectively.

[0898] Example 2

[0899] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0900] To minimize communication delays and misunderstandings caused by language barriers and emotional complaint handling in global call centers, and to achieve effective and neutral communication between users and operators.

[0901] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for accepting inquiry content and translating the text in real time, means for sentiment-analyzing the translated text and neutralizing emotional expressions, means for displaying the translated and neutralized inquiry content on an operator terminal, means for translating a reply entered by the operator in real time, sentiment-analyzing and neutralizing the reply, and sending it to the user's terminal, means for recognizing, analyzing, and monitoring the user's emotional state and suggesting a response method to the operator, and means for accumulating data on the inquiry content and the reply content and storing it for analysis. This minimizes communication delays and misunderstandings due to language barriers and emotional responses to complaints, enabling effective and neutral communication between users and operators.

[0902] The "means for receiving inquiry content" is a function for electronically receiving inquiry content from a user and transferring that information to the server.

[0903] "Means for translating text in real time" refers to the ability to instantly convert received text into another language.

[0904] "Means for sentiment analysis of translated text and neutralizing emotional expressions" is a function that analyzes translated text to detect emotional elements and converts them into neutral expressions.

[0905] The "means for displaying the translated and neutralized inquiry content on the operator terminal" is a function for visually displaying the processed inquiry content on the operator terminal.

[0906] "Means for translating replies entered by operators in real time, analyzing and neutralizing emotions, and sending them to the user's device" refers to a function that instantly translates the operator's response into another language, neutralizes emotional elements, and then sends it to the user.

[0907] "Means for recognizing, analyzing, and monitoring the user's emotional state and suggesting appropriate ways to respond to the operator" refers to a function that detects and analyzes the user's emotions and suggests appropriate ways to respond to the operator based on that information.

[0908] "Means for accumulating data on inquiry and reply content and storing it for analysis" refers to a function that stores the history of all inquiries and replies in a database so that it can be used later for analysis and improvement.

[0909] This invention relates to a system for effectively handling multilingual and emotional complaints in global call centers. This system includes a real-time translation means, a sentiment analysis means, and an emotion engine, and automates the entire process from processing inquiries to replying.

[0910] Hardware and software used

[0911] The main components of the system are the server, the terminal, and the user (operator and user). This system uses the following hardware and software:

[0912] Real-time translation means: As an example, we will use Google Translate API, which is a software function for instantly translating inquiries and replies into different languages.

[0913] Sentiment analysis tools: As an example, we will use IBM Watson Emotion Analysis, which is a function for analyzing and neutralizing the emotional components of received text.

[0914] Emotion engine: Using the Affectiva SDK as an example, this is an engine that monitors the user's emotional state in real time and suggests ways to respond to the operator based on the analysis results.

[0915] The entire process

[0916] 1. Receiving inquiries

[0917] The user (consumer) inputs the inquiry content from their own device and sends it. For example, a user might input "The arrival of the product is delayed. What's going on?" in Japanese and click the send button.

[0918] 2. Translation and Sentiment Analysis

[0919] The server translates the received query in real time using the Google Translate API, while neutralizing emotional expressions using IBM Watson Emotion Analysis.

[0920] 3. Notification to Operator

[0921] The translated and neutralized inquiry is sent from the server to the operator's terminal, where the translated inquiry is displayed and the operator can respond appropriately.

[0922] 4. Operator response and reply

[0923] The operator types a reply based on the displayed inquiry and sends it to the server. For example, they might type, "I'll look into it, so could you please wait a moment?"

[0924] 5. Reply translation and notification

[0925] The server then translates the operator's reply using the Google Translate API, adapting it to the user's language, and uses sentiment analysis tools to recheck and neutralize the emotional content of the reply.

[0926] 6. Use of Emotion Engines

[0927] The emotion engine accumulates the contents of inquiries and replies, suggests appropriate responses to operators, and analyzes the user's emotional state and issues warnings to operators based on the analysis results.

[0928] Specific examples

[0929] Example 1: Inquiry in Japanese

[0930] When a user makes an inquiry in Japanese, such as "The product delivery is delayed, what's going on?", the device sends this message to the server, which uses the Google Translate API to translate it into English as "The product delivery is delayed, what is happening?" IBM Watson Emotion Analysis then neutralizes the emotional expression, saying "The product delivery is delayed, please advise [current status]." The operator considers how to respond based on this message and replies, "We will investigate, so could you please wait a moment?" This reply is then translated again by the server into English as "We will investigate, please wait a moment," and sent to the user.

[0931] Example 2: Complaint in English

[0932] If a user complains in English, saying, "I'm extremely upset with your service. My order was not delivered on time," the device sends this content to the server, which then uses the Google Translate API to translate it into Japanese as, "I'm extremely unhappy with your service. My order was not delivered on time." The emotion analysis means neutralizes strong emotional expressions and notifies the operator, "A customer has inquired about their dissatisfaction with the service and a delayed order." The emotion engine analyzes the user's emotional state and alerts the operator, saying, "The customer is extremely unhappy. Please prioritize their response."

[0933] Example prompts for generative AI models

[0934] Below are some example prompts that explain the operation of programs on this system:

[0935] Please describe in detail the processing steps of your system for multilingual support and emotional complaint handling using specific hardware and software. Include specific behavior for each processing step. For example, please include the processing for when a customer inquires in Japanese, "My item is late. What's going on?", and when a customer in English complains, "I'm extremely upset with your service. My order was not delivered on time."

[0936] The system of the present invention provides an efficient and stress-free environment for both users and operators by centrally utilizing real-time translation, emotion analysis, and an emotion engine. Utilizing the emotion engine enables more advanced emotion recognition and the proposal of response methods, thereby achieving improved service quality.

[0937] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0938] Step 1:

[0939] A user sends a query from their own terminal. Specifically, they enter the query content in their own language and click the send button. At this time, the terminal sends the query content and the user's language information to the server. The input is the query text entered by the user, and the output is an HTTP request sent to the server.

[0940] Step 2:

[0941] The server passes the received inquiry to a real-time translation tool and translates it into the specified language. Specifically, it uses the Google Translate API to translate the received Japanese text into English. The input is the user's inquiry text, and the output is the translated text. At this stage, data processing involves converting the text into a different language.

[0942] Step 3:

[0943] The server passes the translated text to a sentiment analyzer to detect and neutralize emotional expressions. Specifically, it uses IBM Watson Emotion Analysis to convert strong emotional expressions into neutral ones. The input is the translated text, and the output is text with neutralized emotional elements. At this stage, the data operations are sentiment analysis and neutralization.

[0944] Step 4:

[0945] The server sends the neutralized query content to the operator's terminal. Specifically, the translated and neutralized query content is displayed on the operator's terminal. The input is the neutralized text, and the output is the content displayed on the operator's terminal. The server sends the data as an HTTP response, and the operator's terminal receives and displays it.

[0946] Step 5:

[0947] The user (operator) responds based on the displayed inquiry. Specifically, the operator enters a reply and presses the send button, which sends the reply from the terminal to the server. The input is the reply entered by the operator, and the output is an HTTP request sent to the server.

[0948] Step 6:

[0949] The server passes the operator's reply to a real-time translation tool, which translates it into the user's language. Specifically, it uses the Google Translate API to translate replies entered in Japanese into English. The input is the operator's reply text, and the output is the translated text. Data processing at this stage involves another language conversion.

[0950] Step 7:

[0951] The server then rechecks the translated reply content using sentiment analysis and neutralizes it. Specifically, it uses IBM Watson Emotion Analysis to check for emotional expressions in the reply content and neutralize it. The input is the translated reply text, and the output is the neutralized reply text. The data operations at this stage are sentiment analysis and neutralization.

[0952] Step 8:

[0953] The server sends the neutralized reply content to the user's terminal. The server sends the reply content with the emotional elements removed to the user's terminal as an HTTP response. The input is the neutralized reply text, and the output is the content displayed on the user's terminal.

[0954] Step 9:

[0955] The emotion engine monitors the user's emotional state and proposes the analysis results to the operator. Specifically, it uses the Affectiva SDK to detect the user's emotional state in real time and proposes a response. The input is the inquiry and reply, and the output is a response proposal to the operator.

[0956] Step 10:

[0957] The server accumulates data on inquiries and replies and stores it for analysis. The server stores all communication history in a database for later analysis and service improvement. The input is the inquiry and reply data, and the output is the accumulated database.

[0958] (Application example 2)

[0959] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0960] Conventional call centers and user support systems have limited multilingual support and emotion analysis capabilities, making it difficult to provide fast and effective global customer support. Furthermore, they lack the ability to properly recognize users' emotions and propose neutral responses, which can easily cause stress for both operators and users. If these issues continue for a long period of time, they could lead to a decline in service quality and customer satisfaction.

[0961] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0962] In this invention, the server includes knowledge processing means for translating text in real time, sentiment analysis means for performing sentiment analysis and neutralizing sentiment from the translated text, means for displaying on an operation device the inquiry content processed by the translation means and the sentiment analysis means, means for translating a reply entered on the operation device into the language of the terminal by the translation means and the sentiment analysis means and transmitting the neutralized text to the terminal device, means for monitoring the emotional state of the user in real time using a sentiment analysis function and proposing a response method, and means for quickly and effectively processing user feedback based on the response method. This enables advanced user support that combines multilingual real-time translation and sentiment analysis.

[0963] The "knowledge processing means for real-time text translation" is a function for instantly converting the content of a user's inquiry into another language.

[0964] The "sentiment analysis means for performing sentiment analysis and neutralizing emotions from translated text" is a function for analyzing emotional expressions in translated text and converting them into neutral expressions in order to reduce the intensity of the emotions.

[0965] The "means for displaying on an operating device" is a function for displaying the translated and sentiment-analyzed inquiry content on a device operated by an operator.

[0966] The "means for transmitting to the terminal device" is a function that translates and neutralizes the emotion again in order to transmit the reply entered by the operator to the user's terminal.

[0967] "Means of monitoring the user's emotional state in real time using an emotion analysis function and suggesting ways to respond" is a function that detects the user's emotions in real time and suggests the most appropriate way to respond to the operator based on that emotional state.

[0968] "Means for quickly and effectively processing feedback" refers to a function that instantly translates and analyzes sentiment on user feedback and quickly responds appropriately.

[0969] The present invention relates to a system for handling complaints in multiple languages ​​and emotionally. This system has the following specific configuration and operation.

[0970] System configuration

[0971] 1. Server:

[0972] The server is equipped with a knowledge processing means for translating text in real time, a sentiment analysis means for performing sentiment analysis to neutralize emotions from the translated text, and a sentiment analysis function. This server receives, processes, and transmits inquiries from users.

[0973] 2. Terminal:

[0974] The terminal consists of a device used by the user (e.g., a smartphone or smart glasses) and an operating device used by the operator. The terminal displays the translated and sentiment-analyzed content from the server and receives input from the operator.

[0975] 3. User:

[0976] The users include the users who make inquiries and the operators who respond to those inquiries.

[0977] Program processing

[0978] 1. Receiving inquiries

[0979] A user sends a query from their own device to the server, which includes the query content and the user's language information.

[0980] 2. Translation and Sentiment Analysis

[0981] The server translates the received query using a knowledge processing means that translates in real time.Then, a sentiment analysis means analyzes the translated content and neutralizes emotional expressions.

[0982] 3. Notification to operator and response

[0983] The processed inquiry is sent from the server to the operator's operating device. The operator responds based on the displayed information. The reply entered by the operator is translated and neutralized again before being sent to the user's device.

[0984] Hardware and software used

[0985] The system of the present invention mainly uses the following hardware and software.

[0986] Hardware: Server, smartphone, smart glasses, operator control device

[0987] Software: Python, Translators API, TextBlob

[0988] Data processing and calculation

[0989] 1. Real-time translation

[0990] The server translates the received text in real time using the Translators API.

[0991] 2. Sentiment analysis

[0992] Analyze the sentiment of text using TextBlob and neutralize the emotional expressions.

[0993] 3. Monitoring your emotional state

[0994] The server uses emotion analysis to monitor the user's emotional state in real time and present the operator with the most appropriate response.

[0995] Specific examples

[0996] Example 1: User gives feedback in English

[0997] The user sends feedback in English, saying, "I'm extremely upset with your service. My order was not delivered on time." The server receives this feedback and translates it in real time to "I'm extremely upset. My order was not delivered on time." It then performs sentiment analysis, which parses it as an expression of strong anger. The server then neutralizes the emotion and displays on the operator's control device, "A customer has inquired about their dissatisfaction with the service and a delayed order." The sentiment analysis function also issues a warning, such as, "The customer is extremely dissatisfied. Please prioritize their response." The operator enters an appropriate response, which the server translates and sends to the user's device, "We apologize for the inconvenience, we will check immediately."

[0998] Example of input prompt for generative AI model

[0999] "I'm extremely upset with your service. My order was not delivered on time."

[1000] Prompt: "Translate the above English sentence into Japanese and perform sentiment analysis to neutralize the emotional content. Then, suggest how to respond to the operator."

[1001] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1002] Step 1:

[1003] This is the action in which a user (consumer) inputs the content of a query from their own device and sends it to the server. The input is the user's query and its language information, and the output is the raw query data received by the server. The server then performs preliminary preparations to translate this query in real time. Specifically, the query and language information are saved in a database and used in the next step.

[1004] Step 2:

[1005] This is the operation in which the server translates the input inquiry content in real time. The input is the inquiry content and language information received in step 1, and the output is the translated text. The server calls the Translators API to translate from the original language into a common language (such as English) that the operator can understand. Specifically, the Translators API is used to translate the English sentence, "I'm extremely upset with your service. My order was not delivered on time." into Japanese.

[1006] Step 3:

[1007] The server performs sentiment analysis on the translated text and neutralizes emotional expressions as necessary. The input is the translated text obtained in step 2, and the output is the neutralized text. The server uses TextBlob to perform sentiment analysis and detect emotional expressions. For example, it converts a sentence that expresses strong anger into a neutral expression such as "We apologize for the inconvenience."

[1008] Step 4:

[1009] This is the operation in which the server notifies the operator based on emotion analysis. The input is the neutralized text obtained in step 3, and the output is a notification message displayed on the operator's operating device. The server uses its emotion analysis function to evaluate the user's emotional state and sends a notification to the operator, such as "The customer is very dissatisfied. Please handle this as a priority."

[1010] Step 5:

[1011] This is the operation in which the operator inputs a response on the operating device and sends the content to the server. The input is the text of the operator's response, and the output is the response data from the operator received by the server. The operator inputs an appropriate response based on the neutralized inquiry content and clicks the send button.

[1012] Step 6:

[1013] The server then translates the operator's response again and performs sentiment analysis. The input is the operator's response text received in step 5, and the output is the translated and neutralized reply text. The server again uses the Translators API and TextBlob to translate the operator's reply into the user's language and neutralize it to avoid emotional overload.

[1014] Step 7:

[1015] This is the operation in which the server finally sends the translated and sentiment-analyzed reply text to the user's device. The input is the neutralized reply text obtained in step 6, and the output is the reply message displayed on the user's device. The server sends the translated and neutralized text to the user's device, providing a response in a way that is less stressful for the user.

[1016] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1017] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1018] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1019] [Fourth embodiment]

[1020] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1021] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1022] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1023] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1024] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1025] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1026] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1027] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1028] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1029] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1031] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1033] The present invention relates to a system for multilingual and emotional complaint handling, which includes a real-time translation means and a sentiment analysis means, enabling fast and effective responses in global call centers.

[1034] Program processing

[1035] The main components of this system are the server, terminals, and users (operators and users). The roles and operations of each are explained below.

[1036] 1. Receiving inquiries

[1037] The user sends a support request from their device, which includes the content of the inquiry and the language they are using. The device then sends this information to the server.

[1038] 2. Translation and Sentiment Analysis

[1039] The server then passes the received inquiry to a real-time translation unit, which translates the user's language into a language the operator can understand. The emotion analysis unit then analyzes the translated content and converts emotional expressions into neutral ones, allowing the operator to respond calmly based on the information.

[1040] 3. Notification to Operator

[1041] The translated and neutralized query content is sent from the server to the operator's terminal, where it is displayed in an easy-to-understand format.

[1042] 4. Operator response and reply

[1043] The user (operator) responds based on the displayed content and enters a reply, which is then sent from the terminal to the server.

[1044] 5. Reply translation and notification

[1045] The server then passes the operator's reply to the real-time translation tool, which translates it into the user's language. The sentiment analysis tool then converts the translated reply into a neutral expression and sends the final answer to the user's device, ensuring the user receives a calm and clear response.

[1046] Specific examples

[1047] Example 1: When a user makes an inquiry in Japanese

[1048] The user makes an inquiry in Japanese, saying, "The product delivery is delayed. What's going on?" The terminal sends this inquiry to the server, which uses real-time translation means to translate it into English as, "The product delivery is delayed, what is happening?" The emotion analysis means then analyzes this content and neutralizes the emotional expression, "It's delayed, what's going on?" The operator's terminal then displays the neutralized message, "The product delivery is delayed, please advise [current status]."

[1049] The operator replies, "We will investigate, please wait a moment," and sends the reply from the device to the server. The server translates this reply from Japanese to English, saying, "We will investigate, please wait a moment," and then uses sentiment analysis to confirm that it is neutral. The final reply is sent to the user's device.

[1050] Example 2: When a customer makes a complaint in English

[1051] If a user complains in English, "I'm extremely upset with your service. My order was not delivered on time," the device sends this content to the server, which then uses a translation tool to translate it into Japanese as "I'm extremely upset with your service. My order was not delivered on time." The sentiment analysis tool neutralizes the strong expression "I'm dissatisfied," and displays to the operator, "A customer has inquired about their dissatisfaction with the service and the delay in their order."

[1052] The operator replies, "We apologize for the inconvenience, we will check immediately." The server then translates this into English and sends it to the user's device as, "We apologize for the inconvenience, we will check immediately."

[1053] The system of the present invention achieves multilingual support through real-time translation and sentiment analysis, providing a stress-free environment for both operators and users.

[1054] The processing flow will be explained below.

[1055] Step 1:

[1056] A user sends a support request from their own device. This request contains the inquiry content and the user's language information. For example, a user enters a message in Japanese saying, "My product is late. What's going on?" and clicks the send button.

[1057] Step 2:

[1058] The terminal receives the user's input information, sends the query content and the user's language information to the server, packages the user's language and text content, and makes a request to the server.

[1059] Step 3:

[1060] The server passes the received inquiry to an AI translation engine, which translates the user's language into a language the operator can understand. For example, the Japanese phrase "The product delivery is delayed, what's happening?" is translated into English as "The product delivery is delayed, what is happening?"

[1061] Step 4:

[1062] The server then sends the translation results to an AI sentiment analysis engine, which analyzes the translated query, detects any emotional expressions, and converts them into neutral phrases. For example, it neutralizes the emotional expression "what is happening?" to "please advise [current status]."

[1063] Step 5:

[1064] The server sends the translated and neutralized inquiry to the operator terminal. The operator terminal displays the translated and neutralized inquiry. The operator is shown a message saying, "A customer has inquired about a delay in the delivery of their product."

[1065] Step 6:

[1066] The user (operator) responds based on the displayed inquiry and enters a reply. For example, they might type, "I will look into this, so could you please wait a moment?" and click the send button.

[1067] Step 7:

[1068] The terminal sends the operator's input to the server and requests the operator's reply text from the server.

[1069] Step 8:

[1070] The server then passes the operator's reply to the AI ​​translation engine again, which translates it from the operator's language to the user's language. For example, the Japanese phrase "We will investigate, so please wait a moment." is translated into English as "We will investigate, please wait a moment."

[1071] Step 9:

[1072] The server then sends the translated reply back to the AI ​​sentiment analysis engine to neutralize the emotional content. If the content is already neutral, no changes are made.

[1073] Step 10:

[1074] The server then sends the translated and neutralized reply to the user's device. The server then sends the translated reply, "We'll look into this, so could you please wait a moment?"

[1075] Step 11:

[1076] The terminal (user) displays the translated reply received from the server on its screen. The user's screen displays "We will look into this, could you please wait a moment?", and the response is complete.

[1077] The above steps constitute a system that enables multilingual and emotion-neutral communication between users and operators.

[1078] Example 1

[1079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1080] Today's global call centers are required to handle inquiries from users quickly and accurately. However, due to issues such as diverse languages ​​and emotional expressions, it is often difficult for operators to respond efficiently. It is also not easy to respond calmly and neutrally to emotional complaints. This increases the burden on operators and risks a decline in response quality. There is a need for a system that can solve these issues, improve the efficiency of inquiry responses at global call centers, and improve response quality.

[1081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1082] In this invention, the server includes means for receiving inquiries from user terminals, AI translation means for translating text in real time, AI sentiment analysis means for performing sentiment analysis and neutralizing the sentiment of the translated text, means for displaying the inquiry content processed by the translation means and the sentiment analysis means on an operator terminal, and means for translating a reply entered by an operator into the user's language by the translation means and the sentiment analysis means and sending the neutralized text to the user terminal. This enables inquiries to be processed quickly and accurately in a global call center, providing a stress-free environment for both operators and users.

[1083] "User" refers to an individual or company that uses the system to make an inquiry.

[1084] A "terminal" is a device (e.g., PC, smartphone, tablet) used by a user or operator to access the system.

[1085] A "server" is a central computer that manages the processing of the entire system and sends and receives various data.

[1086] "AI translation mechanism" refers to a mechanism that utilizes artificial intelligence technology to translate text into another language in real time.

[1087] An "AI sentiment analysis tool" is a mechanism that uses artificial intelligence technology to analyze and neutralize emotional expressions in text.

[1088] "Inquiry content" refers to the text information of questions or complaints sent by users through the system.

[1089] An "operator" is a system user who responds to inquiries from users and provides necessary replies.

[1090] A "neutralized text" is a text that has been transformed into a calm and objective expression, with emotional expressions suppressed.

[1091] This invention is a system that realizes multilingual support and emotional complaint handling in global call centers. This system is mainly composed of three components: a server, terminals, and users (operators and customers).

[1092] System Overview

[1093] Receiving inquiries

[1094] A user sends a support request from their device (PC, smartphone, etc.). This request includes the inquiry and the language used by the user. The device then sends this information to the server.

[1095] Translation and Sentiment Analysis

[1096] The server passes the received inquiry to a real-time translation tool (e.g., Google Translate API) to translate the user's language into a language the operator can understand. The translated content is then analyzed using a sentiment analysis tool (e.g., IBM Watson Natural Language Understanding) to convert emotional expressions into neutral ones, allowing the operator to respond calmly based on the information.

[1097] Operator notification

[1098] The translated and neutralized inquiry is sent from the server to the operator's terminal, where the inquiry is displayed in an easy-to-understand format through dedicated customer support software.

[1099] Operator response and reply

[1100] The user (operator) responds based on the displayed information and enters a reply, which is then sent from the terminal to the server.

[1101] Reply translation and notification

[1102] The server then passes the operator's reply to the real-time translation unit, which translates it into the user's language. The sentiment analysis unit then converts the translated reply into a neutral expression, and the final answer is sent to the user's device. This allows the user to receive a calm and clear response.

[1103] Specific examples

[1104] Example 1: When a user makes an inquiry in Japanese

[1105] The user makes an inquiry in Japanese, saying, "The product delivery is delayed. What's going on?" The device sends this inquiry to the server. The server uses real-time translation means to translate this into English as, "The product delivery is delayed, what is happening?" The sentiment analysis means then analyzes this content and displays a neutralized response, "The product delivery is delayed, please advise [current status]." on the operator's device. The operator replies, "We will investigate, could you please wait a moment?" and sends this response from the device to the server. The server then translates this response from Japanese to English, saying, "We will investigate, please wait a moment." The sentiment analysis means then confirms the neutral expression, and the final reply is sent to the user's device.

[1106] Example 2: When a customer makes a complaint in English

[1107] A user complains in English, saying, "I'm extremely upset with your service. My order was not delivered on time." The device sends this content to the server. The server uses a translation tool to translate this into Japanese as "I'm extremely unhappy with your service. My order was not delivered on time." The sentiment analysis tool neutralizes the strong expression "unhappy" and displays to the operator, "A customer has inquired about their dissatisfaction with the service and a delayed order." The operator replies, "We apologize, we will check immediately." The server translates this content into English and sends it to the user's device as "We apologize for the inconvenience, we will check immediately."

[1108] Prompt Sentence Examples

[1109] Here are some examples of prompts for generative AI models:

[1110] "Please explain a scenario in which a user makes an inquiry in Japanese and an operator responds. Please also include the specific process flow and the functions used (translation methods, sentiment analysis methods)."

[1111] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1112] Step 1:

[1113] A user sends a support request using their own device. As input, they enter text data containing the inquiry and the user's language information into the device. The device then sends this data to the server as an HTTP request. The output is the inquiry data transferred to the server.

[1114] Step 2:

[1115] The server passes the received query data to a real-time translation method (e.g., Google Translate API). The input is the query text sent by the user and its language information. The server passes this information to the API for translation processing. The output is the translated text returned to the server.

[1116] Step 3:

[1117] The server passes the translated text to a sentiment analyzer (e.g., IBM Watson Natural Language Understanding). The input is the translated query text. The server sends the text to a sentiment analysis API, which analyzes the sentiment in the text. The output is text with neutralized emotional expressions.

[1118] Step 4:

[1119] The server sends the neutralized text to the operator's terminal. The input is the query data processed by the sentiment analysis means. The server transfers this data to the operator's terminal using REST API or WebSocket. The output is the neutralized query content displayed on the operator's terminal.

[1120] Step 5:

[1121] The user (operator) considers how to respond based on the displayed content and enters a reply. As input, the operator enters the proposed response as text into the terminal. The terminal then sends this reply data back to the server. The output is the reply data transferred to the server.

[1122] Step 6:

[1123] The server passes the operator's reply to a real-time translation means. The input is the reply text sent by the operator. The server sends the reply text to a translation means, which translates the reply into the user's language. The output is the translated reply text.

[1124] Step 7:

[1125] The server passes the translated reply to the sentiment analyzer again to confirm the neutralized expression. The input is the translated reply text. The server passes this text to the sentiment analysis API for verification. The output is the reply text with the neutralized expression confirmed.

[1126] Step 8:

[1127] The server sends the final answer to the user's device. The input is the reply text that has been checked for neutrality. The server transfers this text to the user's device using a REST API or WebSocket. The output is the final reply that is displayed on the user's device.

[1128] (Application example 1)

[1129] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1130] Conventional customer service systems have difficulty dealing with customers who speak different languages ​​and have different cultural backgrounds, and have had particular problems with incomplete multilingual support and emotional complaint handling. Furthermore, in brick-and-mortar stores, staff were not equipped with an environment where they could instantly provide multilingual support, making it difficult to provide prompt and appropriate service.

[1131] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1132] In this invention, the server includes an artificial intelligence translation means for translating text in real time, an artificial intelligence emotion analysis means for performing emotion analysis and neutralizing the emotion from the translated text, a means for displaying the inquiry content processed by the translation means and the emotion analysis means on a staff terminal, a means for translating a reply entered by a staff member into the user's language by the translation means and the emotion analysis means and sending the neutralized text to the user terminal, and a means for displaying the inquiry content and the reply content on an augmented reality device used as a staff terminal. This enables multilingual and emotional complaint handling to be performed quickly and effectively even in physical stores, thereby improving customer satisfaction.

[1133] "Artificial intelligence translation means for real-time text translation" refers to a device that uses artificial intelligence to instantly convert the content of an inquiry into a different language.

[1134] The "artificial intelligence emotion analysis means for performing emotion analysis and neutralizing emotions from translated text" is a device that uses artificial intelligence to analyze emotional expressions in translated text and convert them into calm expressions.

[1135] "Staff terminal" means an electronic device used by customer service staff to receive and enter information.

[1136] "User terminal" refers to an electronic device used by a service user to receive support or enter inquiries.

[1137] An "augmented reality device" is a device worn by staff that can overlay virtual information onto the actual environment.

[1138] "Inquiry content" refers to the content of questions, requests, and complaints made by users to customer service.

[1139] A "neutralized text" is text that has been transformed from emotional expression into calm, objective expression.

[1140] A "translation tool" is a device or software for converting text from one language to another.

[1141] A "sentiment analyzer" is a device or software for analyzing and identifying emotional elements in text.

[1142] The present invention is a system for multilingual support and emotional complaint handling in brick-and-mortar stores. This system performs real-time translation and sentiment analysis between customers and customer service staff (hereinafter referred to as staff), enabling staff to respond appropriately and quickly. Specific embodiments are described below.

[1143] Overall system configuration

[1144] 1. Server and Cloud Infrastructure

[1145] The server includes the hardware and software necessary to implement real-time translation and sentiment analysis. The main software used is the Google Translator API, which provides translation functionality, the TextBlob library for sentiment analysis, and the SpeechRecognition library for voice recognition.

[1146] 2. Staff terminals and augmented reality devices

[1147] Smart glasses (e.g., Google Glass) are used as staff terminals, and these devices display inquiries and replies to staff in real time.

[1148] 3. User Device

[1149] A user (customer) makes an inquiry using a personal device such as a smartphone.

[1150] Data processing and calculation

[1151] The server processes and calculates data in the following procedure.

[1152] 1. Voice Recognition

[1153] When a user makes a voice inquiry using their smartphone, the voice data is transmitted to the staff member's smart glasses, which then send the voice data to the server, which then converts the voice data into text using the SpeechRecognition library.

[1154] 2. Real-time translation

[1155] The server translates the text obtained by speech recognition in real time using the Google Translator API. For example, it converts a Japanese inquiry into English.

[1156] 3. Sentiment Analysis and Neutralization

[1157] The server performs sentiment analysis on the translated text using the "TextBlob" library. If the sentiment is strong, it converts certain keywords (e.g., "very dissatisfied") into neutral expressions (e.g., "concerned").

[1158] 4. Staff Display

[1159] The neutralized text is processed on the server and displayed on the staff's smart glasses, allowing them to respond appropriately to the customer.

[1160] 5. Reply Translation and Notification

[1161] The reply entered by the staff is sent back to the server, where it is translated and sentiment-analyzed. The translated text is neutralized and then sent to the customer's smartphone.

[1162] Specific use cases

[1163] If a customer asks in Japanese, "The product delivery is delayed, what's going on?", this speech is converted into text on the server and translated into English. The translated text, "The product delivery is delayed, what is happening?", undergoes sentiment analysis and neutralization (e.g., "The product delivery is delayed, please advise [current status].") and is displayed on the staff member's smart glasses. The staff member responds, "We will investigate, please wait a moment," and the reply is also translated into English and notifies the customer, "We will investigate, please wait a moment."

[1164] Example prompt sentence:

[1165] Analyze the following sentences and convert them into neutral expressions.

[1166] Text: "I'm extremely upset with your service. My order was not delivered on time."

[1167] Neutral: "I'm concerned about the delay in my order delivery. Please advise on the current status."

[1168] As described above, the present invention aims to realize multilingual support and emotional complaint handling, thereby improving the quality of customer service in physical stores.

[1169] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1170] Step 1:

[1171] A user makes a voice inquiry using a smartphone. This voice data is collected through the smartphone's microphone and saved as digital data. The input is voice data, and the output is digital voice data.

[1172] Step 2:

[1173] The user's smartphone sends voice data to the server, which converts it into text using the "SpeechRecognition" library. The input is digital voice data, and the output is text data. This conversion transforms the voice data into concrete text information.

[1174] Step 3:

[1175] The server translates the text data in real time using the Google Translator API. For example, Japanese text is converted into English text. The input is Japanese text data, and the output is English text data. This translation converts the data into a language that the staff can understand.

[1176] Step 4:

[1177] The server performs sentiment analysis on the translated English text data using the "TextBlob" library. The sentiment analysis results in positive, negative, and neutral sentiment. The input is the English text data, and the output is the sentiment analysis results and corrected text data. Sentiment analysis neutralizes expressions as needed.

[1178] Step 5:

[1179] The server sends neutralized text data with reduced emotion to the augmented reality device (smart glasses). The staff member's smart glasses receive this data and display it on their screen. The input is the neutralized text data, and the output is the display on the smart glasses.

[1180] Step 6:

[1181] The staff member checks the neutralized text through the smart glasses and types an appropriate reply to the customer. The reply is recorded as audio by the smart glasses' microphone and sent to the server as text data. The input is audio data and the output is text data.

[1182] Step 7:

[1183] The server again translates the staff member's reply text data into the customer's language (e.g., Japanese) using the "Google Translator API." It then performs sentiment analysis on the translated text using the "TextBlob" library to confirm neutralization. The input is the staff member's English text data, and the output is neutralized Japanese text data.

[1184] Step 8:

[1185] The server sends the neutralized Japanese text data to the user's smartphone. The user's smartphone receives this data and displays it on its screen. The input is the neutralized Japanese text data, and the output is what is displayed on the smartphone's screen.

[1186] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1187] The present invention relates to a system for multilingual and emotional complaint handling, which includes a real-time translation means, a sentiment analysis means, and an emotion engine, enabling fast and effective responses in global call centers.

[1188] Program processing

[1189] The main components of this system are the server, terminals, and users (operators and users). The roles and operations of each are explained below.

[1190] 1. Receiving inquiries

[1191] A user sends a support request from their own device. This request contains the inquiry content and the user's language information. For example, a user enters a message in Japanese saying, "My product is late. What's going on?" and clicks the send button.

[1192] 2. Translation and Sentiment Analysis

[1193] The server passes the received inquiry to a real-time translation means, which translates it from the user's language into a language the operator can understand. For example, the Japanese phrase "The product delivery is delayed, what's going on?" is translated into English as "The product delivery is delayed, what is happening?". Next, the sentiment analysis means analyzes the translated content and converts emotional expressions into neutral expressions. For example, the emotional expression "what is happening?" is neutralized to "please advise [current status]."

[1194] 3. Notification to Operator

[1195] The translated and neutralized inquiry is sent from the server to the operator's terminal. The operator's terminal displays the translated and neutralized inquiry. For example, the operator might see, "A customer has inquired about a delay in the arrival of their product."

[1196] 4. Operator response and reply

[1197] The user (operator) responds based on the displayed inquiry and enters a reply. For example, they might type, "I will look into this, so could you please wait a moment?" and click the send button.

[1198] 5. Reply translation and notification

[1199] The server then passes the operator's reply back to the real-time translation means, which translates it from the operator's language into the user's language. For example, the Japanese phrase "We will investigate, so please wait a moment." is translated into English as "We will investigate, please wait a moment." The emotion analysis means then checks for neutral expressions. The emotion engine monitors the user's emotional state in real time, neutralizes the emotional content, and sends it to the user's device.

[1200] 6. Use of Emotion Engines

[1201] The emotion engine recognizes the user's emotional state and suggests how the operator should respond based on that. For example, if the emotion engine recognizes that the user is angry, it will warn the operator that "the customer is angry, so a polite response is required." In addition, based on emotion analysis, the frequency and patterns of emotional complaints are analyzed and data is accumulated. This data is used for future service improvements and training.

[1202] Specific examples

[1203] Example 1: When a user makes an inquiry in Japanese

[1204] The user makes an inquiry in Japanese, saying, "The product delivery is delayed. What's going on?" The device sends this inquiry to the server, which uses real-time translation to translate it into English as, "The product delivery is delayed, what is happening?" The emotion analysis device analyzes this content and neutralizes the emotional expression, "It's delayed, what's going on?" The neutralized message, "The product delivery is delayed, please advise [current status]," is displayed on the operator's device. The emotion engine recognizes the user's emotions and alerts the operator.

[1205] The operator replies, "We will investigate, so please wait a moment," and sends the reply from the device to the server. The server translates this reply from Japanese to English and sends it to the user's device as "We will investigate, please wait a moment." The emotion engine then rechecks the emotion of the reply, confirming that it is neutral before sending it.

[1206] Example 2: When a customer makes a complaint in English

[1207] If a user complains in English, "I'm extremely upset with your service. My order was not delivered on time," the device sends this content to the server, which then uses a translation tool to translate it into Japanese as "I'm extremely upset with your service. My order was not delivered on time." The emotion analysis tool neutralizes the strong expression "dissatisfied" and displays to the operator, "A customer has inquired about their dissatisfaction with the service and a delayed order." The emotion engine recognizes the user's strong anger and alerts the operator, "The customer is extremely dissatisfied. Please handle this as a priority."

[1208] The operator replies, "We apologize for the inconvenience, we will check immediately," and the server translates this into English and sends it to the user's device as, "We apologize for the inconvenience, we will check immediately." The emotion engine then rechecks the emotion of the reply, making sure it is neutral before sending it.

[1209] The system of the present invention realizes multilingual support through real-time translation, emotion analysis, and an emotion engine, providing a stress-free environment for both operators and users. The emotion engine enables more advanced emotion recognition and suggests ways to respond, improving service quality.

[1210] The processing flow will be explained below.

[1211] Step 1:

[1212] A user sends a support request from their own device. This request contains the inquiry content and the user's language information. For example, a user enters a message in Japanese saying, "My product is late. What's going on?" and clicks the send button.

[1213] Step 2:

[1214] The terminal receives the user's input information and sends the query content and the user's language information to the server. The terminal transfers the text content and language information to the server in packet format.

[1215] Step 3:

[1216] The server passes the received inquiry as a request to the AI ​​translation engine. The AI ​​translation engine translates the user's language into a language the operator can understand. For example, the Japanese phrase "The product delivery is delayed, what's happening?" is translated into English as "The product delivery is delayed, what is happening?"

[1217] Step 4:

[1218] The server sends the translation results to an AI sentiment analysis engine, which analyzes the translated query, detects emotional expressions, and converts them into neutral expressions. For example, it neutralizes the emotional expression "what is happening?" to "please advise [current status]."

[1219] Step 5:

[1220] The server sends the translated and neutralized query content to the operator terminal, which checks the displayed content and starts responding to the user based on it.

[1221] Step 6:

[1222] The emotion engine analyzes the emotional state of the received text and monitors the user's emotional state in real time. The emotion engine determines whether the user is angry or dissatisfied, and if strong emotions are detected, a warning message such as "The customer is angry, so polite response is required" is displayed on the operator terminal.

[1223] Step 7:

[1224] The user (operator) responds based on the displayed inquiry and enters a reply. For example, they might type, "I will look into this, so could you please wait a moment?" and click the send button.

[1225] Step 8:

[1226] The terminal sends the operator's input to the server. The server then passes the reply back to the AI ​​translation engine, which translates it from the operator's language into the user's language. For example, the Japanese phrase "We will investigate, so could you please wait a moment?" is translated into English as "We will investigate, please wait a moment."

[1227] Step 9:

[1228] The server then sends the translated reply back to the AI ​​sentiment analysis engine to neutralize the emotional content. The sentiment analysis engine analyzes the reply and neutralizes it as necessary. Once this process is complete, the server sends the final translated and neutralized reply to the user's device.

[1229] Step 10:

[1230] The terminal (user) displays the translated reply received from the server on its screen and notifies the user. The user confirms the reply displayed on the screen and completes the inquiry.

[1231] Through the above steps, a system is realized in which communication with users is carried out in multiple languages ​​and in an emotion-neutral manner, allowing operators to respond quickly and effectively.

[1232] Example 2

[1233] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1234] To minimize communication delays and misunderstandings caused by language barriers and emotional complaint handling in global call centers, and to achieve effective and neutral communication between users and operators.

[1235] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for accepting inquiry content and translating the text in real time, means for sentiment-analyzing the translated text and neutralizing emotional expressions, means for displaying the translated and neutralized inquiry content on an operator terminal, means for translating a reply entered by the operator in real time, sentiment-analyzing and neutralizing the reply, and sending it to the user's terminal, means for recognizing, analyzing, and monitoring the user's emotional state and suggesting a response method to the operator, and means for accumulating data on the inquiry content and the reply content and storing it for analysis. This minimizes communication delays and misunderstandings due to language barriers and emotional responses to complaints, enabling effective and neutral communication between users and operators.

[1236] The "means for receiving inquiry content" is a function for electronically receiving inquiry content from a user and transferring that information to the server.

[1237] "Means for translating text in real time" refers to the ability to instantly convert received text into another language.

[1238] "Means for sentiment analysis of translated text and neutralizing emotional expressions" is a function that analyzes translated text to detect emotional elements and converts them into neutral expressions.

[1239] The "means for displaying the translated and neutralized inquiry content on the operator terminal" is a function for visually displaying the processed inquiry content on the operator terminal.

[1240] "Means for translating replies entered by operators in real time, analyzing and neutralizing emotions, and sending them to the user's device" refers to a function that instantly translates the operator's response into another language, neutralizes emotional elements, and then sends it to the user.

[1241] "Means for recognizing, analyzing, and monitoring the user's emotional state and suggesting appropriate ways to respond to the operator" refers to a function that detects and analyzes the user's emotions and suggests appropriate ways to respond to the operator based on that information.

[1242] "Means for accumulating data on inquiry and reply content and storing it for analysis" refers to a function that stores the history of all inquiries and replies in a database so that it can be used later for analysis and improvement.

[1243] This invention relates to a system for effectively handling multilingual and emotional complaints in global call centers. This system includes a real-time translation means, a sentiment analysis means, and an emotion engine, and automates the entire process from processing inquiries to replying.

[1244] Hardware and software used

[1245] The main components of the system are the server, the terminal, and the user (operator and user). This system uses the following hardware and software:

[1246] Real-time translation means: As an example, we will use Google Translate API, which is a software function for instantly translating inquiries and replies into different languages.

[1247] Sentiment analysis tools: As an example, we will use IBM Watson Emotion Analysis, which is a function for analyzing and neutralizing the emotional components of received text.

[1248] Emotion engine: Using the Affectiva SDK as an example, this is an engine that monitors the user's emotional state in real time and suggests ways to respond to the operator based on the analysis results.

[1249] The entire process

[1250] 1. Receiving inquiries

[1251] The user (consumer) inputs the inquiry content from their own device and sends it. For example, a user might input "The arrival of the product is delayed. What's going on?" in Japanese and click the send button.

[1252] 2. Translation and Sentiment Analysis

[1253] The server translates the received query in real time using the Google Translate API, while neutralizing emotional expressions using IBM Watson Emotion Analysis.

[1254] 3. Notification to Operator

[1255] The translated and neutralized inquiry is sent from the server to the operator's terminal, where the translated inquiry is displayed and the operator can respond appropriately.

[1256] 4. Operator response and reply

[1257] The operator types a reply based on the displayed inquiry and sends it to the server. For example, they might type, "I'll look into it, so could you please wait a moment?"

[1258] 5. Reply translation and notification

[1259] The server then translates the operator's reply using the Google Translate API, adapting it to the user's language, and uses sentiment analysis tools to recheck and neutralize the emotional content of the reply.

[1260] 6. Use of Emotion Engines

[1261] The emotion engine accumulates the contents of inquiries and replies, suggests appropriate responses to operators, and analyzes the user's emotional state and issues warnings to operators based on the analysis results.

[1262] Specific examples

[1263] Example 1: Inquiry in Japanese

[1264] When a user makes an inquiry in Japanese, such as "The product delivery is delayed, what's going on?", the device sends this message to the server, which uses the Google Translate API to translate it into English as "The product delivery is delayed, what is happening?" IBM Watson Emotion Analysis then neutralizes the emotional expression, saying "The product delivery is delayed, please advise [current status]." The operator considers how to respond based on this message and replies, "We will investigate, so could you please wait a moment?" This reply is then translated again by the server into English as "We will investigate, please wait a moment," and sent to the user.

[1265] Example 2: Complaint in English

[1266] If a user complains in English, saying, "I'm extremely upset with your service. My order was not delivered on time," the device sends this content to the server, which then uses the Google Translate API to translate it into Japanese as, "I'm extremely unhappy with your service. My order was not delivered on time." The emotion analysis means neutralizes strong emotional expressions and notifies the operator, "A customer has inquired about their dissatisfaction with the service and a delayed order." The emotion engine analyzes the user's emotional state and alerts the operator, saying, "The customer is extremely unhappy. Please prioritize their response."

[1267] Example prompts for generative AI models

[1268] Below are some example prompts that explain the operation of programs on this system:

[1269] Please describe in detail the processing steps of your system for multilingual support and emotional complaint handling using specific hardware and software. Include specific behavior for each processing step. For example, please include the processing for when a customer inquires in Japanese, "My item is late. What's going on?", and when a customer in English complains, "I'm extremely upset with your service. My order was not delivered on time."

[1270] The system of the present invention provides an efficient and stress-free environment for both users and operators by centrally utilizing real-time translation, emotion analysis, and an emotion engine. Utilizing the emotion engine enables more advanced emotion recognition and the proposal of response methods, thereby achieving improved service quality.

[1271] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1272] Step 1:

[1273] A user sends a query from their own terminal. Specifically, they enter the query content in their own language and click the send button. At this time, the terminal sends the query content and the user's language information to the server. The input is the query text entered by the user, and the output is an HTTP request sent to the server.

[1274] Step 2:

[1275] The server passes the received inquiry to a real-time translation tool and translates it into the specified language. Specifically, it uses the Google Translate API to translate the received Japanese text into English. The input is the user's inquiry text, and the output is the translated text. At this stage, data processing involves converting the text into a different language.

[1276] Step 3:

[1277] The server passes the translated text to a sentiment analyzer to detect and neutralize emotional expressions. Specifically, it uses IBM Watson Emotion Analysis to convert strong emotional expressions into neutral ones. The input is the translated text, and the output is text with neutralized emotional elements. At this stage, the data operations are sentiment analysis and neutralization.

[1278] Step 4:

[1279] The server sends the neutralized query content to the operator's terminal. Specifically, the translated and neutralized query content is displayed on the operator's terminal. The input is the neutralized text, and the output is the content displayed on the operator's terminal. The server sends the data as an HTTP response, and the operator's terminal receives and displays it.

[1280] Step 5:

[1281] The user (operator) responds based on the displayed inquiry. Specifically, the operator enters a reply and presses the send button, which sends the reply from the terminal to the server. The input is the reply entered by the operator, and the output is an HTTP request sent to the server.

[1282] Step 6:

[1283] The server passes the operator's reply to a real-time translation tool, which translates it into the user's language. Specifically, it uses the Google Translate API to translate replies entered in Japanese into English. The input is the operator's reply text, and the output is the translated text. Data processing at this stage involves another language conversion.

[1284] Step 7:

[1285] The server then rechecks the translated reply content using sentiment analysis and neutralizes it. Specifically, it uses IBM Watson Emotion Analysis to check for emotional expressions in the reply content and neutralize it. The input is the translated reply text, and the output is the neutralized reply text. The data operations at this stage are sentiment analysis and neutralization.

[1286] Step 8:

[1287] The server sends the neutralized reply content to the user's terminal. The server sends the reply content with the emotional elements removed to the user's terminal as an HTTP response. The input is the neutralized reply text, and the output is the content displayed on the user's terminal.

[1288] Step 9:

[1289] The emotion engine monitors the user's emotional state and proposes the analysis results to the operator. Specifically, it uses the Affectiva SDK to detect the user's emotional state in real time and proposes a response. The input is the inquiry and reply, and the output is a response proposal to the operator.

[1290] Step 10:

[1291] The server accumulates data on inquiries and replies and stores it for analysis. The server stores all communication history in a database for later analysis and service improvement. The input is the inquiry and reply data, and the output is the accumulated database.

[1292] (Application example 2)

[1293] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1294] Conventional call centers and user support systems have limited multilingual support and emotion analysis capabilities, making it difficult to provide fast and effective global customer support. Furthermore, they lack the ability to properly recognize users' emotions and propose neutral responses, which can easily cause stress for both operators and users. If these issues continue for a long period of time, they could lead to a decline in service quality and customer satisfaction.

[1295] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1296] In this invention, the server includes knowledge processing means for translating text in real time, sentiment analysis means for performing sentiment analysis and neutralizing sentiment from the translated text, means for displaying on an operation device the inquiry content processed by the translation means and the sentiment analysis means, means for translating a reply entered on the operation device into the language of the terminal by the translation means and the sentiment analysis means and transmitting the neutralized text to the terminal device, means for monitoring the emotional state of the user in real time using a sentiment analysis function and proposing a response method, and means for quickly and effectively processing user feedback based on the response method. This enables advanced user support that combines multilingual real-time translation and sentiment analysis.

[1297] The "knowledge processing means for real-time text translation" is a function for instantly converting the content of a user's inquiry into another language.

[1298] The "sentiment analysis means for performing sentiment analysis and neutralizing emotions from translated text" is a function for analyzing emotional expressions in translated text and converting them into neutral expressions in order to reduce the intensity of the emotions.

[1299] The "means for displaying on an operating device" is a function for displaying the translated and sentiment-analyzed inquiry content on a device operated by an operator.

[1300] The "means for transmitting to the terminal device" is a function that translates and neutralizes the emotion again in order to transmit the reply entered by the operator to the user's terminal.

[1301] "Means of monitoring the user's emotional state in real time using an emotion analysis function and suggesting ways to respond" is a function that detects the user's emotions in real time and suggests the most appropriate way to respond to the operator based on that emotional state.

[1302] "Means for quickly and effectively processing feedback" refers to a function that instantly translates and analyzes sentiment on user feedback and quickly responds appropriately.

[1303] The present invention relates to a system for handling complaints in multiple languages ​​and emotionally. This system has the following specific configuration and operation.

[1304] System configuration

[1305] 1. Server:

[1306] The server is equipped with a knowledge processing means for translating text in real time, a sentiment analysis means for performing sentiment analysis to neutralize emotions from the translated text, and a sentiment analysis function. This server receives, processes, and transmits inquiries from users.

[1307] 2. Terminal:

[1308] The terminal consists of a device used by the user (e.g., a smartphone or smart glasses) and an operating device used by the operator. The terminal displays the translated and sentiment-analyzed content from the server and receives input from the operator.

[1309] 3. User:

[1310] The users include the users who make inquiries and the operators who respond to those inquiries.

[1311] Program processing

[1312] 1. Receiving inquiries

[1313] A user sends a query from their own device to the server, which includes the query content and the user's language information.

[1314] 2. Translation and Sentiment Analysis

[1315] The server translates the received query using a knowledge processing means that translates in real time.Then, a sentiment analysis means analyzes the translated content and neutralizes emotional expressions.

[1316] 3. Notification to operator and response

[1317] The processed inquiry is sent from the server to the operator's operating device. The operator responds based on the displayed information. The reply entered by the operator is translated and neutralized again before being sent to the user's device.

[1318] Hardware and software used

[1319] The system of the present invention mainly uses the following hardware and software.

[1320] Hardware: Server, smartphone, smart glasses, operator control device

[1321] Software: Python, Translators API, TextBlob

[1322] Data processing and calculation

[1323] 1. Real-time translation

[1324] The server translates the received text in real time using the Translators API.

[1325] 2. Sentiment analysis

[1326] Analyze the sentiment of text using TextBlob and neutralize the emotional expressions.

[1327] 3. Monitoring your emotional state

[1328] The server uses emotion analysis to monitor the user's emotional state in real time and present the operator with the most appropriate response.

[1329] Specific examples

[1330] Example 1: User gives feedback in English

[1331] The user sends feedback in English, saying, "I'm extremely upset with your service. My order was not delivered on time." The server receives this feedback and translates it in real time to "I'm extremely upset. My order was not delivered on time." It then performs sentiment analysis, which parses it as an expression of strong anger. The server then neutralizes the emotion and displays on the operator's control device, "A customer has inquired about their dissatisfaction with the service and a delayed order." The sentiment analysis function also issues a warning, such as, "The customer is extremely dissatisfied. Please prioritize their response." The operator enters an appropriate response, which the server translates and sends to the user's device, "We apologize for the inconvenience, we will check immediately."

[1332] Example of input prompt for generative AI model

[1333] "I'm extremely upset with your service. My order was not delivered on time."

[1334] Prompt: "Translate the above English sentence into Japanese and perform sentiment analysis to neutralize the emotional content. Then, suggest how to respond to the operator."

[1335] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1336] Step 1:

[1337] This is the action in which a user (consumer) inputs the content of a query from their own device and sends it to the server. The input is the user's query and its language information, and the output is the raw query data received by the server. The server then performs preliminary preparations to translate this query in real time. Specifically, the query and language information are saved in a database and used in the next step.

[1338] Step 2:

[1339] This is the operation in which the server translates the input inquiry content in real time. The input is the inquiry content and language information received in step 1, and the output is the translated text. The server calls the Translators API to translate from the original language into a common language (such as English) that the operator can understand. Specifically, the Translators API is used to translate the English sentence, "I'm extremely upset with your service. My order was not delivered on time." into Japanese.

[1340] Step 3:

[1341] The server performs sentiment analysis on the translated text and neutralizes emotional expressions as necessary. The input is the translated text obtained in step 2, and the output is the neutralized text. The server uses TextBlob to perform sentiment analysis and detect emotional expressions. For example, it converts a sentence that expresses strong anger into a neutral expression such as "We apologize for the inconvenience."

[1342] Step 4:

[1343] This is the operation in which the server notifies the operator based on emotion analysis. The input is the neutralized text obtained in step 3, and the output is a notification message displayed on the operator's operating device. The server uses its emotion analysis function to evaluate the user's emotional state and sends a notification to the operator, such as "The customer is very dissatisfied. Please handle this as a priority."

[1344] Step 5:

[1345] This is the operation in which the operator inputs a response on the operating device and sends the content to the server. The input is the text of the operator's response, and the output is the response data from the operator received by the server. The operator inputs an appropriate response based on the neutralized inquiry content and clicks the send button.

[1346] Step 6:

[1347] The server then translates the operator's response again and performs sentiment analysis. The input is the operator's response text received in step 5, and the output is the translated and neutralized reply text. The server again uses the Translators API and TextBlob to translate the operator's reply into the user's language and neutralize it to avoid emotional overload.

[1348] Step 7:

[1349] This is the operation in which the server finally sends the translated and sentiment-analyzed reply text to the user's device. The input is the neutralized reply text obtained in step 6, and the output is the reply message displayed on the user's device. The server sends the translated and neutralized text to the user's device, providing a response in a way that is less stressful for the user.

[1350] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1351] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1352] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1353] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1354] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1355] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1356] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1357] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1358] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1359] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1360] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1361] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1362] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1363] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1364] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1365] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1366] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1367] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1368] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1369] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1370] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1371] The following is further disclosed regarding the above embodiment.

[1372] (Claim 1)

[1373] AI translation means that translates text in real time,

[1374] AI sentiment analysis means to perform sentiment analysis and neutralize emotions from translated text;

[1375] a means for displaying the inquiry content processed by the translation means and the sentiment analysis means on an operator terminal;

[1376] a means for translating a reply input by an operator into the user's language using said translation means and said sentiment analysis means, and transmitting the neutralized text to the user terminal;

[1377] A system including:

[1378] (Claim 2)

[1379] 10. The system of claim 1, further comprising means for receiving a query and extracting linguistic information thereof.

[1380] (Claim 3)

[1381] 10. The system of claim 1, further comprising means for detecting emotional expressions in a user's query and converting the emotional expressions into neutral expressions.

[1382] "Example 1"

[1383] (Claim 1)

[1384] means for receiving a query from a user terminal;

[1385] AI translation means that translates text in real time,

[1386] AI sentiment analysis means to perform sentiment analysis and neutralize emotions from translated text;

[1387] a means for displaying the inquiry content processed by the translation means and the sentiment analysis means on an operator terminal;

[1388] a means for translating a reply input by an operator into the user's language using the translation means and the sentiment analysis means, and transmitting the neutralized text to a user terminal;

[1389] A system including:

[1390] (Claim 2)

[1391] 10. The system of claim 1, further comprising means for receiving a query and extracting linguistic information thereof.

[1392] (Claim 3)

[1393] 10. The system of claim 1, further comprising means for detecting emotional expressions in a user's query and converting the emotional expressions into neutral expressions.

[1394] "Application Example 1"

[1395] (Claim 1)

[1396] An artificial intelligence translation means that translates text in real time;

[1397] an artificial intelligence sentiment analysis means for performing sentiment analysis and neutralizing emotions from the translated text;

[1398] a means for displaying the inquiry content processed by the translation means and the sentiment analysis means on a staff terminal;

[1399] a means for translating a reply input by a staff member into the user's language using said translation means and said sentiment analysis means, and transmitting the neutralized text to the user terminal;

[1400] a means for displaying the inquiry content and the reply content on an augmented reality device used as a staff terminal;

[1401] A system including:

[1402] (Claim 2)

[1403] 10. The system of claim 1, further comprising means for receiving a query and extracting linguistic information thereof.

[1404] (Claim 3)

[1405] 10. The system of claim 1, further comprising means for detecting emotional expressions in a user's query and converting the emotional expressions into neutral expressions.

[1406] "Example 2: Combining Emotion Engines"

[1407] (Claim 1)

[1408] A means of accepting inquiries and translating text in real time,

[1409] A means of sentiment analysis of the translated text and neutralizing emotional expressions;

[1410] a means for displaying the translated and neutralized query content on an operator terminal;

[1411] A means for translating the reply entered by the operator in real time, analyzing the sentiment and neutralizing it, and sending it to the user's device;

[1412] means for recognizing, analyzing, and monitoring the user's emotional state and suggesting a response to the operator;

[1413] A means for accumulating and storing data on inquiries and responses for analysis;

[1414] A system including:

[1415] (Claim 2)

[1416] 10. The system of claim 1, further comprising means for receiving a query and extracting linguistic information thereof.

[1417] (Claim 3)

[1418] 10. The system of claim 1, further comprising means for detecting emotional expressions in a user's query and converting the emotional expressions into neutral expressions.

[1419] "Application example 2 when combining emotion engines"

[1420] (Claim 1)

[1421] a knowledge processing means for translating text in real time;

[1422] a sentiment analysis means for performing sentiment analysis and neutralizing sentiment from the translated text;

[1423] a means for displaying, on an operation device, the inquiry content processed by the translation means and the sentiment analysis means;

[1424] a means for translating a reply input via an operation device into a language of the terminal by said translation means and said sentiment analysis means, and transmitting the neutralized text to the terminal device;

[1425] A means to monitor the user's emotional state in real time using emotion analysis function and suggest ways to respond,

[1426] a means for quickly and effectively processing user feedback based on said response method;

[1427] A system including:

[1428] (Claim 2)

[1429] 10. The system of claim 1, further comprising means for receiving a query and extracting linguistic information thereof.

[1430] (Claim 3)

[1431] 10. The system of claim 1, further comprising means for detecting emotional expressions in a user's query and converting the emotional expressions into neutral expressions. [Explanation of symbols]

[1432] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. AI translation means that translates text in real time, AI sentiment analysis means to perform sentiment analysis and neutralize emotions from translated text; a means for displaying the inquiry content processed by the translation means and the sentiment analysis means on an operator terminal; a means for translating a reply input by an operator into the user's language using said translation means and said sentiment analysis means, and transmitting the neutralized text to the user terminal; A system including:

2. The system of claim 1 , further comprising means for receiving a query and extracting linguistic information therefrom.

3. The system of claim 1 , further comprising means for detecting emotional expressions in the content of a user's query and converting the emotional expressions into neutral expressions that are less emotional.

Images