System

Abstract

A system is provided.SOLUTION: A system comprising: means for learning contents of an inquiry for support and generating an illustration as trouble parts for visually expressing the contents; means for distributing the generated illustration to a user terminal and providing an interface through which a user can visually select a problem; means for automatically generating an appropriate inquiry sentence based on illustration information selected by the user; means for providing an interface through which the user can finely adjust the generated inquiry sentence; and means for transmitting a final inquiry sentence adjusted by the user to customer support.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Some users have difficulty verbalizing problems they are having while using a service because they are unfamiliar with technical terminology or appropriate ways of expressing themselves. This makes it difficult for them to make accurate inquiries to customer support, resulting in the problem of not receiving appropriate support. The present invention solves this problem by providing a system that allows even users who are not familiar with technical terminology to easily and accurately report problems. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides the following means. The system includes: means for learning the content of inquiries received by support and generating illustrations as problem components for visually expressing the inquiries; means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem; means for automatically generating an appropriate inquiry sentence based on the illustration information selected by the user; means for providing an interface that allows the user to fine-tune the generated inquiry sentence; and means for sending the final inquiry sentence adjusted by the user to customer support. The system also includes means for analyzing the illustration information selected by the user and text information added by the user and using an AI model to generate the inquiry sentence. The system also includes means for providing an interface that allows the user to visually express their problem by arranging illustrations by dragging and dropping.

[0006] "Problem parts" are illustrations that visually represent users' problems and are generated based on inquiries received by support.

[0007] "Illustrations" are images used to help users visually select their own questions.

[0008] An "interface" is a screen or function that provides the display and input means for a user to interact with a system.

[0009] "Inquiry text" is text generated by a user to communicate the content of their inquiry to customer support.

[0010] An "AI model" is an artificial intelligence algorithm that learns from past inquiry data and automatically generates new inquiry content.

[0011] "Drag and drop" is an operation in which a user selects an object on the screen using a computer mouse or touch operation and moves it to another location.

[0012] A "user terminal" is a device used by a user to operate the system, such as a personal computer or smartphone.

[0013] "Customer Support" is the service department that provides support and solutions to user inquiries.

[0014] "Inquiry data" refers to information about inquiries previously submitted by users.

[0015] "Additional explanatory text" is text entered by the user in their own words that contains detailed information that cannot be fully expressed in an illustration. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system that allows users to easily select a problem visually and automatically generates appropriate query sentences. The program of this system is configured based on the roles of the user, terminal, and server. The specific operation and processing flow are explained below in natural language.

[0038] overview

[0039] This system operates in a mutually linked manner between the server, terminals, and users. The user selects an illustration via the terminal, and the server generates a query based on the selected information. The operation of the entire system is explained below.

[0040] Illustration generation and learning

[0041] server

[0042] The server collects past inquiry data, and the AI ​​model learns from this data. This allows it to illustrate common problems as "problem parts."

[0043] For example, if there are a lot of inquiries about network issues, an illustration of a Wi-Fi router or connection error will be generated.

[0044] Providing an interface

[0045] Terminal

[0046] The terminal provides a user interface that displays illustrations delivered from the server. This interface supports drag and drop, allowing users to operate it intuitively.

[0047] For example, if a user wants to report a poor internet connection issue, they can select and place an icon of a Wi-Fi router or connection error to visually represent the problem.

[0048] Question Selection

[0049] user

[0050] Using the device interface, users select an illustration to visually represent their problem, which is then placed on the device and displayed as an overview of the problem.

[0051] For example, if a user selects the illustrations "Wi-Fi router" and "connection error," this information is sent to the server.

[0052] Generate inquiry content

[0053] server

[0054] The server receives the illustration information selected by the user, analyzes it using an AI model, and generates a natural language query based on the analysis results.

[0055] For example, if the illustrations of "Wi-Fi router" and "connection error" are selected, the server will generate the sentence "Currently, your Internet connection is unstable and there appears to be a problem with your Wi-Fi router."

[0056] Fine-tune and submit your inquiry

[0057] Terminal

[0058] The generated query is sent to the device and displayed to the user, who can then fine-tune it and add details like the model number or specific symptoms if needed.

[0059] For example, a user enters the "Wi-Fi router model number" as an additional detail.

[0060] After making a final confirmation, the user presses the send button to send the inquiry to customer support.

[0061] Transfer to Support

[0062] server

[0063] The server receives the user's final inquiry text and forwards it to the customer support team, who can then respond promptly based on the information received.

[0064] In this way, this system provides an environment where users can visually express their problems and generate and adjust appropriate inquiry sentences based on that information. This system allows users to easily make accurate inquiries even if they are not familiar with technical terminology, and customer support can respond efficiently.

[0065] The processing flow will be explained below.

[0066] Program processing flow

[0067] Step 1:

[0068] server

[0069] The server collects data from past support inquiries, including issues users have reported and detailed metadata associated with them, and stores the collected data in a database, where the AI ​​model learns from it.

[0070] Step 2:

[0071] server

[0072] Using an AI model, the system analyzes collected inquiry data and illustrates common problems as "problem parts." For example, if there is a lot of data related to network connection problems, it generates illustrations of Wi-Fi routers and connection errors. The generated illustrations are stored in a database.

[0073] Step 3:

[0074] Terminal

[0075] When a user accesses the system, the generated illustration is retrieved from the server. The retrieved illustration is then displayed on the terminal's user interface. This interface is designed so that users can operate it by dragging and dropping.

[0076] Step 4:

[0077] user

[0078] Users can view the displayed illustrations and select an illustration to visually represent their problem. The selected illustration is then placed on the device by dragging and dropping, visually representing the user's problem.

[0079] Step 5:

[0080] Terminal

[0081] The illustration information selected by the user and additional text information are sent to the server, including the ID of the illustration selected by the user, its placement information, and any supplementary information entered by the user.

[0082] Step 6:

[0083] server

[0084] The server analyzes the received illustration information and generates an appropriate query using an AI model. For example, if the illustrations "Wi-Fi router" and "connection error" are selected, the server generates the query "Your internet connection is currently unstable and there appears to be a problem with your Wi-Fi router."

[0085] Step 7:

[0086] Terminal

[0087] The generated query text is displayed to the user, who can review it and fine-tune it as needed, for example by adding details about the specific router model or condition.

[0088] Step 8:

[0089] user

[0090] The user makes a final check and presses the send button when the inquiry is complete. This action officially sends the user's inquiry.

[0091] Step 9:

[0092] server

[0093] The server receives the user's final inquiry text and forwards it to the customer support team. The inquiry is registered in the support system so that the support staff can respond promptly.

[0094] This is the specific processing flow of this system. This allows users to make accurate inquiries without knowing the technical details, and enables the support team to respond quickly and accurately.

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] Current customer support systems make it difficult for users to properly explain their problems, especially for those without technical expertise. Furthermore, the vague nature of inquiries makes it difficult for support staff to respond efficiently. This results in lengthy response times and lower user satisfaction.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes means for learning the content of an inquiry and generating an image for visually expressing the inquiry, means for delivering the generated image to a user terminal and providing an interface that allows the user to visually select a problem, means for automatically generating an appropriate inquiry sentence based on the image information selected by the user, means for providing an interface that allows the user to fine-tune the generated inquiry sentence, and means for transmitting the final inquiry sentence adjusted by the user to a support center. This allows the user to visually select a problem and create an appropriate inquiry sentence even without technical expertise, enabling support staff to respond quickly and efficiently based on clear inquiry content.

[0100] "Inquiry Content" refers to the content of the problem or question provided by the user, and is a general term for information that is requested to be resolved by customer support.

[0101] "Visual representation" means using pictures, diagrams, automatically generated icons, etc. instead of words to make information or issues understandable at a glance.

[0102] "Pictures" are images or icons generated to visually represent the problem, allowing users to easily specify the problem.

[0103] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet, which allows the user to access and operate the system.

[0104] An "interface" refers to the screen and operation method used for interaction between the user and the system, providing an environment in which the user can operate intuitively.

[0105] A "generative model" is a type of artificial intelligence that is used to learn large amounts of data to find patterns and regularities and generate new data.

[0106] "Drag and drop" refers to a method of operation in a computer user interface where an object on the screen is moved using a mouse or touch and placed in another location.

[0107] "Support Center" refers to a specialized department or institution that accepts inquiries and problems from users and provides solutions.

[0108] This invention is a system that allows users to easily select a problem visually and automatically generates an appropriate query based on that selection. This system consists of a server, a terminal, and a user component, which operate in conjunction with each other. The details are given below.

[0109] Illustration generation and learning

[0110] server

[0111] The server collects past inquiry data from an inquiry database using database queries.

[0112] The server provides the collected data to a generative AI model (e.g., GPT-3) for learning. As a result of the learning, the AI ​​model illustrates common problems as "problem parts."

[0113] For example, if there are many inquiries about Internet connections, the server will generate illustrations of Wi-Fi routers and connection errors.

[0114] Providing an interface

[0115] Terminal

[0116] The device provides a user interface that displays illustrations delivered from the server, using a web application built with HTML5 and JavaScript.

[0117] The user interface is intuitive with a drag-and-drop feature, allowing users to select and place illustrations on the screen to visually represent the problem.

[0118] For example, a user might drag a "Wi-Fi router" icon across the screen and place a "Connection Error" icon next to it to represent the problem.

[0119] Selecting a problem and generating a query

[0120] user

[0121] The user uses the device interface to select the illustrations they need to visually represent the problem.

[0122] The selected illustration information is displayed in real time on the device, giving the user an overview of the problem, and once the selection is complete, the information is sent to the server.

[0123] server

[0124] The server receives the illustration information selected by the user and inputs it into a generative AI model (e.g., GPT-3) for analysis. As a result of the analysis, a natural language query is generated.

[0125] For example, if a user selects the illustrations of "Wi-Fi router" and "connection error," the server generates the sentence, "Your Internet connection is currently unstable and there appears to be a problem with your Wi-Fi router."

[0126] Fine-tuning the inquiry and final submission

[0127] Terminal

[0128] The generated query text is sent to the device and displayed to the user, who can fine-tune it and enter additional details (e.g., the model number of the Wi-Fi router) in the text boxes if necessary.

[0129] Once the user has finalized the content and pressed the "Send" button, the inquiry will be sent to customer support.

[0130] server

[0131] The server receives the final inquiry text sent from the device and forwards it to the customer support team, who then responds quickly and efficiently based on the information received.

[0132] Prompt Sentence Examples

[0133] As a concrete example, the following prompt sentence is input to the generative AI model:

[0134] If the user selects the illustrations of "Wi-Fi router" and "Connection error",

[0135] Appropriate message: Your internet connection is currently unstable and there appears to be an issue with your Wi-Fi router.

[0136] In this way, the system of the present invention combines visual problem selection with natural language query generation to provide an environment where users can accurately report problems without technical expertise, enabling customer support to respond quickly and efficiently.

[0137] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0138] Step 1:

[0139] Illustration generation

[0140] The server collects past inquiry data from an inquiry database. This data mainly includes inquiry content and category information. These data are extracted through database queries. Next, the server inputs this data into a generative AI model (e.g., GPT-3) and trains the model. Through training, the model generates pictures to visually represent common problems. The generated pictures are output in the form of icons, such as "Wi-Fi router" or "connection error."

[0141] Step 2:

[0142] Illustration distribution and display

[0143] The server delivers the generated images to the user's device. The user's device receives the images delivered from the server and displays them on a web application built with HTML5 and JavaScript. This interface has a drag-and-drop function, allowing users to easily operate it visually. This allows users to select how to express the problem in a picture. The input to the user's device is the image data from the server, and the output is an interface that the user can operate.

[0144] Step 3:

[0145] Question Selection

[0146] The user selects a picture on the device interface to visually represent their problem. The selected picture is displayed on the screen in real time and positioned by the user. For example, the user selects a "Wi-Fi router" icon and positions it on the screen along with a "Connection Error" icon. This action generates a selection, which is sent to the server. The user's input is the selected picture, and the output is the selection sent to the server.

[0147] Step 4:

[0148] Query generation

[0149] The server receives the picture information selected by the user and inputs that information into the generative AI model. The model analyzes this and generates a natural language query. For example, if the pictures of "Wi-Fi router" and "connection error" are selected, the model will generate the sentence "Currently, your internet connection is unstable and there appears to be a problem with your Wi-Fi router." The server's input is the picture information from the user, and its output is the generated query.

[0150] Step 5:

[0151] Fine-tuning of inquiry text

[0152] The device displays the generated query text to the user, who can refer to it and make any necessary adjustments. Specifically, the user can enter additional details, such as "Wi-Fi router model number," in the text box to complete the text. The user's input is the adjusted query text, and the output is the final query text.

[0153] Step 6:

[0154] Sending an inquiry

[0155] After the user has finished fine-tuning the query text, they press the "Send" button to send it to customer support. At this point, the final version of the query text is sent from the user's device to the server. The server then forwards the received final version of the query text to the support center. The server's input is the user's final query text, and its output is the query text sent to the support center.

[0156] (Application example 1)

[0157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0158] When handling customer support inquiries at physical stores, it is difficult for customers to accurately communicate their problems, and it is also difficult for support staff to quickly understand and address the issues. For this reason, there is a need for a system that allows customers to intuitively report problems and enables support staff to respond quickly and accurately.

[0159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0160] In this invention, the server includes means for learning the content of inquiries received by the support center and generating actions as problem parts for visually expressing the content, means for distributing the generated actions to a user device and providing an interface for the user to visually select a problem, means for automatically generating an appropriate inquiry sentence based on the action information selected by the user, means for providing an interface for the user to fine-tune the generated inquiry sentence, means for sending the final inquiry sentence adjusted by the user to a support staff member, means for using an artificial intelligence (AI) model to generate the inquiry sentence, means for sending the inquiry sentence generated by the artificial intelligence (AI) model to an appropriate staff member, and means for managing the transmission history. This allows customers to intuitively report problems and enables support staff to respond efficiently and quickly.

[0161] "Learning support inquiries" is the process of collecting inquiry data from users and training a machine learning model based on that data.

[0162] "Generating actions as parts of a problem to visually express it" means generating specific illustrations and action parts related to the content of the inquiry and displaying them visually.

[0163] "User Device" refers to an electronic device used by a user, such as a smartphone, tablet, or PC.

[0164] "Providing an interface" means providing an operation screen and operation means that users can operate intuitively.

[0165] "Action information" refers to information about specific illustrations and actions selected by the user on the interface.

[0166] "Automatically generating an appropriate inquiry sentence" means generating an inquiry sentence in natural language based on the selected action information.

[0167] A "fine-tunable interface" means providing an operation screen that allows the user to check, edit, and modify the generated query text.

[0168] "Support personnel" refers to staff or personnel in charge of customer support.

[0169] "Using an artificial intelligence (AI) model to generate a query sentence" means using AI technology to analyze the query content and generate a sentence in natural language.

[0170] "Managing transmission history" means saving and managing the history of the contents of inquiries sent.

[0171] A "prompt sentence" is the input text that an artificial intelligence (AI) model uses to generate a query sentence.

[0172] The present invention relates to a system that enables users to intuitively report problems at physical stores and allows support staff to quickly respond. Specific embodiments for realizing this system will be described below.

[0173] First, the server learns the content of inquiries received by the support center and generates behavior as problem parts to visually represent them. The hardware used for this is a high-performance server computer, and the software used includes open-source machine learning libraries and natural language processing libraries.

[0174] The server then delivers the generated actions to the user device, which is typically an electronic device such as a smartphone or tablet. The interface is designed to be intuitive and allows drag and drop, allowing users to visually select problems.

[0175] When a user selects an action using the interface, the action information is sent from the user device to the server. The server then automatically generates an appropriate query sentence based on the action information selected by the user. This automatic generation uses a generative AI model using the OpenAI API. The following is an example of a specific prompt sentence.

[0176] plaintext

[0177] User-selected illustrations: cashier trouble, product placement

[0178] Additional details: The cashier screen is frozen and the product cannot be found

[0179] Generate a query.

[0180] The generated query text is displayed on the user's device, allowing the user to fine-tune it and add any necessary details. For example, the user might add the cash register model number or the specific product name.

[0181] The final query text, fine-tuned by the user, is sent from the user's device to the support staff. The support staff can then respond quickly and appropriately based on the text sent. Furthermore, by managing the transmission history, past inquiry information can be referenced, which can be useful for future support responses.

[0182] Through this process, users can intuitively report problems and support staff can respond efficiently.

[0183] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0184] Step 1:

[0185] Learning inquiry content and generating problem parts

[0186] The server collects user inquiry data and uses a machine learning algorithm to learn from it. This generates action icons and illustrations to visually represent common problems. For example, if many inquiries are about "Wi-Fi router problems," the server generates an illustration of a Wi-Fi router.

[0187] Input: Past inquiry data

[0188] Data processing: Data analysis and icon generation using machine learning algorithms

[0189] Output: Action icons and illustrations

[0190] Step 2:

[0191] Delivery of action icons and illustrations

[0192] The server then delivers the generated action icons and illustrations to the user's device, providing a visual interface for the user to select problems. For example, a Wi-Fi router icon is displayed on the user's smartphone.

[0193] Input: Action icons and illustrations

[0194] Data processing: None (transfer only)

[0195] Output: Action icons and illustrations displayed on the user's device

[0196] Step 3:

[0197] Selecting and sending operational information

[0198] The user selects an action icon or illustration on the interface that corresponds to their problem, and the selection is sent from the user device to the server.

[0199] Input: User-selected action icons and illustrations

[0200] Data processing: collecting and formatting selected information

[0201] Output: Selection information sent to the server

[0202] Step 4:

[0203] Automatic generation of inquiry text

[0204] The server uses the received behavior information to call the generative AI model and generate an appropriate query sentence. At this time, the query sentence is automatically generated using a prompt sentence. For example, the prompt might be "Illustration selected by the user: Wi-Fi router, connection error."

[0205] Input: User-selected action information, prompt text

[0206] Data Processing: Natural Language Generation Using Generative AI Models

[0207] Output: Generated query text

[0208] Step 5:

[0209] Fine-tuning of inquiry text

[0210] The device displays the generated query text to the user and provides a screen where the user can review, edit, and fine-tune the content. For example, the user can enter the "Wi-Fi router model number" or "specific connection error situation."

[0211] Input: Generated query text

[0212] Data processing: User editing and fine-tuning

[0213] Output: The final query text that has been reviewed and refined by the user

[0214] Step 6:

[0215] Sending the final inquiry

[0216] The user checks the finely adjusted inquiry text and presses the send button to send it to the support staff. This transmission is performed from the user device via the server.

[0217] Input: The final query text that the user has reviewed and refined

[0218] Data processing: None (transfer only)

[0219] Output: Final query sent to support

[0220] Step 7:

[0221] Managing sending history

[0222] The server stores the final query text sent as a history and makes it available for future inquiries, allowing past history to be referenced.

[0223] Input: Final query text

[0224] Data processing: Save to database

[0225] Output: Saved query history

[0226] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0227] This system allows users to easily select a problem visually and automatically generates appropriate query sentences. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the quality and efficiency of queries can be improved.

[0228] overview

[0229] This system works in conjunction with a server, a terminal, and an emotion engine that recognizes the user's emotional state. The user selects an illustration via the terminal, and the server generates a query sentence based on the selection information and the user's emotional state. The specific operation and processing flow are explained below in natural language.

[0230] Illustration generation and learning

[0231] server

[0232] The server collects data from past support inquiries, including the content and metadata of users' past inquiries, and the collected data is used to train an AI model.

[0233] The AI ​​model uses this data to illustrate common problems as "problem parts." For example, if there is a lot of data about network connection problems, it will generate illustrations of Wi-Fi routers and connection errors. The generated illustrations are stored in a database.

[0234] Providing an interface

[0235] Terminal

[0236] The device retrieves the illustrations sent from the server and displays them on a user interface that users can manipulate using drag and drop.

[0237] If a user reports a poor internet connection issue, you can select and place a Wi-Fi router or connection error icon to visually represent the problem.

[0238] emotion recognition

[0239] Terminal

[0240] While the user is interacting with the interface, the emotion engine recognizes the user's emotions through facial recognition and voice analysis, collecting emotional data. For example, if the user is frustrated, their emotion is analyzed in real time.

[0241] Question Selection

[0242] user

[0243] Users can view the displayed illustrations and select an illustration to visually represent their problem. The selected illustration can then be placed on the device by dragging and dropping.

[0244] For example, if a user selects the illustrations "Wi-Fi router" and "connection error," that information is sent to the server.

[0245] Generate inquiry content

[0246] server

[0247] The server receives the illustration information selected by the user and the emotion data obtained by the emotion engine, and uses the AI ​​model to generate an appropriate query sentence.

[0248] For example, if the illustrations of "Wi-Fi router" and "connection error" are selected and the user is feeling irritated, the server will generate the sentence "Currently, your Internet connection is very unstable and there seems to be a problem with your Wi-Fi router."

[0249] Fine-tune and submit your inquiry

[0250] Terminal

[0251] The generated query text is displayed to the user, who can review it and fine-tune it as needed, for example by adding details about the specific router model or condition.

[0252] After the user makes some fine adjustments, they make a final check and press the send button, which officially sends the inquiry text.

[0253] Transfer to Support

[0254] server

[0255] The server receives the user's final inquiry text and forwards it to the customer support team. The inquiry is registered in the support system so that the support staff can respond promptly.

[0256] Emotional Escalation

[0257] server

[0258] The server also has the means to automatically escalate a query if the user's emotion exceeds a certain threshold. For example, if the user is very annoyed, the query will be given high priority and dealt with quickly.

[0259] In this way, this system allows users to visually express their problems and generates appropriate inquiry sentences based on that information and their emotional state. Furthermore, by combining it with an emotion engine, it becomes possible to respond according to the user's emotions, improving the efficiency and quality of customer support.

[0260] The processing flow will be explained below.

[0261] MODE FOR CARRYING OUT THE INVENTION

[0262] The system of the present invention allows users to visually select a problem and provides appropriate and prompt support by having the AI ​​generate an inquiry that reflects the user's feelings. The specific process flow is shown below.

[0263] Step 1:

[0264] server

[0265] The server collects past inquiry data and trains the AI ​​model. This data includes issues reported by users in the past, their details, and the results of their responses. Repeated data collection and training improves the accuracy of the model.

[0266] Step 2:

[0267] server

[0268] Based on the learned data, the AI ​​model illustrates common problems as "problem parts." For example, for the common problem of "unstable network," it generates illustrations of "Wi-Fi router" and "connection error."

[0269] Step 3:

[0270] Terminal

[0271] When a user starts the system, the terminal retrieves the generated illustrations from the server and displays them on the user interface, which has a drag-and-drop function for intuitive operation.

[0272] Step 4:

[0273] user

[0274] Users can select an illustration to visually represent their problem by viewing the displayed illustrations. For example, a user can select an illustration of a "Wi-Fi router" and a "connection error" and place them on the screen.

[0275] Step 5:

[0276] Terminal

[0277] The illustration information selected by the user and additional explanatory text are sent to the server. This information includes the ID of the selected illustration, its location, and any additional explanation entered by the user.

[0278] Step 6:

[0279] Terminal

[0280] While the user is selecting an illustration, the emotion engine recognizes the user's emotions in real time through facial recognition and voice analysis, and the emotion data is also sent to the server in parallel.

[0281] Step 7:

[0282] server

[0283] The server analyzes the received illustration information and emotion data and generates an appropriate query using an AI model. For example, if the illustrations of "Wi-Fi router" and "connection error" are selected and the user is feeling irritated, the server will generate the following sentence: "Currently, your internet connection is very unstable and there seems to be a problem with your Wi-Fi router."

[0284] Step 8:

[0285] Terminal

[0286] The generated query text is displayed to the user, who can review it and fine-tune it as needed, for example adding details about a specific router model or condition.

[0287] Step 9:

[0288] user

[0289] The user makes a final check and presses the send button when the inquiry is complete. This action officially sends the user's inquiry.

[0290] Step 10:

[0291] server

[0292] The server receives the user's final inquiry text and forwards it to the customer support team. The inquiry is registered in the support system so that the support staff can respond promptly.

[0293] Step 11:

[0294] server

[0295] If the user's emotion exceeds a certain threshold, the server escalates the query. For example, if the user is very annoyed, the query is set to high priority and requires immediate attention.

[0296] This system makes it easier for users to visually and emotionally express their problems, allowing customer support to understand the user's situation and emotions and respond appropriately.

[0297] Example 2

[0298] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0299] In modern customer support systems, it is extremely important for users to quickly and appropriately communicate the problems they face. However, conventional systems often require users to spend a lot of time and effort to accurately describe their problems. Furthermore, they respond without taking the user's emotional state into consideration, which can lead to a decline in the quality and efficiency of support. In particular, they may be unable to respond appropriately to emotionally charged users, which can increase their dissatisfaction. Therefore, there is a need for a system that allows users to visually express their problems and automatically generates inquiry text that takes their emotional state into account.

[0300] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0301] In this invention, the server includes means for learning the content of inquiries received by the support center and generating illustrations as problem parts to visually express the inquiries, means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem, and means for recognizing the user's emotions and generating an inquiry sentence that includes that information. This enables the user to visually express their problem and automatically generate a specific inquiry sentence that includes their emotional state.

[0302] "Learning the content of inquiries received by support" means collecting inquiry data from past users and analyzing and learning from that data using a machine learning model.

[0303] "Generating illustrations as problem parts" means generating illustrations to visually represent common problems faced by users based on inquiry data.

[0304] "Delivering to the user's device" means sending the generated illustration from the server to the user's device.

[0305] "Providing an interface that allows users to visually select problems" means providing an operation screen that allows users to visually select and express their own problems using displayed illustrations.

[0306] "Automatically generating appropriate inquiry text based on illustration information selected by the user" means using an AI model to automatically write down the corresponding inquiry content based on the information of the illustration selected by the user.

[0307] "Providing an interface that allows users to fine-tune query sentences" means providing an operation screen that allows users to edit and modify the generated query sentences.

[0308] "Emotion recognition" means analyzing a user's emotional state from their facial expressions and voice and acquiring it as data.

[0309] "Send inquiry to customer support" means that the user sends the inquiry that has been finally confirmed and corrected to the support staff.

[0310] "Using an AI model" means using a model that analyzes and generates data using algorithms based on machine learning and natural language processing.

[0311] "Placing by drag and drop" refers to the operation in which the user selects an illustration with the mouse, drags it to the desired location, and drops it.

[0312] This invention is a system that allows users to easily select a problem visually and automatically generates appropriate query sentences. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the quality and efficiency of inquiries can be improved. This system works in conjunction with a server, a user terminal, and an emotion engine that recognizes the user's emotional state.

[0313] Data collection and learning

[0314] The server collects past inquiry data, including text data and metadata (time, category, etc.). By training this data using a machine learning model (e.g., BERT, GPT-3), it is possible to understand the relationship between inquiry content and sentiment. This improves the accuracy of inquiry sentence generation, as described below.

[0315] Illustration generation and saving to database

[0316] The server uses the trained machine learning model to generate illustrations of "problem parts" that visually represent common problems. For example, if there is a lot of data related to network connection problems, it will generate illustrations of Wi-Fi routers and connection errors. The generated illustrations are stored in a database, and each part is also saved with its corresponding tag information.

[0317] Illustration distribution and user interface display

[0318] The device retrieves illustration data from the database and displays it on the user interface. The interface is built using HTML5 and JavaScript, and users can manipulate the illustrations with drag and drop. For example, a user who wants to report a poor internet connection can select an icon of a Wi-Fi router or a connection error.

[0319] emotion recognition

[0320] The device uses an emotion engine to recognize the user's emotions in real time while the user is operating the interface. Specifically, an emotion recognition engine (e.g., OpenFace or SpeechEmotionRecognition) is used to collect emotion data from the user's facial expressions and voice. The emotion data is sent from the device to the server and used to generate query sentences.

[0321] Generate and refine inquiries

[0322] The server receives the illustration information and emotion data selected by the user and automatically generates an appropriate query using an AI model (e.g., GPT-3). For example, if the illustration of "Wi-Fi router" and "connection error" is selected and the emotion data indicates frustration, the server generates a query such as, "Your internet connection is currently very unstable, and there appears to be a problem with your Wi-Fi router." The generated query is then sent to the user's device, where the user can review the displayed message and make adjustments as necessary. For example, the user can add information about the router's specific model number or other detailed information about the situation.

[0323] Submitting a ticket and routing to support

[0324] The user checks the final query sentence after making some fine adjustments and presses the send button, which sends the query sentence to the server.

[0325] The server forwards the received inquiry text to the customer support team and registers the inquiry content in the support system. In particular, if the emotional data exceeds a certain threshold, the inquiry is automatically escalated and set to a high priority.

[0326] Examples of concrete examples and prompts

[0327] Here are some examples of prompts to input to a generative AI model:

[0328] "We are currently experiencing connectivity issues with our Wi-Fi router. The internet connection is very unstable and frequently drops out. Users are extremely frustrated with this issue and are requesting a prompt response from support staff."

[0329] An example of a query that might be generated in response to this prompt is:

[0330] "Currently, my internet connection is very unstable and there seems to be an issue with my Wi-Fi router. I am frequently disconnected and this is very frustrating. I would appreciate a quick response."

[0331] Through the above steps and concrete examples, users can visually express their problems and automatically generate specific inquiry sentences that include their emotional state, which can greatly improve the efficiency and quality of customer support.

[0332] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0333] Step 1: Data collection and learning

[0334] The server collects past inquiry data from a database. As input, it receives text data, inquiry content, and metadata (categories, timestamps, etc.). The collected data is analyzed and trained using a natural language processing model (e.g., BERT, GPT-3). Data processing involves preprocessing the text (tokenization, removal of stop words, etc.). As output, a model that has learned the association between inquiry content patterns and sentiment is obtained.

[0335] Specific behavior:

[0336] The server periodically retrieves new query data from the database, preprocesses the text data using Python, and then trains the data using a machine learning model.

[0337] Step 2: Generate illustrations and save them to the database

[0338] The server uses the trained model to generate illustrations to visually represent the user's problem. As input, it receives the results of the trained model and query data. As data processing, it uses a generative algorithm (e.g., GAN, a deep learning-based image generation model) to create illustrations. As output, it obtains illustrations of the generated problem parts. These illustrations are stored in a database.

[0339] Specific behavior:

[0340] The server uses Python's PIL library to generate illustrations and stores them in MongoDB, with each illustration also containing corresponding tag information.

[0341] Step 3: Delivering illustrations and displaying the user interface

[0342] The device retrieves illustration data from the server and displays it on the user interface. As input, it receives illustration data from the server. For data processing, it retrieves the data using an AJAX request and renders it on the interface using HTML5 and JavaScript. As output, it provides an interface that the user can operate using drag and drop.

[0343] Specific behavior:

[0344] The device uses AJAX requests to pull down illustration data from the server and uses Dragula.js to enable drag-and-drop operation.

[0345] Step 4: Emotion Recognition

[0346] The device uses an emotion engine to recognize the user's emotions in real time. It receives camera footage and audio data as input. It analyzes the data using an emotion engine (e.g., OpenFace, SpeechEmotionRecognition) to generate user emotion data. The analyzed emotion data is obtained as output. The emotion data is sent from the device to the server.

[0347] Specific behavior:

[0348] The device uses WebRTC to capture real-time data from the camera and microphone and passes it to an emotion recognition engine for analysis.

[0349] Step 5: Selecting the problem illustration

[0350] The user looks at the displayed illustrations and selects one to visually represent their problem. The system receives the illustration information selected by the user as input. The system processes the data by selecting and placing the illustrations using drag and drop. The system obtains the selected illustration information as output, and sends it to the server.

[0351] Specific behavior:

[0352] Users select illustrations via drag-and-drop and click a button to submit the information.

[0353] Step 6: Generate a query

[0354] The server automatically generates an appropriate query sentence using an AI model based on the selected illustration information and emotion data. The server receives the illustration information and emotion data as input. The server uses an AI model (e.g., GPT-3) to generate a query sentence as data calculation. The generated query sentence is obtained as output.

[0355] Specific behavior:

[0356] The server uses the Flask framework and Python scripts to invoke the AI ​​model and generate queries.

[0357] Step 7: View and fine-tune your inquiry

[0358] The terminal displays the generated query sentence to the user. As input, it receives the query sentence from the server. As output, it provides an interface that the user can fine-tune. The user can edit and modify the sentence through this interface.

[0359] Specific behavior:

[0360] The terminal displays the generated text in a text area, allowing the user to freely edit it.

[0361] Step 8: Submitting and forwarding inquiries

[0362] The user checks the final adjusted query sentence and clicks the send button. The revised query sentence is received as input. The query sentence is sent to the server as data processing. The query sentence received by the server is forwarded to customer support as output.

[0363] Specific behavior:

[0364] When the user clicks the submit button, the data is sent in a POST request to the server, which then forwards the data to the customer support system using a REST API.

[0365] Step 9: Emotional Escalation

[0366] The server analyzes the user's emotional data and automatically escalates queries when a certain threshold is exceeded. The input is the emotional data. The data calculation evaluates the emotional data that exceeds the threshold. The output is a query with a high priority.

[0367] Specific behavior:

[0368] The server uses a Python script to analyze the sentiment data and, if certain conditions are met, sets an escalation flag and notifies the support team.

[0369] (Application example 2)

[0370] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0371] Conventional customer support systems have had problems with users being unable to accurately and quickly communicate their problems, resulting in poor quality and efficiency of inquiries, especially when the problem is complex or the user is in an emotional state. Furthermore, the inability to respond appropriately to emotional users risks reducing customer satisfaction. To address these issues, the present invention aims to provide a system that makes it easy for users to visually select a problem and automatically generates appropriate inquiry sentences that take the user's emotional state into account.

[0372] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0373] In this invention, the server includes means for learning the content of a query and generating illustrations as problem parts for visually expressing the query, means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem, means for recognizing the user's emotional state and acquiring emotional data, and means for generating a query sentence based on the emotional data and illustration information, thereby enabling the user to visually select a problem and generating an appropriate query sentence according to the user's emotional state.

[0374] "Support" means the assistance provided to resolve any problems or questions regarding a service or product.

[0375] "Query" means the details of a question or problem submitted by a User to Support.

[0376] "Problem parts" are elements of a problem that are visually and symbolically represented so that users can select them visually.

[0377] An "illustration" is visual content created to visually represent an issue or situation.

[0378] "User terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[0379] A "visual question selection interface" is a user interface that allows a user to select a question from a visually displayed set of options.

[0380] "Emotional state" refers to the user's psychological and emotional state, which is determined through facial recognition and voice analysis.

[0381] "Emotional Data" refers to data about a user's emotional state, captured in real time.

[0382] "Automatic generation" refers to the process of using AI technology and algorithms to automatically generate query sentences based on pre-set rules and data.

[0383] "Customer support" refers to the support team or department that responds to customer inquiries.

[0384] This invention is a system that automatically generates query sentences that make it easier to visually select problems and take into account emotional states. This system operates by combining a user terminal, a server, and an emotion engine, and can improve the quality and efficiency of user queries.

[0385] System Configuration

[0386] The system consists of the following main components:

[0387] server

[0388] Learning and illustration generation

[0389] The server collects past inquiry data and uses an AI model to illustrate common problems as "problem parts," including information about network connection issues and product defects. The server uses an AI model (e.g., a GPT-3 model) to generate illustrations and stores them in a database.

[0390] Query generation

[0391] The server uses an AI model to automatically generate a query based on the illustration information selected by the user and the emotional data obtained by the emotion engine. For example, if a user selects the illustrations of "connection error" and "Wi-Fi router" and their emotional state is "irritated," the server generates the sentence, "Currently, your internet connection is very unstable, and there seems to be a problem with your Wi-Fi router."

[0392] User Device

[0393] Visual Interface

[0394] The device retrieves the illustrations sent from the server and displays them on a user interface that allows drag-and-drop operation, allowing users to intuitively select problems.

[0395] emotion recognition

[0396] The user device is equipped with an emotion engine and uses a camera and microphone to recognize the user's emotional state in real time, allowing it to collect emotional data while the user is operating the device.

[0397] Example of operation

[0398] Let's take the example of an unstable Wi-Fi connection in a store. A user uses their smartphone to report a problem with the store's Wi-Fi connection. They select illustrations of "Wi-Fi router" and "connection error" on the device, and these illustrations are sent to the server. Using the device's camera and microphone, the emotion engine recognizes that the user is frustrated. The server uses an AI model based on the illustration information and emotion data to generate a query.

[0399] Software and hardware used

[0400] AI model: GPT-3

[0401] Emotion recognition library: FER (Face Emotion Recognition)

[0402] Hardware: Smartphone, server (cloud environment)

[0403] Prompt Sentence Examples

[0404] Below are some examples of specific prompt sentences.

[0405] Select the illustration below and the emotion is as follows:

[0406] Illustration: ['Wi-Fi router', 'Connection error']

[0407] Emotion data: {'Anger': 0.8, 'Happiness': 0.1}

[0408] As described above, this system allows users to visually select a problem and generate an appropriate inquiry sentence based on their emotional state, thereby improving the efficiency and quality of customer support.

[0409] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0410] Step 1:

[0411] The server learns from past inquiry data and uses an AI model to illustrate common problem areas. Specifically, it collects data on network connection issues and product defects and generates illustrations to visually represent them. The generated illustrations are stored in a database.

[0412] Input: Past inquiry data

[0413] Output: Illustration of the problem part

[0414] Step 2:

[0415] The terminal retrieves illustrations delivered from the server and displays them on the user interface. Users visually select their own problems using drag-and-drop operations. This interface is designed to be intuitive.

[0416] Input: Illustration delivered from the server

[0417] Output: User's question selection information

[0418] Step 3:

[0419] The device transmits the illustration information visually selected by the user to the server. At the same time, the device uses a camera and microphone to recognize the user's emotional state in real time and acquire emotional data. This data is then analyzed by the emotion engine.

[0420] Input: User-selected illustration information, real-time emotion data

[0421] Output: Send illustration information and emotion data to the server

[0422] Step 4:

[0423] The server receives illustration information and emotion data sent by the user and uses the AI ​​model to automatically generate a query based on them. For example, if the illustrations of "Connection Error" and "Wi-Fi Router" are selected and the user is feeling irritated, the server will generate the sentence "Currently, your internet connection is very unstable and there seems to be a problem with your Wi-Fi router."

[0424] Input: illustration information, emotion data

[0425] Output: Generated query text

[0426] Step 5:

[0427] The device displays the generated query to the user and provides an interface for fine-tuning, allowing the user to modify the query as needed by adding specific router model numbers and status details.

[0428] Input: Generated query text

[0429] Output: User-adjusted query text

[0430] Step 6:

[0431] After the user has fine-tuned the query text, they send it to the server, and by pressing the send button on their device, the final query text is forwarded to the customer support team.

[0432] Input: Tweaked query text

[0433] Output: Send to customer support team

[0434] Step 7:

[0435] The server has the means to automatically escalate a query if the user's emotional state exceeds a certain threshold: for example, if the user is very annoyed, the query will be given high priority and dealt with quickly.

[0436] Input: User emotion data

[0437] Output: Escalation of tickets with high priority

[0438] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0439] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0440] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0441] [Second embodiment]

[0442] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0443] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0444] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0445] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0446] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0447] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0448] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0449] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0450] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0451] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0452] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0453] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0454] This invention is a system that allows users to easily select a problem visually and automatically generates appropriate query sentences. The program of this system is configured based on the roles of the user, terminal, and server. The specific operation and processing flow are explained below in natural language.

[0455] overview

[0456] This system operates in a mutually linked manner between the server, terminals, and users. The user selects an illustration via the terminal, and the server generates a query based on the selected information. The operation of the entire system is explained below.

[0457] Illustration generation and learning

[0458] server

[0459] The server collects past inquiry data, and the AI ​​model learns from this data. This allows it to illustrate common problems as "problem parts."

[0460] For example, if there are a lot of inquiries about network issues, an illustration of a Wi-Fi router or connection error will be generated.

[0461] Providing an interface

[0462] Terminal

[0463] The terminal provides a user interface that displays illustrations delivered from the server. This interface supports drag and drop, allowing users to operate it intuitively.

[0464] For example, if a user wants to report a poor internet connection issue, they can select and place an icon of a Wi-Fi router or connection error to visually represent the problem.

[0465] Question Selection

[0466] user

[0467] Using the device interface, users select an illustration to visually represent their problem, which is then placed on the device and displayed as an overview of the problem.

[0468] For example, if a user selects the illustrations "Wi-Fi router" and "connection error," this information is sent to the server.

[0469] Generate inquiry content

[0470] server

[0471] The server receives the illustration information selected by the user, analyzes it using an AI model, and generates a natural language query based on the analysis results.

[0472] For example, if the illustrations of "Wi-Fi router" and "connection error" are selected, the server will generate the sentence "Currently, your Internet connection is unstable and there appears to be a problem with your Wi-Fi router."

[0473] Fine-tune and submit your inquiry

[0474] Terminal

[0475] The generated query is sent to the device and displayed to the user, who can then fine-tune it and add details like the model number or specific symptoms if needed.

[0476] For example, a user enters the "Wi-Fi router model number" as an additional detail.

[0477] After making a final confirmation, the user presses the send button to send the inquiry to customer support.

[0478] Transfer to Support

[0479] server

[0480] The server receives the user's final inquiry text and forwards it to the customer support team, who can then respond promptly based on the information received.

[0481] In this way, this system provides an environment where users can visually express their problems and generate and adjust appropriate inquiry sentences based on that information. This system allows users to easily make accurate inquiries even if they are not familiar with technical terminology, and customer support can respond efficiently.

[0482] The processing flow will be explained below.

[0483] Program processing flow

[0484] Step 1:

[0485] server

[0486] The server collects data from past support inquiries, including issues users have reported and detailed metadata associated with them, and stores the collected data in a database, where the AI ​​model learns from it.

[0487] Step 2:

[0488] server

[0489] Using an AI model, the system analyzes collected inquiry data and illustrates common problems as "problem parts." For example, if there is a lot of data related to network connection problems, it generates illustrations of Wi-Fi routers and connection errors. The generated illustrations are stored in a database.

[0490] Step 3:

[0491] Terminal

[0492] When a user accesses the system, the generated illustration is retrieved from the server. The retrieved illustration is then displayed on the terminal's user interface. This interface is designed so that users can operate it by dragging and dropping.

[0493] Step 4:

[0494] user

[0495] Users can view the displayed illustrations and select an illustration to visually represent their problem. The selected illustration is then placed on the device by dragging and dropping, visually representing the user's problem.

[0496] Step 5:

[0497] Terminal

[0498] The illustration information selected by the user and additional text information are sent to the server, including the ID of the illustration selected by the user, its placement information, and any supplementary information entered by the user.

[0499] Step 6:

[0500] server

[0501] The server analyzes the received illustration information and generates an appropriate query using an AI model. For example, if the illustrations "Wi-Fi router" and "connection error" are selected, the server generates the query "Your internet connection is currently unstable and there appears to be a problem with your Wi-Fi router."

[0502] Step 7:

[0503] Terminal

[0504] The generated query text is displayed to the user, who can review it and fine-tune it as needed, for example by adding details about the specific router model or condition.

[0505] Step 8:

[0506] user

[0507] The user makes a final check and presses the send button when the inquiry is complete. This action officially sends the user's inquiry.

[0508] Step 9:

[0509] server

[0510] The server receives the user's final inquiry text and forwards it to the customer support team. The inquiry is registered in the support system so that the support staff can respond promptly.

[0511] This is the specific processing flow of this system. This allows users to make accurate inquiries without knowing the technical details, and enables the support team to respond quickly and accurately.

[0512] Example 1

[0513] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0514] Current customer support systems make it difficult for users to properly explain their problems, especially for those without technical expertise. Furthermore, the vague nature of inquiries makes it difficult for support staff to respond efficiently. This results in lengthy response times and lower user satisfaction.

[0515] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0516] In this invention, the server includes means for learning the content of an inquiry and generating an image for visually expressing the inquiry, means for delivering the generated image to a user terminal and providing an interface that allows the user to visually select a problem, means for automatically generating an appropriate inquiry sentence based on the image information selected by the user, means for providing an interface that allows the user to fine-tune the generated inquiry sentence, and means for transmitting the final inquiry sentence adjusted by the user to a support center. This allows the user to visually select a problem and create an appropriate inquiry sentence even without technical expertise, enabling support staff to respond quickly and efficiently based on clear inquiry content.

[0517] "Inquiry Content" refers to the content of the problem or question provided by the user, and is a general term for information that is requested to be resolved by customer support.

[0518] "Visual representation" means using pictures, diagrams, automatically generated icons, etc. instead of words to make information or issues understandable at a glance.

[0519] "Pictures" are images or icons generated to visually represent the problem, allowing users to easily specify the problem.

[0520] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet, which allows the user to access and operate the system.

[0521] An "interface" refers to the screen and operation method used for interaction between the user and the system, providing an environment in which the user can operate intuitively.

[0522] A "generative model" is a type of artificial intelligence that is used to learn large amounts of data to find patterns and regularities and generate new data.

[0523] "Drag and drop" refers to a method of operation in a computer user interface where an object on the screen is moved using a mouse or touch and placed in another location.

[0524] "Support Center" refers to a specialized department or institution that accepts inquiries and problems from users and provides solutions.

[0525] This invention is a system that allows users to easily select a problem visually and automatically generates an appropriate query based on that selection. This system consists of a server, a terminal, and a user component, which operate in conjunction with each other. The details are given below.

[0526] Illustration generation and learning

[0527] server

[0528] The server collects past inquiry data from an inquiry database using database queries.

[0529] The server provides the collected data to a generative AI model (e.g., GPT-3) for learning. As a result of the learning, the AI ​​model illustrates common problems as "problem parts."

[0530] For example, if there are many inquiries about Internet connections, the server will generate illustrations of Wi-Fi routers and connection errors.

[0531] Providing an interface

[0532] Terminal

[0533] The device provides a user interface that displays illustrations delivered from the server, using a web application built with HTML5 and JavaScript.

[0534] The user interface is intuitive with a drag-and-drop feature, allowing users to select and place illustrations on the screen to visually represent the problem.

[0535] For example, a user might drag a "Wi-Fi router" icon across the screen and place a "Connection Error" icon next to it to represent the problem.

[0536] Selecting a problem and generating a query

[0537] user

[0538] The user uses the device interface to select the illustrations they need to visually represent the problem.

[0539] The selected illustration information is displayed in real time on the device, giving the user an overview of the problem, and once the selection is complete, the information is sent to the server.

[0540] server

[0541] The server receives the illustration information selected by the user and inputs it into a generative AI model (e.g., GPT-3) for analysis. As a result of the analysis, a natural language query is generated.

[0542] For example, if a user selects the illustrations of "Wi-Fi router" and "connection error," the server generates the sentence, "Your Internet connection is currently unstable and there appears to be a problem with your Wi-Fi router."

[0543] Fine-tuning the inquiry and final submission

[0544] Terminal

[0545] The generated query text is sent to the device and displayed to the user, who can fine-tune it and enter additional details (e.g., the model number of the Wi-Fi router) in the text boxes if necessary.

[0546] Once the user has finalized the content and pressed the "Send" button, the inquiry will be sent to customer support.

[0547] server

[0548] The server receives the final inquiry text sent from the device and forwards it to the customer support team, who then responds quickly and efficiently based on the information received.

[0549] Prompt Sentence Examples

[0550] As a concrete example, the following prompt sentence is input to the generative AI model:

[0551] If the user selects the illustrations of "Wi-Fi router" and "Connection error",

[0552] Appropriate message: Your internet connection is currently unstable and there appears to be an issue with your Wi-Fi router.

[0553] In this way, the system of the present invention combines visual problem selection with natural language query generation to provide an environment where users can accurately report problems without technical expertise, enabling customer support to respond quickly and efficiently.

[0554] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0555] Step 1:

[0556] Illustration generation

[0557] The server collects past inquiry data from an inquiry database. This data mainly includes inquiry content and category information. These data are extracted through database queries. Next, the server inputs this data into a generative AI model (e.g., GPT-3) and trains the model. Through training, the model generates pictures to visually represent common problems. The generated pictures are output in the form of icons, such as "Wi-Fi router" or "connection error."

[0558] Step 2:

[0559] Illustration distribution and display

[0560] The server delivers the generated images to the user's device. The user's device receives the images delivered from the server and displays them on a web application built with HTML5 and JavaScript. This interface has a drag-and-drop function, allowing users to easily operate it visually. This allows users to select how to express the problem in a picture. The input to the user's device is the image data from the server, and the output is an interface that the user can operate.

[0561] Step 3:

[0562] Question Selection

[0563] The user selects a picture on the device interface to visually represent their problem. The selected picture is displayed on the screen in real time and positioned by the user. For example, the user selects a "Wi-Fi router" icon and positions it on the screen along with a "Connection Error" icon. This action generates a selection, which is sent to the server. The user's input is the selected picture, and the output is the selection sent to the server.

[0564] Step 4:

[0565] Query generation

[0566] The server receives the picture information selected by the user and inputs that information into the generative AI model. The model analyzes this and generates a natural language query. For example, if the pictures of "Wi-Fi router" and "connection error" are selected, the model will generate the sentence "Currently, your internet connection is unstable and there appears to be a problem with your Wi-Fi router." The server's input is the picture information from the user, and its output is the generated query.

[0567] Step 5:

[0568] Fine-tuning of inquiry text

[0569] The device displays the generated query text to the user, who can refer to it and make any necessary adjustments. Specifically, the user can enter additional details, such as "Wi-Fi router model number," in the text box to complete the text. The user's input is the adjusted query text, and the output is the final query text.

[0570] Step 6:

[0571] Sending an inquiry

[0572] After the user has finished fine-tuning the query text, they press the "Send" button to send it to customer support. At this point, the final version of the query text is sent from the user's device to the server. The server then forwards the received final version of the query text to the support center. The server's input is the user's final query text, and its output is the query text sent to the support center.

[0573] (Application example 1)

[0574] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0575] When handling customer support inquiries at physical stores, it is difficult for customers to accurately communicate their problems, and it is also difficult for support staff to quickly understand and address the issues. For this reason, there is a need for a system that allows customers to intuitively report problems and enables support staff to respond quickly and accurately.

[0576] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0577] In this invention, the server includes means for learning the content of inquiries received by the support center and generating actions as problem parts for visually expressing the content, means for distributing the generated actions to a user device and providing an interface for the user to visually select a problem, means for automatically generating an appropriate inquiry sentence based on the action information selected by the user, means for providing an interface for the user to fine-tune the generated inquiry sentence, means for sending the final inquiry sentence adjusted by the user to a support staff member, means for using an artificial intelligence (AI) model to generate the inquiry sentence, means for sending the inquiry sentence generated by the artificial intelligence (AI) model to an appropriate staff member, and means for managing the transmission history. This allows customers to intuitively report problems and enables support staff to respond efficiently and quickly.

[0578] "Learning support inquiries" is the process of collecting inquiry data from users and training a machine learning model based on that data.

[0579] "Generating actions as parts of a problem to visually express it" means generating specific illustrations and action parts related to the content of the inquiry and displaying them visually.

[0580] "User Device" refers to an electronic device used by a user, such as a smartphone, tablet, or PC.

[0581] "Providing an interface" means providing an operation screen and operation means that users can operate intuitively.

[0582] "Action information" refers to information about specific illustrations and actions selected by the user on the interface.

[0583] "Automatically generating an appropriate inquiry sentence" means generating an inquiry sentence in natural language based on the selected action information.

[0584] A "fine-tunable interface" means providing an operation screen that allows the user to check, edit, and modify the generated query text.

[0585] "Support personnel" refers to staff or personnel in charge of customer support.

[0586] "Using an artificial intelligence (AI) model to generate a query sentence" means using AI technology to analyze the query content and generate a sentence in natural language.

[0587] "Managing transmission history" means saving and managing the history of the contents of inquiries sent.

[0588] A "prompt sentence" is the input text that an artificial intelligence (AI) model uses to generate a query sentence.

[0589] The present invention relates to a system that enables users to intuitively report problems at physical stores and allows support staff to quickly respond. Specific embodiments for realizing this system will be described below.

[0590] First, the server learns the content of inquiries received by the support center and generates behavior as problem parts to visually represent them. The hardware used for this is a high-performance server computer, and the software used includes open-source machine learning libraries and natural language processing libraries.

[0591] The server then delivers the generated actions to the user device, which is typically an electronic device such as a smartphone or tablet. The interface is designed to be intuitive and allows drag and drop, allowing users to visually select problems.

[0592] When a user selects an action using the interface, the action information is sent from the user device to the server. The server then automatically generates an appropriate query sentence based on the action information selected by the user. This automatic generation uses a generative AI model using the OpenAI API. The following is an example of a specific prompt sentence.

[0593] plaintext

[0594] User-selected illustrations: cashier trouble, product placement

[0595] Additional details: The cashier screen is frozen and the product cannot be found

[0596] Generate a query.

[0597] The generated query text is displayed on the user's device, allowing the user to fine-tune it and add any necessary details. For example, the user might add the cash register model number or the specific product name.

[0598] The final query text, fine-tuned by the user, is sent from the user's device to the support staff. The support staff can then respond quickly and appropriately based on the text sent. Furthermore, by managing the transmission history, past inquiry information can be referenced, which can be useful for future support responses.

[0599] Through this process, users can intuitively report problems and support staff can respond efficiently.

[0600] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0601] Step 1:

[0602] Learning inquiry content and generating problem parts

[0603] The server collects user inquiry data and uses a machine learning algorithm to learn from it. This generates action icons and illustrations to visually represent common problems. For example, if many inquiries are about "Wi-Fi router problems," the server generates an illustration of a Wi-Fi router.

[0604] Input: Past inquiry data

[0605] Data processing: Data analysis and icon generation using machine learning algorithms

[0606] Output: Action icons and illustrations

[0607] Step 2:

[0608] Delivery of action icons and illustrations

[0609] The server then delivers the generated action icons and illustrations to the user's device, providing a visual interface for the user to select problems. For example, a Wi-Fi router icon is displayed on the user's smartphone.

[0610] Input: Action icons and illustrations

[0611] Data processing: None (transfer only)

[0612] Output: Action icons and illustrations displayed on the user's device

[0613] Step 3:

[0614] Selecting and sending operational information

[0615] The user selects an action icon or illustration on the interface that corresponds to their problem, and the selection is sent from the user device to the server.

[0616] Input: User-selected action icons and illustrations

[0617] Data processing: collecting and formatting selected information

[0618] Output: Selection information sent to the server

[0619] Step 4:

[0620] Automatic generation of inquiry text

[0621] The server uses the received behavior information to call the generative AI model and generate an appropriate query sentence. At this time, the query sentence is automatically generated using a prompt sentence. For example, the prompt might be "Illustration selected by the user: Wi-Fi router, connection error."

[0622] Input: User-selected action information, prompt text

[0623] Data Processing: Natural Language Generation Using Generative AI Models

[0624] Output: Generated query text

[0625] Step 5:

[0626] Fine-tuning of inquiry text

[0627] The device displays the generated query text to the user and provides a screen where the user can review, edit, and fine-tune the content. For example, the user can enter the "Wi-Fi router model number" or "specific connection error situation."

[0628] Input: Generated query text

[0629] Data processing: User editing and fine-tuning

[0630] Output: The final query text that has been reviewed and refined by the user

[0631] Step 6:

[0632] Sending the final inquiry

[0633] The user checks the finely adjusted inquiry text and presses the send button to send it to the support staff. This transmission is performed from the user device via the server.

[0634] Input: The final query text that the user has reviewed and refined

[0635] Data processing: None (transfer only)

[0636] Output: Final query sent to support

[0637] Step 7:

[0638] Managing sending history

[0639] The server stores the final query text sent as a history and makes it available for future inquiries, allowing past history to be referenced.

[0640] Input: Final query text

[0641] Data processing: Save to database

[0642] Output: Saved query history

[0643] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0644] This system allows users to easily select a problem visually and automatically generates appropriate query sentences. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the quality and efficiency of queries can be improved.

[0645] overview

[0646] This system works in conjunction with a server, a terminal, and an emotion engine that recognizes the user's emotional state. The user selects an illustration via the terminal, and the server generates a query sentence based on the selection information and the user's emotional state. The specific operation and processing flow are explained below in natural language.

[0647] Illustration generation and learning

[0648] server

[0649] The server collects data from past support inquiries, including the content and metadata of users' past inquiries, and the collected data is used to train an AI model.

[0650] The AI ​​model uses this data to illustrate common problems as "problem parts." For example, if there is a lot of data about network connection problems, it will generate illustrations of Wi-Fi routers and connection errors. The generated illustrations are stored in a database.

[0651] Providing an interface

[0652] Terminal

[0653] The device retrieves the illustrations sent from the server and displays them on a user interface that users can manipulate using drag and drop.

[0654] If a user reports a poor internet connection issue, you can select and place a Wi-Fi router or connection error icon to visually represent the problem.

[0655] emotion recognition

[0656] Terminal

[0657] While the user is interacting with the interface, the emotion engine recognizes the user's emotions through facial recognition and voice analysis, collecting emotional data. For example, if the user is frustrated, their emotion is analyzed in real time.

[0658] Question Selection

[0659] user

[0660] Users can view the displayed illustrations and select an illustration to visually represent their problem. The selected illustration can then be placed on the device by dragging and dropping.

[0661] For example, if a user selects the illustrations "Wi-Fi router" and "connection error," that information is sent to the server.

[0662] Generate inquiry content

[0663] server

[0664] The server receives the illustration information selected by the user and the emotion data obtained by the emotion engine, and uses the AI ​​model to generate an appropriate query sentence.

[0665] For example, if the illustrations of "Wi-Fi router" and "connection error" are selected and the user is feeling irritated, the server will generate the sentence "Currently, your Internet connection is very unstable and there seems to be a problem with your Wi-Fi router."

[0666] Fine-tune and submit your inquiry

[0667] Terminal

[0668] The generated query text is displayed to the user, who can review it and fine-tune it as needed, for example by adding details about the specific router model or condition.

[0669] After the user makes some fine adjustments, they make a final check and press the send button, which officially sends the inquiry text.

[0670] Transfer to Support

[0671] server

[0672] The server receives the user's final inquiry text and forwards it to the customer support team. The inquiry is registered in the support system so that the support staff can respond promptly.

[0673] Emotional Escalation

[0674] server

[0675] The server also has the means to automatically escalate a query if the user's emotion exceeds a certain threshold. For example, if the user is very annoyed, the query will be given high priority and dealt with quickly.

[0676] In this way, this system allows users to visually express their problems and generates appropriate inquiry sentences based on that information and their emotional state. Furthermore, by combining it with an emotion engine, it becomes possible to respond according to the user's emotions, improving the efficiency and quality of customer support.

[0677] The processing flow will be explained below.

[0678] MODE FOR CARRYING OUT THE INVENTION

[0679] The system of the present invention allows users to visually select a problem and provides appropriate and prompt support by having the AI ​​generate an inquiry that reflects the user's feelings. The specific process flow is shown below.

[0680] Step 1:

[0681] server

[0682] The server collects past inquiry data and trains the AI ​​model. This data includes issues reported by users in the past, their details, and the results of their responses. Repeated data collection and training improves the accuracy of the model.

[0683] Step 2:

[0684] server

[0685] Based on the learned data, the AI ​​model illustrates common problems as "problem parts." For example, for the common problem of "unstable network," it generates illustrations of "Wi-Fi router" and "connection error."

[0686] Step 3:

[0687] Terminal

[0688] When a user starts the system, the terminal retrieves the generated illustrations from the server and displays them on the user interface, which has a drag-and-drop function for intuitive operation.

[0689] Step 4:

[0690] user

[0691] Users can select an illustration to visually represent their problem by viewing the displayed illustrations. For example, a user can select an illustration of a "Wi-Fi router" and a "connection error" and place them on the screen.

[0692] Step 5:

[0693] Terminal

[0694] The illustration information selected by the user and additional explanatory text are sent to the server. This information includes the ID of the selected illustration, its location, and any additional explanation entered by the user.

[0695] Step 6:

[0696] Terminal

[0697] While the user is selecting an illustration, the emotion engine recognizes the user's emotions in real time through facial recognition and voice analysis, and the emotion data is also sent to the server in parallel.

[0698] Step 7:

[0699] server

[0700] The server analyzes the received illustration information and emotion data and generates an appropriate query using an AI model. For example, if the illustrations of "Wi-Fi router" and "connection error" are selected and the user is feeling irritated, the server will generate the following sentence: "Currently, your internet connection is very unstable and there seems to be a problem with your Wi-Fi router."

[0701] Step 8:

[0702] Terminal

[0703] The generated query text is displayed to the user, who can review it and fine-tune it as needed, for example adding details about a specific router model or condition.

[0704] Step 9:

[0705] user

[0706] The user makes a final check and presses the send button when the inquiry is complete. This action officially sends the user's inquiry.

[0707] Step 10:

[0708] server

[0709] The server receives the user's final inquiry text and forwards it to the customer support team. The inquiry is registered in the support system so that the support staff can respond promptly.

[0710] Step 11:

[0711] server

[0712] If the user's emotion exceeds a certain threshold, the server escalates the query. For example, if the user is very annoyed, the query is set to high priority and requires immediate attention.

[0713] This system makes it easier for users to visually and emotionally express their problems, allowing customer support to understand the user's situation and emotions and respond appropriately.

[0714] Example 2

[0715] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0716] In modern customer support systems, it is extremely important for users to quickly and appropriately communicate the problems they face. However, conventional systems often require users to spend a lot of time and effort to accurately describe their problems. Furthermore, they respond without taking the user's emotional state into consideration, which can lead to a decline in the quality and efficiency of support. In particular, they may be unable to respond appropriately to emotionally charged users, which can increase their dissatisfaction. Therefore, there is a need for a system that allows users to visually express their problems and automatically generates inquiry text that takes their emotional state into account.

[0717] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0718] In this invention, the server includes means for learning the content of inquiries received by the support center and generating illustrations as problem parts to visually express the inquiries, means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem, and means for recognizing the user's emotions and generating an inquiry sentence that includes that information. This enables the user to visually express their problem and automatically generate a specific inquiry sentence that includes their emotional state.

[0719] "Learning the content of inquiries received by support" means collecting inquiry data from past users and analyzing and learning from that data using a machine learning model.

[0720] "Generating illustrations as problem parts" means generating illustrations to visually represent common problems faced by users based on inquiry data.

[0721] "Delivering to the user's device" means sending the generated illustration from the server to the user's device.

[0722] "Providing an interface that allows users to visually select problems" means providing an operation screen that allows users to visually select and express their own problems using displayed illustrations.

[0723] "Automatically generating appropriate inquiry text based on illustration information selected by the user" means using an AI model to automatically write down the corresponding inquiry content based on the information of the illustration selected by the user.

[0724] "Providing an interface that allows users to fine-tune query sentences" means providing an operation screen that allows users to edit and modify the generated query sentences.

[0725] "Emotion recognition" means analyzing a user's emotional state from their facial expressions and voice and acquiring it as data.

[0726] "Send inquiry to customer support" means that the user sends the inquiry that has been finally confirmed and corrected to the support staff.

[0727] "Using an AI model" means using a model that analyzes and generates data using algorithms based on machine learning and natural language processing.

[0728] "Placing by drag and drop" refers to the operation in which the user selects an illustration with the mouse, drags it to the desired location, and drops it.

[0729] This invention is a system that allows users to easily select a problem visually and automatically generates appropriate query sentences. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the quality and efficiency of inquiries can be improved. This system works in conjunction with a server, a user terminal, and an emotion engine that recognizes the user's emotional state.

[0730] Data collection and learning

[0731] The server collects past inquiry data, including text data and metadata (time, category, etc.). By training this data using a machine learning model (e.g., BERT, GPT-3), it is possible to understand the relationship between inquiry content and sentiment. This improves the accuracy of inquiry sentence generation, as described below.

[0732] Illustration generation and saving to database

[0733] The server uses the trained machine learning model to generate illustrations of "problem parts" that visually represent common problems. For example, if there is a lot of data related to network connection problems, it will generate illustrations of Wi-Fi routers and connection errors. The generated illustrations are stored in a database, and each part is also saved with its corresponding tag information.

[0734] Illustration distribution and user interface display

[0735] The device retrieves illustration data from the database and displays it on the user interface. The interface is built using HTML5 and JavaScript, and users can manipulate the illustrations with drag and drop. For example, a user who wants to report a poor internet connection can select an icon of a Wi-Fi router or a connection error.

[0736] emotion recognition

[0737] The device uses an emotion engine to recognize the user's emotions in real time while the user is operating the interface. Specifically, an emotion recognition engine (e.g., OpenFace or SpeechEmotionRecognition) is used to collect emotion data from the user's facial expressions and voice. The emotion data is sent from the device to the server and used to generate query sentences.

[0738] Generate and refine inquiries

[0739] The server receives the illustration information and emotion data selected by the user and automatically generates an appropriate query using an AI model (e.g., GPT-3). For example, if the illustration of "Wi-Fi router" and "connection error" is selected and the emotion data indicates frustration, the server generates a query such as, "Your internet connection is currently very unstable, and there appears to be a problem with your Wi-Fi router." The generated query is then sent to the user's device, where the user can review the displayed message and make adjustments as necessary. For example, the user can add information about the router's specific model number or other detailed information about the situation.

[0740] Submitting a ticket and routing to support

[0741] The user checks the final query sentence after making some fine adjustments and presses the send button, which sends the query sentence to the server.

[0742] The server forwards the received inquiry text to the customer support team and registers the inquiry content in the support system. In particular, if the emotional data exceeds a certain threshold, the inquiry is automatically escalated and set to a high priority.

[0743] Examples of concrete examples and prompts

[0744] Here are some examples of prompts to input to a generative AI model:

[0745] "We are currently experiencing connectivity issues with our Wi-Fi router. The internet connection is very unstable and frequently drops out. Users are extremely frustrated with this issue and are requesting a prompt response from support staff."

[0746] An example of a query that might be generated in response to this prompt is:

[0747] "Currently, my internet connection is very unstable and there seems to be an issue with my Wi-Fi router. I am frequently disconnected and this is very frustrating. I would appreciate a quick response."

[0748] Through the above steps and concrete examples, users can visually express their problems and automatically generate specific inquiry sentences that include their emotional state, which can greatly improve the efficiency and quality of customer support.

[0749] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0750] Step 1: Data collection and learning

[0751] The server collects past inquiry data from a database. As input, it receives text data, inquiry content, and metadata (categories, timestamps, etc.). The collected data is analyzed and trained using a natural language processing model (e.g., BERT, GPT-3). Data processing involves preprocessing the text (tokenization, removal of stop words, etc.). As output, a model that has learned the association between inquiry content patterns and sentiment is obtained.

[0752] Specific behavior:

[0753] The server periodically retrieves new query data from the database, preprocesses the text data using Python, and then trains the data using a machine learning model.

[0754] Step 2: Generate illustrations and save them to the database

[0755] The server uses the trained model to generate illustrations to visually represent the user's problem. As input, it receives the results of the trained model and query data. As data processing, it uses a generative algorithm (e.g., GAN, a deep learning-based image generation model) to create illustrations. As output, it obtains illustrations of the generated problem parts. These illustrations are stored in a database.

[0756] Specific behavior:

[0757] The server uses Python's PIL library to generate illustrations and stores them in MongoDB, with each illustration also containing corresponding tag information.

[0758] Step 3: Delivering illustrations and displaying the user interface

[0759] The device retrieves illustration data from the server and displays it on the user interface. As input, it receives illustration data from the server. For data processing, it retrieves the data using an AJAX request and renders it on the interface using HTML5 and JavaScript. As output, it provides an interface that the user can operate using drag and drop.

[0760] Specific behavior:

[0761] The device uses AJAX requests to pull down illustration data from the server and uses Dragula.js to enable drag-and-drop operation.

[0762] Step 4: Emotion Recognition

[0763] The device uses an emotion engine to recognize the user's emotions in real time. It receives camera footage and audio data as input. It analyzes the data using an emotion engine (e.g., OpenFace, SpeechEmotionRecognition) to generate user emotion data. The analyzed emotion data is obtained as output. The emotion data is sent from the device to the server.

[0764] Specific behavior:

[0765] The device uses WebRTC to capture real-time data from the camera and microphone and passes it to an emotion recognition engine for analysis.

[0766] Step 5: Selecting the problem illustration

[0767] The user looks at the displayed illustrations and selects one to visually represent their problem. The system receives the illustration information selected by the user as input. The system processes the data by selecting and placing the illustrations using drag and drop. The system obtains the selected illustration information as output, and sends it to the server.

[0768] Specific behavior:

[0769] Users select illustrations via drag-and-drop and click a button to submit the information.

[0770] Step 6: Generate a query

[0771] The server automatically generates an appropriate query sentence using an AI model based on the selected illustration information and emotion data. The server receives the illustration information and emotion data as input. The server uses an AI model (e.g., GPT-3) to generate a query sentence as data calculation. The generated query sentence is obtained as output.

[0772] Specific behavior:

[0773] The server uses the Flask framework and Python scripts to invoke the AI ​​model and generate queries.

[0774] Step 7: View and fine-tune your inquiry

[0775] The terminal displays the generated query sentence to the user. As input, it receives the query sentence from the server. As output, it provides an interface that the user can fine-tune. The user can edit and modify the sentence through this interface.

[0776] Specific behavior:

[0777] The terminal displays the generated text in a text area, allowing the user to freely edit it.

[0778] Step 8: Submitting and forwarding inquiries

[0779] The user checks the final adjusted query sentence and clicks the send button. The revised query sentence is received as input. The query sentence is sent to the server as data processing. The query sentence received by the server is forwarded to customer support as output.

[0780] Specific behavior:

[0781] When the user clicks the submit button, the data is sent in a POST request to the server, which then forwards the data to the customer support system using a REST API.

[0782] Step 9: Emotional Escalation

[0783] The server analyzes the user's emotional data and automatically escalates queries when a certain threshold is exceeded. The input is the emotional data. The data calculation evaluates the emotional data that exceeds the threshold. The output is a query with a high priority.

[0784] Specific behavior:

[0785] The server uses a Python script to analyze the sentiment data and, if certain conditions are met, sets an escalation flag and notifies the support team.

[0786] (Application example 2)

[0787] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0788] Conventional customer support systems have had problems with users being unable to accurately and quickly communicate their problems, resulting in poor quality and efficiency of inquiries, especially when the problem is complex or the user is in an emotional state. Furthermore, the inability to respond appropriately to emotional users risks reducing customer satisfaction. To address these issues, the present invention aims to provide a system that makes it easy for users to visually select a problem and automatically generates appropriate inquiry sentences that take the user's emotional state into account.

[0789] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0790] In this invention, the server includes means for learning the content of a query and generating illustrations as problem parts for visually expressing the query, means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem, means for recognizing the user's emotional state and acquiring emotional data, and means for generating a query sentence based on the emotional data and illustration information, thereby enabling the user to visually select a problem and generating an appropriate query sentence according to the user's emotional state.

[0791] "Support" means the assistance provided to resolve any problems or questions regarding a service or product.

[0792] "Query" means the details of a question or problem submitted by a User to Support.

[0793] "Problem parts" are elements of a problem that are visually and symbolically represented so that users can select them visually.

[0794] An "illustration" is visual content created to visually represent an issue or situation.

[0795] "User terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[0796] A "visual question selection interface" is a user interface that allows a user to select a question from a visually displayed set of options.

[0797] "Emotional state" refers to the user's psychological and emotional state, which is determined through facial recognition and voice analysis.

[0798] "Emotional Data" refers to data about a user's emotional state, captured in real time.

[0799] "Automatic generation" refers to the process of using AI technology and algorithms to automatically generate query sentences based on pre-set rules and data.

[0800] "Customer support" refers to the support team or department that responds to customer inquiries.

[0801] This invention is a system that automatically generates query sentences that make it easier to visually select problems and take into account emotional states. This system operates by combining a user terminal, a server, and an emotion engine, and can improve the quality and efficiency of user queries.

[0802] System Configuration

[0803] The system consists of the following main components:

[0804] server

[0805] Learning and illustration generation

[0806] The server collects past inquiry data and uses an AI model to illustrate common problems as "problem parts," including information about network connection issues and product defects. The server uses an AI model (e.g., a GPT-3 model) to generate illustrations and stores them in a database.

[0807] Query generation

[0808] The server uses an AI model to automatically generate a query based on the illustration information selected by the user and the emotional data obtained by the emotion engine. For example, if a user selects the illustrations of "connection error" and "Wi-Fi router" and their emotional state is "irritated," the server generates the sentence, "Currently, your internet connection is very unstable, and there seems to be a problem with your Wi-Fi router."

[0809] User Device

[0810] Visual Interface

[0811] The device retrieves the illustrations sent from the server and displays them on a user interface that allows drag-and-drop operation, allowing users to intuitively select problems.

[0812] emotion recognition

[0813] The user device is equipped with an emotion engine and uses a camera and microphone to recognize the user's emotional state in real time, allowing it to collect emotional data while the user is operating the device.

[0814] Example of operation

[0815] Let's take the example of an unstable Wi-Fi connection in a store. A user uses their smartphone to report a problem with the store's Wi-Fi connection. They select illustrations of "Wi-Fi router" and "connection error" on the device, and these illustrations are sent to the server. Using the device's camera and microphone, the emotion engine recognizes that the user is frustrated. The server uses an AI model based on the illustration information and emotion data to generate a query.

[0816] Software and hardware used

[0817] AI model: GPT-3

[0818] Emotion recognition library: FER (Face Emotion Recognition)

[0819] Hardware: Smartphone, server (cloud environment)

[0820] Prompt Sentence Examples

[0821] Below are some examples of specific prompt sentences.

[0822] Select the illustration below and the emotion is as follows:

[0823] Illustration: ['Wi-Fi router', 'Connection error']

[0824] Emotion data: {'Anger': 0.8, 'Happiness': 0.1}

[0825] As described above, this system allows users to visually select a problem and generate an appropriate inquiry sentence based on their emotional state, thereby improving the efficiency and quality of customer support.

[0826] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0827] Step 1:

[0828] The server learns from past inquiry data and uses an AI model to illustrate common problem areas. Specifically, it collects data on network connection issues and product defects and generates illustrations to visually represent them. The generated illustrations are stored in a database.

[0829] Input: Past inquiry data

[0830] Output: Illustration of the problem part

[0831] Step 2:

[0832] The terminal retrieves illustrations delivered from the server and displays them on the user interface. Users visually select their own problems using drag-and-drop operations. This interface is designed to be intuitive.

[0833] Input: Illustration delivered from the server

[0834] Output: User's question selection information

[0835] Step 3:

[0836] The device transmits the illustration information visually selected by the user to the server. At the same time, the device uses a camera and microphone to recognize the user's emotional state in real time and acquire emotional data. This data is then analyzed by the emotion engine.

[0837] Input: User-selected illustration information, real-time emotion data

[0838] Output: Send illustration information and emotion data to the server

[0839] Step 4:

[0840] The server receives illustration information and emotion data sent by the user and uses the AI ​​model to automatically generate a query based on them. For example, if the illustrations of "Connection Error" and "Wi-Fi Router" are selected and the user is feeling irritated, the server will generate the sentence "Currently, your internet connection is very unstable and there seems to be a problem with your Wi-Fi router."

[0841] Input: illustration information, emotion data

[0842] Output: Generated query text

[0843] Step 5:

[0844] The device displays the generated query to the user and provides an interface for fine-tuning, allowing the user to modify the query as needed by adding specific router model numbers and status details.

[0845] Input: Generated query text

[0846] Output: User-adjusted query text

[0847] Step 6:

[0848] After the user has fine-tuned the query text, they send it to the server, and by pressing the send button on their device, the final query text is forwarded to the customer support team.

[0849] Input: Tweaked query text

[0850] Output: Send to customer support team

[0851] Step 7:

[0852] The server has the means to automatically escalate a query if the user's emotional state exceeds a certain threshold: for example, if the user is very annoyed, the query will be given high priority and dealt with quickly.

[0853] Input: User emotion data

[0854] Output: Escalation of tickets with high priority

[0855] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0856] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0857] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0858] [Third embodiment]

[0859] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0860] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0861] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0862] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0863] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0864] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0865] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0866] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0867] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0868] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0869] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0870] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0871] This invention is a system that allows users to easily select a problem visually and automatically generates appropriate query sentences. The program of this system is configured based on the roles of the user, terminal, and server. The specific operation and processing flow are explained below in natural language.

[0872] overview

[0873] This system operates in a mutually linked manner between the server, terminals, and users. The user selects an illustration via the terminal, and the server generates a query based on the selected information. The operation of the entire system is explained below.

[0874] Illustration generation and learning

[0875] server

[0876] The server collects past inquiry data, and the AI ​​model learns from this data. This allows it to illustrate common problems as "problem parts."

[0877] For example, if there are a lot of inquiries about network issues, an illustration of a Wi-Fi router or connection error will be generated.

[0878] Providing an interface

[0879] Terminal

[0880] The terminal provides a user interface that displays illustrations delivered from the server. This interface supports drag and drop, allowing users to operate it intuitively.

[0881] For example, if a user wants to report a poor internet connection issue, they can select and place an icon of a Wi-Fi router or connection error to visually represent the problem.

[0882] Question Selection

[0883] user

[0884] Using the device interface, users select an illustration to visually represent their problem, which is then placed on the device and displayed as an overview of the problem.

[0885] For example, if a user selects the illustrations "Wi-Fi router" and "connection error," this information is sent to the server.

[0886] Generate inquiry content

[0887] server

[0888] The server receives the illustration information selected by the user, analyzes it using an AI model, and generates a natural language query based on the analysis results.

[0889] For example, if the illustrations of "Wi-Fi router" and "connection error" are selected, the server will generate the sentence "Currently, your Internet connection is unstable and there appears to be a problem with your Wi-Fi router."

[0890] Fine-tune and submit your inquiry

[0891] Terminal

[0892] The generated query is sent to the device and displayed to the user, who can then fine-tune it and add details like the model number or specific symptoms if needed.

[0893] For example, a user enters the "Wi-Fi router model number" as an additional detail.

[0894] After making a final confirmation, the user presses the send button to send the inquiry to customer support.

[0895] Transfer to Support

[0896] server

[0897] The server receives the user's final inquiry text and forwards it to the customer support team, who can then respond promptly based on the information received.

[0898] In this way, this system provides an environment where users can visually express their problems and generate and adjust appropriate inquiry sentences based on that information. This system allows users to easily make accurate inquiries even if they are not familiar with technical terminology, and customer support can respond efficiently.

[0899] The processing flow will be explained below.

[0900] Program processing flow

[0901] Step 1:

[0902] server

[0903] The server collects data from past support inquiries, including issues users have reported and detailed metadata associated with them, and stores the collected data in a database, where the AI ​​model learns from it.

[0904] Step 2:

[0905] server

[0906] Using an AI model, the system analyzes collected inquiry data and illustrates common problems as "problem parts." For example, if there is a lot of data related to network connection problems, it generates illustrations of Wi-Fi routers and connection errors. The generated illustrations are stored in a database.

[0907] Step 3:

[0908] Terminal

[0909] When a user accesses the system, the generated illustration is retrieved from the server. The retrieved illustration is then displayed on the terminal's user interface. This interface is designed so that users can operate it by dragging and dropping.

[0910] Step 4:

[0911] user

[0912] Users can view the displayed illustrations and select an illustration to visually represent their problem. The selected illustration is then placed on the device by dragging and dropping, visually representing the user's problem.

[0913] Step 5:

[0914] Terminal

[0915] The illustration information selected by the user and additional text information are sent to the server, including the ID of the illustration selected by the user, its placement information, and any supplementary information entered by the user.

[0916] Step 6:

[0917] server

[0918] The server analyzes the received illustration information and generates an appropriate query using an AI model. For example, if the illustrations "Wi-Fi router" and "connection error" are selected, the server generates the query "Your internet connection is currently unstable and there appears to be a problem with your Wi-Fi router."

[0919] Step 7:

[0920] Terminal

[0921] The generated query text is displayed to the user, who can review it and fine-tune it as needed, for example by adding details about the specific router model or condition.

[0922] Step 8:

[0923] user

[0924] The user makes a final check and presses the send button when the inquiry is complete. This action officially sends the user's inquiry.

[0925] Step 9:

[0926] server

[0927] The server receives the user's final inquiry text and forwards it to the customer support team. The inquiry is registered in the support system so that the support staff can respond promptly.

[0928] This is the specific processing flow of this system. This allows users to make accurate inquiries without knowing the technical details, and enables the support team to respond quickly and accurately.

[0929] Example 1

[0930] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0931] Current customer support systems make it difficult for users to properly explain their problems, especially for those without technical expertise. Furthermore, the vague nature of inquiries makes it difficult for support staff to respond efficiently. This results in lengthy response times and lower user satisfaction.

[0932] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0933] In this invention, the server includes means for learning the content of an inquiry and generating an image for visually expressing the inquiry, means for delivering the generated image to a user terminal and providing an interface that allows the user to visually select a problem, means for automatically generating an appropriate inquiry sentence based on the image information selected by the user, means for providing an interface that allows the user to fine-tune the generated inquiry sentence, and means for transmitting the final inquiry sentence adjusted by the user to a support center. This allows the user to visually select a problem and create an appropriate inquiry sentence even without technical expertise, enabling support staff to respond quickly and efficiently based on clear inquiry content.

[0934] "Inquiry Content" refers to the content of the problem or question provided by the user, and is a general term for information that is requested to be resolved by customer support.

[0935] "Visual representation" means using pictures, diagrams, automatically generated icons, etc. instead of words to make information or issues understandable at a glance.

[0936] "Pictures" are images or icons generated to visually represent the problem, allowing users to easily specify the problem.

[0937] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet, which allows the user to access and operate the system.

[0938] An "interface" refers to the screen and operation method used for interaction between the user and the system, providing an environment in which the user can operate intuitively.

[0939] A "generative model" is a type of artificial intelligence that is used to learn large amounts of data to find patterns and regularities and generate new data.

[0940] "Drag and drop" refers to a method of operation in a computer user interface where an object on the screen is moved using a mouse or touch and placed in another location.

[0941] "Support Center" refers to a specialized department or institution that accepts inquiries and problems from users and provides solutions.

[0942] This invention is a system that allows users to easily select a problem visually and automatically generates an appropriate query based on that selection. This system consists of a server, a terminal, and a user component, which operate in conjunction with each other. The details are given below.

[0943] Illustration generation and learning

[0944] server

[0945] The server collects past inquiry data from an inquiry database using database queries.

[0946] The server provides the collected data to a generative AI model (e.g., GPT-3) for learning. As a result of the learning, the AI ​​model illustrates common problems as "problem parts."

[0947] For example, if there are many inquiries about Internet connections, the server will generate illustrations of Wi-Fi routers and connection errors.

[0948] Providing an interface

[0949] Terminal

[0950] The device provides a user interface that displays illustrations delivered from the server, using a web application built with HTML5 and JavaScript.

[0951] The user interface is intuitive with a drag-and-drop feature, allowing users to select and place illustrations on the screen to visually represent the problem.

[0952] For example, a user might drag a "Wi-Fi router" icon across the screen and place a "Connection Error" icon next to it to represent the problem.

[0953] Selecting a problem and generating a query

[0954] user

[0955] The user uses the device interface to select the illustrations they need to visually represent the problem.

[0956] The selected illustration information is displayed in real time on the device, giving the user an overview of the problem, and once the selection is complete, the information is sent to the server.

[0957] server

[0958] The server receives the illustration information selected by the user and inputs it into a generative AI model (e.g., GPT-3) for analysis. As a result of the analysis, a natural language query is generated.

[0959] For example, if a user selects the illustrations of "Wi-Fi router" and "connection error," the server generates the sentence, "Your Internet connection is currently unstable and there appears to be a problem with your Wi-Fi router."

[0960] Fine-tuning the inquiry and final submission

[0961] Terminal

[0962] The generated query text is sent to the device and displayed to the user, who can fine-tune it and enter additional details (e.g., the model number of the Wi-Fi router) in the text boxes if necessary.

[0963] Once the user has finalized the content and pressed the "Send" button, the inquiry will be sent to customer support.

[0964] server

[0965] The server receives the final inquiry text sent from the device and forwards it to the customer support team, who then responds quickly and efficiently based on the information received.

[0966] Prompt Sentence Examples

[0967] As a concrete example, the following prompt sentence is input to the generative AI model:

[0968] If the user selects the illustrations of "Wi-Fi router" and "Connection error",

[0969] Appropriate message: Your internet connection is currently unstable and there appears to be an issue with your Wi-Fi router.

[0970] In this way, the system of the present invention combines visual problem selection with natural language query generation to provide an environment where users can accurately report problems without technical expertise, enabling customer support to respond quickly and efficiently.

[0971] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0972] Step 1:

[0973] Illustration generation

[0974] The server collects past inquiry data from an inquiry database. This data mainly includes inquiry content and category information. These data are extracted through database queries. Next, the server inputs this data into a generative AI model (e.g., GPT-3) and trains the model. Through training, the model generates pictures to visually represent common problems. The generated pictures are output in the form of icons, such as "Wi-Fi router" or "connection error."

[0975] Step 2:

[0976] Illustration distribution and display

[0977] The server delivers the generated images to the user's device. The user's device receives the images delivered from the server and displays them on a web application built with HTML5 and JavaScript. This interface has a drag-and-drop function, allowing users to easily operate it visually. This allows users to select how to express the problem in a picture. The input to the user's device is the image data from the server, and the output is an interface that the user can operate.

[0978] Step 3:

[0979] Question Selection

[0980] The user selects a picture on the device interface to visually represent their problem. The selected picture is displayed on the screen in real time and positioned by the user. For example, the user selects a "Wi-Fi router" icon and positions it on the screen along with a "Connection Error" icon. This action generates a selection, which is sent to the server. The user's input is the selected picture, and the output is the selection sent to the server.

[0981] Step 4:

[0982] Query generation

[0983] The server receives the picture information selected by the user and inputs that information into the generative AI model. The model analyzes this and generates a natural language query. For example, if the pictures of "Wi-Fi router" and "connection error" are selected, the model will generate the sentence "Currently, your internet connection is unstable and there appears to be a problem with your Wi-Fi router." The server's input is the picture information from the user, and its output is the generated query.

[0984] Step 5:

[0985] Fine-tuning of inquiry text

[0986] The device displays the generated query text to the user, who can refer to it and make any necessary adjustments. Specifically, the user can enter additional details, such as "Wi-Fi router model number," in the text box to complete the text. The user's input is the adjusted query text, and the output is the final query text.

[0987] Step 6:

[0988] Sending an inquiry

[0989] After the user has finished fine-tuning the query text, they press the "Send" button to send it to customer support. At this point, the final version of the query text is sent from the user's device to the server. The server then forwards the received final version of the query text to the support center. The server's input is the user's final query text, and its output is the query text sent to the support center.

[0990] (Application example 1)

[0991] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0992] When handling customer support inquiries at physical stores, it is difficult for customers to accurately communicate their problems, and it is also difficult for support staff to quickly understand and address the issues. For this reason, there is a need for a system that allows customers to intuitively report problems and enables support staff to respond quickly and accurately.

[0993] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0994] In this invention, the server includes means for learning the content of inquiries received by the support center and generating actions as problem parts for visually expressing the content, means for distributing the generated actions to a user device and providing an interface for the user to visually select a problem, means for automatically generating an appropriate inquiry sentence based on the action information selected by the user, means for providing an interface for the user to fine-tune the generated inquiry sentence, means for sending the final inquiry sentence adjusted by the user to a support staff member, means for using an artificial intelligence (AI) model to generate the inquiry sentence, means for sending the inquiry sentence generated by the artificial intelligence (AI) model to an appropriate staff member, and means for managing the transmission history. This allows customers to intuitively report problems and enables support staff to respond efficiently and quickly.

[0995] "Learning support inquiries" is the process of collecting inquiry data from users and training a machine learning model based on that data.

[0996] "Generating actions as parts of a problem to visually express it" means generating specific illustrations and action parts related to the content of the inquiry and displaying them visually.

[0997] "User Device" refers to an electronic device used by a user, such as a smartphone, tablet, or PC.

[0998] "Providing an interface" means providing an operation screen and operation means that users can operate intuitively.

[0999] "Action information" refers to information about specific illustrations and actions selected by the user on the interface.

[1000] "Automatically generating an appropriate inquiry sentence" means generating an inquiry sentence in natural language based on the selected action information.

[1001] A "fine-tunable interface" means providing an operation screen that allows the user to check, edit, and modify the generated query text.

[1002] "Support personnel" refers to staff or personnel in charge of customer support.

[1003] "Using an artificial intelligence (AI) model to generate a query sentence" means using AI technology to analyze the query content and generate a sentence in natural language.

[1004] "Managing transmission history" means saving and managing the history of the contents of inquiries sent.

[1005] A "prompt sentence" is the input text that an artificial intelligence (AI) model uses to generate a query sentence.

[1006] The present invention relates to a system that enables users to intuitively report problems at physical stores and allows support staff to quickly respond. Specific embodiments for realizing this system will be described below.

[1007] First, the server learns the content of inquiries received by the support center and generates behavior as problem parts to visually represent them. The hardware used for this is a high-performance server computer, and the software used includes open-source machine learning libraries and natural language processing libraries.

[1008] The server then delivers the generated actions to the user device, which is typically an electronic device such as a smartphone or tablet. The interface is designed to be intuitive and allows drag and drop, allowing users to visually select problems.

[1009] When a user selects an action using the interface, the action information is sent from the user device to the server. The server then automatically generates an appropriate query sentence based on the action information selected by the user. This automatic generation uses a generative AI model using the OpenAI API. The following is an example of a specific prompt sentence.

[1010] plaintext

[1011] User-selected illustrations: cashier trouble, product placement

[1012] Additional details: The cashier screen is frozen and the product cannot be found

[1013] Generate a query.

[1014] The generated query text is displayed on the user's device, allowing the user to fine-tune it and add any necessary details. For example, the user might add the cash register model number or the specific product name.

[1015] The final query text, fine-tuned by the user, is sent from the user's device to the support staff. The support staff can then respond quickly and appropriately based on the text sent. Furthermore, by managing the transmission history, past inquiry information can be referenced, which can be useful for future support responses.

[1016] Through this process, users can intuitively report problems and support staff can respond efficiently.

[1017] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1018] Step 1:

[1019] Learning inquiry content and generating problem parts

[1020] The server collects user inquiry data and uses a machine learning algorithm to learn from it. This generates action icons and illustrations to visually represent common problems. For example, if many inquiries are about "Wi-Fi router problems," the server generates an illustration of a Wi-Fi router.

[1021] Input: Past inquiry data

[1022] Data processing: Data analysis and icon generation using machine learning algorithms

[1023] Output: Action icons and illustrations

[1024] Step 2:

[1025] Delivery of action icons and illustrations

[1026] The server then delivers the generated action icons and illustrations to the user's device, providing a visual interface for the user to select problems. For example, a Wi-Fi router icon is displayed on the user's smartphone.

[1027] Input: Action icons and illustrations

[1028] Data processing: None (transfer only)

[1029] Output: Action icons and illustrations displayed on the user's device

[1030] Step 3:

[1031] Selecting and sending operational information

[1032] The user selects an action icon or illustration on the interface that corresponds to their problem, and the selection is sent from the user device to the server.

[1033] Input: User-selected action icons and illustrations

[1034] Data processing: collecting and formatting selected information

[1035] Output: Selection information sent to the server

[1036] Step 4:

[1037] Automatic generation of inquiry text

[1038] The server uses the received behavior information to call the generative AI model and generate an appropriate query sentence. At this time, the query sentence is automatically generated using a prompt sentence. For example, the prompt might be "Illustration selected by the user: Wi-Fi router, connection error."

[1039] Input: User-selected action information, prompt text

[1040] Data Processing: Natural Language Generation Using Generative AI Models

[1041] Output: Generated query text

[1042] Step 5:

[1043] Fine-tuning of inquiry text

[1044] The device displays the generated query text to the user and provides a screen where the user can review, edit, and fine-tune the content. For example, the user can enter the "Wi-Fi router model number" or "specific connection error situation."

[1045] Input: Generated query text

[1046] Data processing: User editing and fine-tuning

[1047] Output: The final query text that has been reviewed and refined by the user

[1048] Step 6:

[1049] Sending the final inquiry

[1050] The user checks the finely adjusted inquiry text and presses the send button to send it to the support staff. This transmission is performed from the user device via the server.

[1051] Input: The final query text that the user has reviewed and refined

[1052] Data processing: None (transfer only)

[1053] Output: Final query sent to support

[1054] Step 7:

[1055] Managing sending history

[1056] The server stores the final query text sent as a history and makes it available for future inquiries, allowing past history to be referenced.

[1057] Input: Final query text

[1058] Data processing: Save to database

[1059] Output: Saved query history

[1060] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1061] This system allows users to easily select a problem visually and automatically generates appropriate query sentences. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the quality and efficiency of queries can be improved.

[1062] overview

[1063] This system works in conjunction with a server, a terminal, and an emotion engine that recognizes the user's emotional state. The user selects an illustration via the terminal, and the server generates a query sentence based on the selection information and the user's emotional state. The specific operation and processing flow are explained below in natural language.

[1064] Illustration generation and learning

[1065] server

[1066] The server collects data from past support inquiries, including the content and metadata of users' past inquiries, and the collected data is used to train an AI model.

[1067] The AI ​​model uses this data to illustrate common problems as "problem parts." For example, if there is a lot of data about network connection problems, it will generate illustrations of Wi-Fi routers and connection errors. The generated illustrations are stored in a database.

[1068] Providing an interface

[1069] Terminal

[1070] The device retrieves the illustrations sent from the server and displays them on a user interface that users can manipulate using drag and drop.

[1071] If a user reports a poor internet connection issue, you can select and place a Wi-Fi router or connection error icon to visually represent the problem.

[1072] emotion recognition

[1073] Terminal

[1074] While the user is interacting with the interface, the emotion engine recognizes the user's emotions through facial recognition and voice analysis, collecting emotional data. For example, if the user is frustrated, their emotion is analyzed in real time.

[1075] Question Selection

[1076] user

[1077] Users can view the displayed illustrations and select an illustration to visually represent their problem. The selected illustration can then be placed on the device by dragging and dropping.

[1078] For example, if a user selects the illustrations "Wi-Fi router" and "connection error," that information is sent to the server.

[1079] Generate inquiry content

[1080] server

[1081] The server receives the illustration information selected by the user and the emotion data obtained by the emotion engine, and uses the AI ​​model to generate an appropriate query sentence.

[1082] For example, if the illustrations of "Wi-Fi router" and "connection error" are selected and the user is feeling irritated, the server will generate the sentence "Currently, your Internet connection is very unstable and there seems to be a problem with your Wi-Fi router."

[1083] Fine-tune and submit your inquiry

[1084] Terminal

[1085] The generated query text is displayed to the user, who can review it and fine-tune it as needed, for example by adding details about the specific router model or condition.

[1086] After the user makes some fine adjustments, they make a final check and press the send button, which officially sends the inquiry text.

[1087] Transfer to Support

[1088] server

[1089] The server receives the user's final inquiry text and forwards it to the customer support team. The inquiry is registered in the support system so that the support staff can respond promptly.

[1090] Emotional Escalation

[1091] server

[1092] The server also has the means to automatically escalate a query if the user's emotion exceeds a certain threshold. For example, if the user is very annoyed, the query will be given high priority and dealt with quickly.

[1093] In this way, this system allows users to visually express their problems and generates appropriate inquiry sentences based on that information and their emotional state. Furthermore, by combining it with an emotion engine, it becomes possible to respond according to the user's emotions, improving the efficiency and quality of customer support.

[1094] The processing flow will be explained below.

[1095] MODE FOR CARRYING OUT THE INVENTION

[1096] The system of the present invention allows users to visually select a problem and provides appropriate and prompt support by having the AI ​​generate an inquiry that reflects the user's feelings. The specific process flow is shown below.

[1097] Step 1:

[1098] server

[1099] The server collects past inquiry data and trains the AI ​​model. This data includes issues reported by users in the past, their details, and the results of their responses. Repeated data collection and training improves the accuracy of the model.

[1100] Step 2:

[1101] server

[1102] Based on the learned data, the AI ​​model illustrates common problems as "problem parts." For example, for the common problem of "unstable network," it generates illustrations of "Wi-Fi router" and "connection error."

[1103] Step 3:

[1104] Terminal

[1105] When a user starts the system, the terminal retrieves the generated illustrations from the server and displays them on the user interface, which has a drag-and-drop function for intuitive operation.

[1106] Step 4:

[1107] user

[1108] Users can select an illustration to visually represent their problem by viewing the displayed illustrations. For example, a user can select an illustration of a "Wi-Fi router" and a "connection error" and place them on the screen.

[1109] Step 5:

[1110] Terminal

[1111] The illustration information selected by the user and additional explanatory text are sent to the server. This information includes the ID of the selected illustration, its location, and any additional explanation entered by the user.

[1112] Step 6:

[1113] Terminal

[1114] While the user is selecting an illustration, the emotion engine recognizes the user's emotions in real time through facial recognition and voice analysis, and the emotion data is also sent to the server in parallel.

[1115] Step 7:

[1116] server

[1117] The server analyzes the received illustration information and emotion data and generates an appropriate query using an AI model. For example, if the illustrations of "Wi-Fi router" and "connection error" are selected and the user is feeling irritated, the server will generate the following sentence: "Currently, your internet connection is very unstable and there seems to be a problem with your Wi-Fi router."

[1118] Step 8:

[1119] Terminal

[1120] The generated query text is displayed to the user, who can review it and fine-tune it as needed, for example adding details about a specific router model or condition.

[1121] Step 9:

[1122] user

[1123] The user makes a final check and presses the send button when the inquiry is complete. This action officially sends the user's inquiry.

[1124] Step 10:

[1125] server

[1126] The server receives the user's final inquiry text and forwards it to the customer support team. The inquiry is registered in the support system so that the support staff can respond promptly.

[1127] Step 11:

[1128] server

[1129] If the user's emotion exceeds a certain threshold, the server escalates the query. For example, if the user is very annoyed, the query is set to high priority and requires immediate attention.

[1130] This system makes it easier for users to visually and emotionally express their problems, allowing customer support to understand the user's situation and emotions and respond appropriately.

[1131] Example 2

[1132] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1133] In modern customer support systems, it is extremely important for users to quickly and appropriately communicate the problems they face. However, conventional systems often require users to spend a lot of time and effort to accurately describe their problems. Furthermore, they respond without taking the user's emotional state into consideration, which can lead to a decline in the quality and efficiency of support. In particular, they may be unable to respond appropriately to emotionally charged users, which can increase their dissatisfaction. Therefore, there is a need for a system that allows users to visually express their problems and automatically generates inquiry text that takes their emotional state into account.

[1134] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1135] In this invention, the server includes means for learning the content of inquiries received by the support center and generating illustrations as problem parts to visually express the inquiries, means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem, and means for recognizing the user's emotions and generating an inquiry sentence that includes that information. This enables the user to visually express their problem and automatically generate a specific inquiry sentence that includes their emotional state.

[1136] "Learning the content of inquiries received by support" means collecting inquiry data from past users and analyzing and learning from that data using a machine learning model.

[1137] "Generating illustrations as problem parts" means generating illustrations to visually represent common problems faced by users based on inquiry data.

[1138] "Delivering to the user's device" means sending the generated illustration from the server to the user's device.

[1139] "Providing an interface that allows users to visually select problems" means providing an operation screen that allows users to visually select and express their own problems using displayed illustrations.

[1140] "Automatically generating appropriate inquiry text based on illustration information selected by the user" means using an AI model to automatically write down the corresponding inquiry content based on the information of the illustration selected by the user.

[1141] "Providing an interface that allows users to fine-tune query sentences" means providing an operation screen that allows users to edit and modify the generated query sentences.

[1142] "Emotion recognition" means analyzing a user's emotional state from their facial expressions and voice and acquiring it as data.

[1143] "Send inquiry to customer support" means that the user sends the inquiry that has been finally confirmed and corrected to the support staff.

[1144] "Using an AI model" means using a model that analyzes and generates data using algorithms based on machine learning and natural language processing.

[1145] "Placing by drag and drop" refers to the operation in which the user selects an illustration with the mouse, drags it to the desired location, and drops it.

[1146] This invention is a system that allows users to easily select a problem visually and automatically generates appropriate query sentences. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the quality and efficiency of inquiries can be improved. This system works in conjunction with a server, a user terminal, and an emotion engine that recognizes the user's emotional state.

[1147] Data collection and learning

[1148] The server collects past inquiry data, including text data and metadata (time, category, etc.). By training this data using a machine learning model (e.g., BERT, GPT-3), it is possible to understand the relationship between inquiry content and sentiment. This improves the accuracy of inquiry sentence generation, as described below.

[1149] Illustration generation and saving to database

[1150] The server uses the trained machine learning model to generate illustrations of "problem parts" that visually represent common problems. For example, if there is a lot of data related to network connection problems, it will generate illustrations of Wi-Fi routers and connection errors. The generated illustrations are stored in a database, and each part is also saved with its corresponding tag information.

[1151] Illustration distribution and user interface display

[1152] The device retrieves illustration data from the database and displays it on the user interface. The interface is built using HTML5 and JavaScript, and users can manipulate the illustrations with drag and drop. For example, a user who wants to report a poor internet connection can select an icon of a Wi-Fi router or a connection error.

[1153] emotion recognition

[1154] The device uses an emotion engine to recognize the user's emotions in real time while the user is operating the interface. Specifically, an emotion recognition engine (e.g., OpenFace or SpeechEmotionRecognition) is used to collect emotion data from the user's facial expressions and voice. The emotion data is sent from the device to the server and used to generate query sentences.

[1155] Generate and refine inquiries

[1156] The server receives the illustration information and emotion data selected by the user and automatically generates an appropriate query using an AI model (e.g., GPT-3). For example, if the illustration of "Wi-Fi router" and "connection error" is selected and the emotion data indicates frustration, the server generates a query such as, "Your internet connection is currently very unstable, and there appears to be a problem with your Wi-Fi router." The generated query is then sent to the user's device, where the user can review the displayed message and make adjustments as necessary. For example, the user can add information about the router's specific model number or other detailed information about the situation.

[1157] Submitting a ticket and routing to support

[1158] The user checks the final query sentence after making some fine adjustments and presses the send button, which sends the query sentence to the server.

[1159] The server forwards the received inquiry text to the customer support team and registers the inquiry content in the support system. In particular, if the emotional data exceeds a certain threshold, the inquiry is automatically escalated and set to a high priority.

[1160] Examples of concrete examples and prompts

[1161] Here are some examples of prompts to input to a generative AI model:

[1162] "We are currently experiencing connectivity issues with our Wi-Fi router. The internet connection is very unstable and frequently drops out. Users are extremely frustrated with this issue and are requesting a prompt response from support staff."

[1163] An example of a query that might be generated in response to this prompt is:

[1164] "Currently, my internet connection is very unstable and there seems to be an issue with my Wi-Fi router. I am frequently disconnected and this is very frustrating. I would appreciate a quick response."

[1165] Through the above steps and concrete examples, users can visually express their problems and automatically generate specific inquiry sentences that include their emotional state, which can greatly improve the efficiency and quality of customer support.

[1166] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1167] Step 1: Data collection and learning

[1168] The server collects past inquiry data from a database. As input, it receives text data, inquiry content, and metadata (categories, timestamps, etc.). The collected data is analyzed and trained using a natural language processing model (e.g., BERT, GPT-3). Data processing involves preprocessing the text (tokenization, removal of stop words, etc.). As output, a model that has learned the association between inquiry content patterns and sentiment is obtained.

[1169] Specific behavior:

[1170] The server periodically retrieves new query data from the database, preprocesses the text data using Python, and then trains the data using a machine learning model.

[1171] Step 2: Generate illustrations and save them to the database

[1172] The server uses the trained model to generate illustrations to visually represent the user's problem. As input, it receives the results of the trained model and query data. As data processing, it uses a generative algorithm (e.g., GAN, a deep learning-based image generation model) to create illustrations. As output, it obtains illustrations of the generated problem parts. These illustrations are stored in a database.

[1173] Specific behavior:

[1174] The server uses Python's PIL library to generate illustrations and stores them in MongoDB, with each illustration also containing corresponding tag information.

[1175] Step 3: Delivering illustrations and displaying the user interface

[1176] The device retrieves illustration data from the server and displays it on the user interface. As input, it receives illustration data from the server. For data processing, it retrieves the data using an AJAX request and renders it on the interface using HTML5 and JavaScript. As output, it provides an interface that the user can operate using drag and drop.

[1177] Specific behavior:

[1178] The device uses AJAX requests to pull down illustration data from the server and uses Dragula.js to enable drag-and-drop operation.

[1179] Step 4: Emotion Recognition

[1180] The device uses an emotion engine to recognize the user's emotions in real time. It receives camera footage and audio data as input. It analyzes the data using an emotion engine (e.g., OpenFace, SpeechEmotionRecognition) to generate user emotion data. The analyzed emotion data is obtained as output. The emotion data is sent from the device to the server.

[1181] Specific behavior:

[1182] The device uses WebRTC to capture real-time data from the camera and microphone and passes it to an emotion recognition engine for analysis.

[1183] Step 5: Selecting the problem illustration

[1184] The user looks at the displayed illustrations and selects one to visually represent their problem. The system receives the illustration information selected by the user as input. The system processes the data by selecting and placing the illustrations using drag and drop. The system obtains the selected illustration information as output, and sends it to the server.

[1185] Specific behavior:

[1186] Users select illustrations via drag-and-drop and click a button to submit the information.

[1187] Step 6: Generate a query

[1188] The server automatically generates an appropriate query sentence using an AI model based on the selected illustration information and emotion data. The server receives the illustration information and emotion data as input. The server uses an AI model (e.g., GPT-3) to generate a query sentence as data calculation. The generated query sentence is obtained as output.

[1189] Specific behavior:

[1190] The server uses the Flask framework and Python scripts to invoke the AI ​​model and generate queries.

[1191] Step 7: View and fine-tune your inquiry

[1192] The terminal displays the generated query sentence to the user. As input, it receives the query sentence from the server. As output, it provides an interface that the user can fine-tune. The user can edit and modify the sentence through this interface.

[1193] Specific behavior:

[1194] The terminal displays the generated text in a text area, allowing the user to freely edit it.

[1195] Step 8: Submitting and forwarding inquiries

[1196] The user checks the final adjusted query sentence and clicks the send button. The revised query sentence is received as input. The query sentence is sent to the server as data processing. The query sentence received by the server is forwarded to customer support as output.

[1197] Specific behavior:

[1198] When the user clicks the submit button, the data is sent in a POST request to the server, which then forwards the data to the customer support system using a REST API.

[1199] Step 9: Emotional Escalation

[1200] The server analyzes the user's emotional data and automatically escalates queries when a certain threshold is exceeded. The input is the emotional data. The data calculation evaluates the emotional data that exceeds the threshold. The output is a query with a high priority.

[1201] Specific behavior:

[1202] The server uses a Python script to analyze the sentiment data and, if certain conditions are met, sets an escalation flag and notifies the support team.

[1203] (Application example 2)

[1204] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1205] Conventional customer support systems have had problems with users being unable to accurately and quickly communicate their problems, resulting in poor quality and efficiency of inquiries, especially when the problem is complex or the user is in an emotional state. Furthermore, the inability to respond appropriately to emotional users risks reducing customer satisfaction. To address these issues, the present invention aims to provide a system that makes it easy for users to visually select a problem and automatically generates appropriate inquiry sentences that take the user's emotional state into account.

[1206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1207] In this invention, the server includes means for learning the content of a query and generating illustrations as problem parts for visually expressing the query, means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem, means for recognizing the user's emotional state and acquiring emotional data, and means for generating a query sentence based on the emotional data and illustration information, thereby enabling the user to visually select a problem and generating an appropriate query sentence according to the user's emotional state.

[1208] "Support" means the assistance provided to resolve any problems or questions regarding a service or product.

[1209] "Query" means the details of a question or problem submitted by a User to Support.

[1210] "Problem parts" are elements of a problem that are visually and symbolically represented so that users can select them visually.

[1211] An "illustration" is visual content created to visually represent an issue or situation.

[1212] "User terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[1213] A "visual question selection interface" is a user interface that allows a user to select a question from a visually displayed set of options.

[1214] "Emotional state" refers to the user's psychological and emotional state, which is determined through facial recognition and voice analysis.

[1215] "Emotional Data" refers to data about a user's emotional state, captured in real time.

[1216] "Automatic generation" refers to the process of using AI technology and algorithms to automatically generate query sentences based on pre-set rules and data.

[1217] "Customer support" refers to the support team or department that responds to customer inquiries.

[1218] This invention is a system that automatically generates query sentences that make it easier to visually select problems and take into account emotional states. This system operates by combining a user terminal, a server, and an emotion engine, and can improve the quality and efficiency of user queries.

[1219] System Configuration

[1220] The system consists of the following main components:

[1221] server

[1222] Learning and illustration generation

[1223] The server collects past inquiry data and uses an AI model to illustrate common problems as "problem parts," including information about network connection issues and product defects. The server uses an AI model (e.g., a GPT-3 model) to generate illustrations and stores them in a database.

[1224] Query generation

[1225] The server uses an AI model to automatically generate a query based on the illustration information selected by the user and the emotional data obtained by the emotion engine. For example, if a user selects the illustrations of "connection error" and "Wi-Fi router" and their emotional state is "irritated," the server generates the sentence, "Currently, your internet connection is very unstable, and there seems to be a problem with your Wi-Fi router."

[1226] User Device

[1227] Visual Interface

[1228] The device retrieves the illustrations sent from the server and displays them on a user interface that allows drag-and-drop operation, allowing users to intuitively select problems.

[1229] emotion recognition

[1230] The user device is equipped with an emotion engine and uses a camera and microphone to recognize the user's emotional state in real time, allowing it to collect emotional data while the user is operating the device.

[1231] Example of operation

[1232] Let's take the example of an unstable Wi-Fi connection in a store. A user uses their smartphone to report a problem with the store's Wi-Fi connection. They select illustrations of "Wi-Fi router" and "connection error" on the device, and these illustrations are sent to the server. Using the device's camera and microphone, the emotion engine recognizes that the user is frustrated. The server uses an AI model based on the illustration information and emotion data to generate a query.

[1233] Software and hardware used

[1234] AI model: GPT-3

[1235] Emotion recognition library: FER (Face Emotion Recognition)

[1236] Hardware: Smartphone, server (cloud environment)

[1237] Prompt Sentence Examples

[1238] Below are some examples of specific prompt sentences.

[1239] Select the illustration below and the emotion is as follows:

[1240] Illustration: ['Wi-Fi router', 'Connection error']

[1241] Emotion data: {'Anger': 0.8, 'Happiness': 0.1}

[1242] As described above, this system allows users to visually select a problem and generate an appropriate inquiry sentence based on their emotional state, thereby improving the efficiency and quality of customer support.

[1243] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1244] Step 1:

[1245] The server learns from past inquiry data and uses an AI model to illustrate common problem areas. Specifically, it collects data on network connection issues and product defects and generates illustrations to visually represent them. The generated illustrations are stored in a database.

[1246] Input: Past inquiry data

[1247] Output: Illustration of the problem part

[1248] Step 2:

[1249] The terminal retrieves illustrations delivered from the server and displays them on the user interface. Users visually select their own problems using drag-and-drop operations. This interface is designed to be intuitive.

[1250] Input: Illustration delivered from the server

[1251] Output: User's question selection information

[1252] Step 3:

[1253] The device transmits the illustration information visually selected by the user to the server. At the same time, the device uses a camera and microphone to recognize the user's emotional state in real time and acquire emotional data. This data is then analyzed by the emotion engine.

[1254] Input: User-selected illustration information, real-time emotion data

[1255] Output: Send illustration information and emotion data to the server

[1256] Step 4:

[1257] The server receives illustration information and emotion data sent by the user and uses the AI ​​model to automatically generate a query based on them. For example, if the illustrations of "Connection Error" and "Wi-Fi Router" are selected and the user is feeling irritated, the server will generate the sentence "Currently, your internet connection is very unstable and there seems to be a problem with your Wi-Fi router."

[1258] Input: illustration information, emotion data

[1259] Output: Generated query text

[1260] Step 5:

[1261] The device displays the generated query to the user and provides an interface for fine-tuning, allowing the user to modify the query as needed by adding specific router model numbers and status details.

[1262] Input: Generated query text

[1263] Output: User-adjusted query text

[1264] Step 6:

[1265] After the user has fine-tuned the query text, they send it to the server, and by pressing the send button on their device, the final query text is forwarded to the customer support team.

[1266] Input: Tweaked query text

[1267] Output: Send to customer support team

[1268] Step 7:

[1269] The server has the means to automatically escalate a query if the user's emotional state exceeds a certain threshold: for example, if the user is very annoyed, the query will be given high priority and dealt with quickly.

[1270] Input: User emotion data

[1271] Output: Escalation of tickets with high priority

[1272] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1273] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1274] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1275] [Fourth embodiment]

[1276] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1277] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1278] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1279] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1280] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1281] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1282] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1283] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1284] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1285] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1286] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1287] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1288] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1289] This invention is a system that allows users to easily select a problem visually and automatically generates appropriate query sentences. The program of this system is configured based on the roles of the user, terminal, and server. The specific operation and processing flow are explained below in natural language.

[1290] overview

[1291] This system operates in a mutually linked manner between the server, terminals, and users. The user selects an illustration via the terminal, and the server generates a query based on the selected information. The operation of the entire system is explained below.

[1292] Illustration generation and learning

[1293] server

[1294] The server collects past inquiry data, and the AI ​​model learns from this data. This allows it to illustrate common problems as "problem parts."

[1295] For example, if there are a lot of inquiries about network issues, an illustration of a Wi-Fi router or connection error will be generated.

[1296] Providing an interface

[1297] Terminal

[1298] The terminal provides a user interface that displays illustrations delivered from the server. This interface supports drag and drop, allowing users to operate it intuitively.

[1299] For example, if a user wants to report a poor internet connection issue, they can select and place an icon of a Wi-Fi router or connection error to visually represent the problem.

[1300] Question Selection

[1301] user

[1302] Using the device interface, users select an illustration to visually represent their problem, which is then placed on the device and displayed as an overview of the problem.

[1303] For example, if a user selects the illustrations "Wi-Fi router" and "connection error," this information is sent to the server.

[1304] Generate inquiry content

[1305] server

[1306] The server receives the illustration information selected by the user, analyzes it using an AI model, and generates a natural language query based on the analysis results.

[1307] For example, if the illustrations of "Wi-Fi router" and "connection error" are selected, the server will generate the sentence "Currently, your Internet connection is unstable and there appears to be a problem with your Wi-Fi router."

[1308] Fine-tune and submit your inquiry

[1309] Terminal

[1310] The generated query is sent to the device and displayed to the user, who can then fine-tune it and add details like the model number or specific symptoms if needed.

[1311] For example, a user enters the "Wi-Fi router model number" as an additional detail.

[1312] After making a final confirmation, the user presses the send button to send the inquiry to customer support.

[1313] Transfer to Support

[1314] server

[1315] The server receives the user's final inquiry text and forwards it to the customer support team, who can then respond promptly based on the information received.

[1316] In this way, this system provides an environment where users can visually express their problems and generate and adjust appropriate inquiry sentences based on that information. This system allows users to easily make accurate inquiries even if they are not familiar with technical terminology, and customer support can respond efficiently.

[1317] The processing flow will be explained below.

[1318] Program processing flow

[1319] Step 1:

[1320] server

[1321] The server collects data from past support inquiries, including issues users have reported and detailed metadata associated with them, and stores the collected data in a database, where the AI ​​model learns from it.

[1322] Step 2:

[1323] server

[1324] Using an AI model, the system analyzes collected inquiry data and illustrates common problems as "problem parts." For example, if there is a lot of data related to network connection problems, it generates illustrations of Wi-Fi routers and connection errors. The generated illustrations are stored in a database.

[1325] Step 3:

[1326] Terminal

[1327] When a user accesses the system, the generated illustration is retrieved from the server. The retrieved illustration is then displayed on the terminal's user interface. This interface is designed so that users can operate it by dragging and dropping.

[1328] Step 4:

[1329] user

[1330] Users can view the displayed illustrations and select an illustration to visually represent their problem. The selected illustration is then placed on the device by dragging and dropping, visually representing the user's problem.

[1331] Step 5:

[1332] Terminal

[1333] The illustration information selected by the user and additional text information are sent to the server, including the ID of the illustration selected by the user, its placement information, and any supplementary information entered by the user.

[1334] Step 6:

[1335] server

[1336] The server analyzes the received illustration information and generates an appropriate query using an AI model. For example, if the illustrations "Wi-Fi router" and "connection error" are selected, the server generates the query "Your internet connection is currently unstable and there appears to be a problem with your Wi-Fi router."

[1337] Step 7:

[1338] Terminal

[1339] The generated query text is displayed to the user, who can review it and fine-tune it as needed, for example by adding details about the specific router model or condition.

[1340] Step 8:

[1341] user

[1342] The user makes a final check and presses the send button when the inquiry is complete. This action officially sends the user's inquiry.

[1343] Step 9:

[1344] server

[1345] The server receives the user's final inquiry text and forwards it to the customer support team. The inquiry is registered in the support system so that the support staff can respond promptly.

[1346] This is the specific processing flow of this system. This allows users to make accurate inquiries without knowing the technical details, and enables the support team to respond quickly and accurately.

[1347] Example 1

[1348] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1349] Current customer support systems make it difficult for users to properly explain their problems, especially for those without technical expertise. Furthermore, the vague nature of inquiries makes it difficult for support staff to respond efficiently. This results in lengthy response times and lower user satisfaction.

[1350] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1351] In this invention, the server includes means for learning the content of an inquiry and generating an image for visually expressing the inquiry, means for delivering the generated image to a user terminal and providing an interface that allows the user to visually select a problem, means for automatically generating an appropriate inquiry sentence based on the image information selected by the user, means for providing an interface that allows the user to fine-tune the generated inquiry sentence, and means for transmitting the final inquiry sentence adjusted by the user to a support center. This allows the user to visually select a problem and create an appropriate inquiry sentence even without technical expertise, enabling support staff to respond quickly and efficiently based on clear inquiry content.

[1352] "Inquiry Content" refers to the content of the problem or question provided by the user, and is a general term for information that is requested to be resolved by customer support.

[1353] "Visual representation" means using pictures, diagrams, automatically generated icons, etc. instead of words to make information or issues understandable at a glance.

[1354] "Pictures" are images or icons generated to visually represent the problem, allowing users to easily specify the problem.

[1355] "User terminal" refers to a device used by a user, such as a computer, smartphone, or tablet, which allows the user to access and operate the system.

[1356] An "interface" refers to the screen and operation method used for interaction between the user and the system, providing an environment in which the user can operate intuitively.

[1357] A "generative model" is a type of artificial intelligence that is used to learn large amounts of data to find patterns and regularities and generate new data.

[1358] "Drag and drop" refers to a method of operation in a computer user interface where an object on the screen is moved using a mouse or touch and placed in another location.

[1359] "Support Center" refers to a specialized department or institution that accepts inquiries and problems from users and provides solutions.

[1360] This invention is a system that allows users to easily select a problem visually and automatically generates an appropriate query based on that selection. This system consists of a server, a terminal, and a user component, which operate in conjunction with each other. The details are given below.

[1361] Illustration generation and learning

[1362] server

[1363] The server collects past inquiry data from an inquiry database using database queries.

[1364] The server provides the collected data to a generative AI model (e.g., GPT-3) for learning. As a result of the learning, the AI ​​model illustrates common problems as "problem parts."

[1365] For example, if there are many inquiries about Internet connections, the server will generate illustrations of Wi-Fi routers and connection errors.

[1366] Providing an interface

[1367] Terminal

[1368] The device provides a user interface that displays illustrations delivered from the server, using a web application built with HTML5 and JavaScript.

[1369] The user interface is intuitive with a drag-and-drop feature, allowing users to select and place illustrations on the screen to visually represent the problem.

[1370] For example, a user might drag a "Wi-Fi router" icon across the screen and place a "Connection Error" icon next to it to represent the problem.

[1371] Selecting a problem and generating a query

[1372] user

[1373] The user uses the device interface to select the illustrations they need to visually represent the problem.

[1374] The selected illustration information is displayed in real time on the device, giving the user an overview of the problem, and once the selection is complete, the information is sent to the server.

[1375] server

[1376] The server receives the illustration information selected by the user and inputs it into a generative AI model (e.g., GPT-3) for analysis. As a result of the analysis, a natural language query is generated.

[1377] For example, if a user selects the illustrations of "Wi-Fi router" and "connection error," the server generates the sentence, "Your Internet connection is currently unstable and there appears to be a problem with your Wi-Fi router."

[1378] Fine-tuning the inquiry and final submission

[1379] Terminal

[1380] The generated query text is sent to the device and displayed to the user, who can fine-tune it and enter additional details (e.g., the model number of the Wi-Fi router) in the text boxes if necessary.

[1381] Once the user has finalized the content and pressed the "Send" button, the inquiry will be sent to customer support.

[1382] server

[1383] The server receives the final inquiry text sent from the device and forwards it to the customer support team, who then responds quickly and efficiently based on the information received.

[1384] Prompt Sentence Examples

[1385] As a concrete example, the following prompt sentence is input to the generative AI model:

[1386] If the user selects the illustrations of "Wi-Fi router" and "Connection error",

[1387] Appropriate message: Your internet connection is currently unstable and there appears to be an issue with your Wi-Fi router.

[1388] In this way, the system of the present invention combines visual problem selection with natural language query generation to provide an environment where users can accurately report problems without technical expertise, enabling customer support to respond quickly and efficiently.

[1389] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1390] Step 1:

[1391] Illustration generation

[1392] The server collects past inquiry data from an inquiry database. This data mainly includes inquiry content and category information. These data are extracted through database queries. Next, the server inputs this data into a generative AI model (e.g., GPT-3) and trains the model. Through training, the model generates pictures to visually represent common problems. The generated pictures are output in the form of icons, such as "Wi-Fi router" or "connection error."

[1393] Step 2:

[1394] Illustration distribution and display

[1395] The server delivers the generated images to the user's device. The user's device receives the images delivered from the server and displays them on a web application built with HTML5 and JavaScript. This interface has a drag-and-drop function, allowing users to easily operate it visually. This allows users to select how to express the problem in a picture. The input to the user's device is the image data from the server, and the output is an interface that the user can operate.

[1396] Step 3:

[1397] Question Selection

[1398] The user selects a picture on the device interface to visually represent their problem. The selected picture is displayed on the screen in real time and positioned by the user. For example, the user selects a "Wi-Fi router" icon and positions it on the screen along with a "Connection Error" icon. This action generates a selection, which is sent to the server. The user's input is the selected picture, and the output is the selection sent to the server.

[1399] Step 4:

[1400] Query generation

[1401] The server receives the picture information selected by the user and inputs that information into the generative AI model. The model analyzes this and generates a natural language query. For example, if the pictures of "Wi-Fi router" and "connection error" are selected, the model will generate the sentence "Currently, your internet connection is unstable and there appears to be a problem with your Wi-Fi router." The server's input is the picture information from the user, and its output is the generated query.

[1402] Step 5:

[1403] Fine-tuning of inquiry text

[1404] The device displays the generated query text to the user, who can refer to it and make any necessary adjustments. Specifically, the user can enter additional details, such as "Wi-Fi router model number," in the text box to complete the text. The user's input is the adjusted query text, and the output is the final query text.

[1405] Step 6:

[1406] Sending an inquiry

[1407] After the user has finished fine-tuning the query text, they press the "Send" button to send it to customer support. At this point, the final version of the query text is sent from the user's device to the server. The server then forwards the received final version of the query text to the support center. The server's input is the user's final query text, and its output is the query text sent to the support center.

[1408] (Application example 1)

[1409] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1410] When handling customer support inquiries at physical stores, it is difficult for customers to accurately communicate their problems, and it is also difficult for support staff to quickly understand and address the issues. For this reason, there is a need for a system that allows customers to intuitively report problems and enables support staff to respond quickly and accurately.

[1411] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1412] In this invention, the server includes means for learning the content of inquiries received by the support center and generating actions as problem parts for visually expressing the content, means for distributing the generated actions to a user device and providing an interface for the user to visually select a problem, means for automatically generating an appropriate inquiry sentence based on the action information selected by the user, means for providing an interface for the user to fine-tune the generated inquiry sentence, means for sending the final inquiry sentence adjusted by the user to a support staff member, means for using an artificial intelligence (AI) model to generate the inquiry sentence, means for sending the inquiry sentence generated by the artificial intelligence (AI) model to an appropriate staff member, and means for managing the transmission history. This allows customers to intuitively report problems and enables support staff to respond efficiently and quickly.

[1413] "Learning support inquiries" is the process of collecting inquiry data from users and training a machine learning model based on that data.

[1414] "Generating actions as parts of a problem to visually express it" means generating specific illustrations and action parts related to the content of the inquiry and displaying them visually.

[1415] "User Device" refers to an electronic device used by a user, such as a smartphone, tablet, or PC.

[1416] "Providing an interface" means providing an operation screen and operation means that users can operate intuitively.

[1417] "Action information" refers to information about specific illustrations and actions selected by the user on the interface.

[1418] "Automatically generating an appropriate inquiry sentence" means generating an inquiry sentence in natural language based on the selected action information.

[1419] A "fine-tunable interface" means providing an operation screen that allows the user to check, edit, and modify the generated query text.

[1420] "Support personnel" refers to staff or personnel in charge of customer support.

[1421] "Using an artificial intelligence (AI) model to generate a query sentence" means using AI technology to analyze the query content and generate a sentence in natural language.

[1422] "Managing transmission history" means saving and managing the history of the contents of inquiries sent.

[1423] A "prompt sentence" is the input text that an artificial intelligence (AI) model uses to generate a query sentence.

[1424] The present invention relates to a system that enables users to intuitively report problems at physical stores and allows support staff to quickly respond. Specific embodiments for realizing this system will be described below.

[1425] First, the server learns the content of inquiries received by the support center and generates behavior as problem parts to visually represent them. The hardware used for this is a high-performance server computer, and the software used includes open-source machine learning libraries and natural language processing libraries.

[1426] The server then delivers the generated actions to the user device, which is typically an electronic device such as a smartphone or tablet. The interface is designed to be intuitive and allows drag and drop, allowing users to visually select problems.

[1427] When a user selects an action using the interface, the action information is sent from the user device to the server. The server then automatically generates an appropriate query sentence based on the action information selected by the user. This automatic generation uses a generative AI model using the OpenAI API. The following is an example of a specific prompt sentence.

[1428] plaintext

[1429] User-selected illustrations: cashier trouble, product placement

[1430] Additional details: The cashier screen is frozen and the product cannot be found

[1431] Generate a query.

[1432] The generated query text is displayed on the user's device, allowing the user to fine-tune it and add any necessary details. For example, the user might add the cash register model number or the specific product name.

[1433] The final query text, fine-tuned by the user, is sent from the user's device to the support staff. The support staff can then respond quickly and appropriately based on the text sent. Furthermore, by managing the transmission history, past inquiry information can be referenced, which can be useful for future support responses.

[1434] Through this process, users can intuitively report problems and support staff can respond efficiently.

[1435] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1436] Step 1:

[1437] Learning inquiry content and generating problem parts

[1438] The server collects user inquiry data and uses a machine learning algorithm to learn from it. This generates action icons and illustrations to visually represent common problems. For example, if many inquiries are about "Wi-Fi router problems," the server generates an illustration of a Wi-Fi router.

[1439] Input: Past inquiry data

[1440] Data processing: Data analysis and icon generation using machine learning algorithms

[1441] Output: Action icons and illustrations

[1442] Step 2:

[1443] Delivery of action icons and illustrations

[1444] The server then delivers the generated action icons and illustrations to the user's device, providing a visual interface for the user to select problems. For example, a Wi-Fi router icon is displayed on the user's smartphone.

[1445] Input: Action icons and illustrations

[1446] Data processing: None (transfer only)

[1447] Output: Action icons and illustrations displayed on the user's device

[1448] Step 3:

[1449] Selecting and sending operational information

[1450] The user selects an action icon or illustration on the interface that corresponds to their problem, and the selection is sent from the user device to the server.

[1451] Input: User-selected action icons and illustrations

[1452] Data processing: collecting and formatting selected information

[1453] Output: Selection information sent to the server

[1454] Step 4:

[1455] Automatic generation of inquiry text

[1456] The server uses the received behavior information to call the generative AI model and generate an appropriate query sentence. At this time, the query sentence is automatically generated using a prompt sentence. For example, the prompt might be "Illustration selected by the user: Wi-Fi router, connection error."

[1457] Input: User-selected action information, prompt text

[1458] Data Processing: Natural Language Generation Using Generative AI Models

[1459] Output: Generated query text

[1460] Step 5:

[1461] Fine-tuning of inquiry text

[1462] The device displays the generated query text to the user and provides a screen where the user can review, edit, and fine-tune the content. For example, the user can enter the "Wi-Fi router model number" or "specific connection error situation."

[1463] Input: Generated query text

[1464] Data processing: User editing and fine-tuning

[1465] Output: The final query text that has been reviewed and refined by the user

[1466] Step 6:

[1467] Sending the final inquiry

[1468] The user checks the finely adjusted inquiry text and presses the send button to send it to the support staff. This transmission is performed from the user device via the server.

[1469] Input: The final query text that the user has reviewed and refined

[1470] Data processing: None (transfer only)

[1471] Output: Final query sent to support

[1472] Step 7:

[1473] Managing sending history

[1474] The server stores the final query text sent as a history and makes it available for future inquiries, allowing past history to be referenced.

[1475] Input: Final query text

[1476] Data processing: Save to database

[1477] Output: Saved query history

[1478] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1479] This system allows users to easily select a problem visually and automatically generates appropriate query sentences. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the quality and efficiency of queries can be improved.

[1480] overview

[1481] This system works in conjunction with a server, a terminal, and an emotion engine that recognizes the user's emotional state. The user selects an illustration via the terminal, and the server generates a query sentence based on the selection information and the user's emotional state. The specific operation and processing flow are explained below in natural language.

[1482] Illustration generation and learning

[1483] server

[1484] The server collects data from past support inquiries, including the content and metadata of users' past inquiries, and the collected data is used to train an AI model.

[1485] The AI ​​model uses this data to illustrate common problems as "problem parts." For example, if there is a lot of data about network connection problems, it will generate illustrations of Wi-Fi routers and connection errors. The generated illustrations are stored in a database.

[1486] Providing an interface

[1487] Terminal

[1488] The device retrieves the illustrations sent from the server and displays them on a user interface that users can manipulate using drag and drop.

[1489] If a user reports a poor internet connection issue, you can select and place a Wi-Fi router or connection error icon to visually represent the problem.

[1490] emotion recognition

[1491] Terminal

[1492] While the user is interacting with the interface, the emotion engine recognizes the user's emotions through facial recognition and voice analysis, collecting emotional data. For example, if the user is frustrated, their emotion is analyzed in real time.

[1493] Question Selection

[1494] user

[1495] Users can view the displayed illustrations and select an illustration to visually represent their problem. The selected illustration can then be placed on the device by dragging and dropping.

[1496] For example, if a user selects the illustrations "Wi-Fi router" and "connection error," that information is sent to the server.

[1497] Generate inquiry content

[1498] server

[1499] The server receives the illustration information selected by the user and the emotion data obtained by the emotion engine, and uses the AI ​​model to generate an appropriate query sentence.

[1500] For example, if the illustrations of "Wi-Fi router" and "connection error" are selected and the user is feeling irritated, the server will generate the sentence "Currently, your Internet connection is very unstable and there seems to be a problem with your Wi-Fi router."

[1501] Fine-tune and submit your inquiry

[1502] Terminal

[1503] The generated query text is displayed to the user, who can review it and fine-tune it as needed, for example by adding details about the specific router model or condition.

[1504] After the user makes some fine adjustments, they make a final check and press the send button, which officially sends the inquiry text.

[1505] Transfer to Support

[1506] server

[1507] The server receives the user's final inquiry text and forwards it to the customer support team. The inquiry is registered in the support system so that the support staff can respond promptly.

[1508] Emotional Escalation

[1509] server

[1510] The server also has the means to automatically escalate a query if the user's emotion exceeds a certain threshold. For example, if the user is very annoyed, the query will be given high priority and dealt with quickly.

[1511] In this way, this system allows users to visually express their problems and generates appropriate inquiry sentences based on that information and their emotional state. Furthermore, by combining it with an emotion engine, it becomes possible to respond according to the user's emotions, improving the efficiency and quality of customer support.

[1512] The processing flow will be explained below.

[1513] MODE FOR CARRYING OUT THE INVENTION

[1514] The system of the present invention allows users to visually select a problem and provides appropriate and prompt support by having the AI ​​generate an inquiry that reflects the user's feelings. The specific process flow is shown below.

[1515] Step 1:

[1516] server

[1517] The server collects past inquiry data and trains the AI ​​model. This data includes issues reported by users in the past, their details, and the results of their responses. Repeated data collection and training improves the accuracy of the model.

[1518] Step 2:

[1519] server

[1520] Based on the learned data, the AI ​​model illustrates common problems as "problem parts." For example, for the common problem of "unstable network," it generates illustrations of "Wi-Fi router" and "connection error."

[1521] Step 3:

[1522] Terminal

[1523] When a user starts the system, the terminal retrieves the generated illustrations from the server and displays them on the user interface, which has a drag-and-drop function for intuitive operation.

[1524] Step 4:

[1525] user

[1526] Users can select an illustration to visually represent their problem by viewing the displayed illustrations. For example, a user can select an illustration of a "Wi-Fi router" and a "connection error" and place them on the screen.

[1527] Step 5:

[1528] Terminal

[1529] The illustration information selected by the user and additional explanatory text are sent to the server. This information includes the ID of the selected illustration, its location, and any additional explanation entered by the user.

[1530] Step 6:

[1531] Terminal

[1532] While the user is selecting an illustration, the emotion engine recognizes the user's emotions in real time through facial recognition and voice analysis, and the emotion data is also sent to the server in parallel.

[1533] Step 7:

[1534] server

[1535] The server analyzes the received illustration information and emotion data and generates an appropriate query using an AI model. For example, if the illustrations of "Wi-Fi router" and "connection error" are selected and the user is feeling irritated, the server will generate the following sentence: "Currently, your internet connection is very unstable and there seems to be a problem with your Wi-Fi router."

[1536] Step 8:

[1537] Terminal

[1538] The generated query text is displayed to the user, who can review it and fine-tune it as needed, for example adding details about a specific router model or condition.

[1539] Step 9:

[1540] user

[1541] The user makes a final check and presses the send button when the inquiry is complete. This action officially sends the user's inquiry.

[1542] Step 10:

[1543] server

[1544] The server receives the user's final inquiry text and forwards it to the customer support team. The inquiry is registered in the support system so that the support staff can respond promptly.

[1545] Step 11:

[1546] server

[1547] If the user's emotion exceeds a certain threshold, the server escalates the query. For example, if the user is very annoyed, the query is set to high priority and requires immediate attention.

[1548] This system makes it easier for users to visually and emotionally express their problems, allowing customer support to understand the user's situation and emotions and respond appropriately.

[1549] Example 2

[1550] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1551] In modern customer support systems, it is extremely important for users to quickly and appropriately communicate the problems they face. However, conventional systems often require users to spend a lot of time and effort to accurately describe their problems. Furthermore, they respond without taking the user's emotional state into consideration, which can lead to a decline in the quality and efficiency of support. In particular, they may be unable to respond appropriately to emotionally charged users, which can increase their dissatisfaction. Therefore, there is a need for a system that allows users to visually express their problems and automatically generates inquiry text that takes their emotional state into account.

[1552] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1553] In this invention, the server includes means for learning the content of inquiries received by the support center and generating illustrations as problem parts to visually express the inquiries, means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem, and means for recognizing the user's emotions and generating an inquiry sentence that includes that information. This enables the user to visually express their problem and automatically generate a specific inquiry sentence that includes their emotional state.

[1554] "Learning the content of inquiries received by support" means collecting inquiry data from past users and analyzing and learning from that data using a machine learning model.

[1555] "Generating illustrations as problem parts" means generating illustrations to visually represent common problems faced by users based on inquiry data.

[1556] "Delivering to the user's device" means sending the generated illustration from the server to the user's device.

[1557] "Providing an interface that allows users to visually select problems" means providing an operation screen that allows users to visually select and express their own problems using displayed illustrations.

[1558] "Automatically generating appropriate inquiry text based on illustration information selected by the user" means using an AI model to automatically write down the corresponding inquiry content based on the information of the illustration selected by the user.

[1559] "Providing an interface that allows users to fine-tune query sentences" means providing an operation screen that allows users to edit and modify the generated query sentences.

[1560] "Emotion recognition" means analyzing a user's emotional state from their facial expressions and voice and acquiring it as data.

[1561] "Send inquiry to customer support" means that the user sends the inquiry that has been finally confirmed and corrected to the support staff.

[1562] "Using an AI model" means using a model that analyzes and generates data using algorithms based on machine learning and natural language processing.

[1563] "Placing by drag and drop" refers to the operation in which the user selects an illustration with the mouse, drags it to the desired location, and drops it.

[1564] This invention is a system that allows users to easily select a problem visually and automatically generates appropriate query sentences. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the quality and efficiency of inquiries can be improved. This system works in conjunction with a server, a user terminal, and an emotion engine that recognizes the user's emotional state.

[1565] Data collection and learning

[1566] The server collects past inquiry data, including text data and metadata (time, category, etc.). By training this data using a machine learning model (e.g., BERT, GPT-3), it is possible to understand the relationship between inquiry content and sentiment. This improves the accuracy of inquiry sentence generation, as described below.

[1567] Illustration generation and saving to database

[1568] The server uses the trained machine learning model to generate illustrations of "problem parts" that visually represent common problems. For example, if there is a lot of data related to network connection problems, it will generate illustrations of Wi-Fi routers and connection errors. The generated illustrations are stored in a database, and each part is also saved with its corresponding tag information.

[1569] Illustration distribution and user interface display

[1570] The device retrieves illustration data from the database and displays it on the user interface. The interface is built using HTML5 and JavaScript, and users can manipulate the illustrations with drag and drop. For example, a user who wants to report a poor internet connection can select an icon of a Wi-Fi router or a connection error.

[1571] emotion recognition

[1572] The device uses an emotion engine to recognize the user's emotions in real time while the user is operating the interface. Specifically, an emotion recognition engine (e.g., OpenFace or SpeechEmotionRecognition) is used to collect emotion data from the user's facial expressions and voice. The emotion data is sent from the device to the server and used to generate query sentences.

[1573] Generate and refine inquiries

[1574] The server receives the illustration information and emotion data selected by the user and automatically generates an appropriate query using an AI model (e.g., GPT-3). For example, if the illustration of "Wi-Fi router" and "connection error" is selected and the emotion data indicates frustration, the server generates a query such as, "Your internet connection is currently very unstable, and there appears to be a problem with your Wi-Fi router." The generated query is then sent to the user's device, where the user can review the displayed message and make adjustments as necessary. For example, the user can add information about the router's specific model number or other detailed information about the situation.

[1575] Submitting a ticket and routing to support

[1576] The user checks the final query sentence after making some fine adjustments and presses the send button, which sends the query sentence to the server.

[1577] The server forwards the received inquiry text to the customer support team and registers the inquiry content in the support system. In particular, if the emotional data exceeds a certain threshold, the inquiry is automatically escalated and set to a high priority.

[1578] Examples of concrete examples and prompts

[1579] Here are some examples of prompts to input to a generative AI model:

[1580] "We are currently experiencing connectivity issues with our Wi-Fi router. The internet connection is very unstable and frequently drops out. Users are extremely frustrated with this issue and are requesting a prompt response from support staff."

[1581] An example of a query that might be generated in response to this prompt is:

[1582] "Currently, my internet connection is very unstable and there seems to be an issue with my Wi-Fi router. I am frequently disconnected and this is very frustrating. I would appreciate a quick response."

[1583] Through the above steps and concrete examples, users can visually express their problems and automatically generate specific inquiry sentences that include their emotional state, which can greatly improve the efficiency and quality of customer support.

[1584] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1585] Step 1: Data collection and learning

[1586] The server collects past inquiry data from a database. As input, it receives text data, inquiry content, and metadata (categories, timestamps, etc.). The collected data is analyzed and trained using a natural language processing model (e.g., BERT, GPT-3). Data processing involves preprocessing the text (tokenization, removal of stop words, etc.). As output, a model that has learned the association between inquiry content patterns and sentiment is obtained.

[1587] Specific behavior:

[1588] The server periodically retrieves new query data from the database, preprocesses the text data using Python, and then trains the data using a machine learning model.

[1589] Step 2: Generate illustrations and save them to the database

[1590] The server uses the trained model to generate illustrations to visually represent the user's problem. As input, it receives the results of the trained model and query data. As data processing, it uses a generative algorithm (e.g., GAN, a deep learning-based image generation model) to create illustrations. As output, it obtains illustrations of the generated problem parts. These illustrations are stored in a database.

[1591] Specific behavior:

[1592] The server uses Python's PIL library to generate illustrations and stores them in MongoDB, with each illustration also containing corresponding tag information.

[1593] Step 3: Delivering illustrations and displaying the user interface

[1594] The device retrieves illustration data from the server and displays it on the user interface. As input, it receives illustration data from the server. For data processing, it retrieves the data using an AJAX request and renders it on the interface using HTML5 and JavaScript. As output, it provides an interface that the user can operate using drag and drop.

[1595] Specific behavior:

[1596] The device uses AJAX requests to pull down illustration data from the server and uses Dragula.js to enable drag-and-drop operation.

[1597] Step 4: Emotion Recognition

[1598] The device uses an emotion engine to recognize the user's emotions in real time. It receives camera footage and audio data as input. It analyzes the data using an emotion engine (e.g., OpenFace, SpeechEmotionRecognition) to generate user emotion data. The analyzed emotion data is obtained as output. The emotion data is sent from the device to the server.

[1599] Specific behavior:

[1600] The device uses WebRTC to capture real-time data from the camera and microphone and passes it to an emotion recognition engine for analysis.

[1601] Step 5: Selecting the problem illustration

[1602] The user looks at the displayed illustrations and selects one to visually represent their problem. The system receives the illustration information selected by the user as input. The system processes the data by selecting and placing the illustrations using drag and drop. The system obtains the selected illustration information as output, and sends it to the server.

[1603] Specific behavior:

[1604] Users select illustrations via drag-and-drop and click a button to submit the information.

[1605] Step 6: Generate a query

[1606] The server automatically generates an appropriate query sentence using an AI model based on the selected illustration information and emotion data. The server receives the illustration information and emotion data as input. The server uses an AI model (e.g., GPT-3) to generate a query sentence as data calculation. The generated query sentence is obtained as output.

[1607] Specific behavior:

[1608] The server uses the Flask framework and Python scripts to invoke the AI ​​model and generate queries.

[1609] Step 7: View and fine-tune your inquiry

[1610] The terminal displays the generated query sentence to the user. As input, it receives the query sentence from the server. As output, it provides an interface that the user can fine-tune. The user can edit and modify the sentence through this interface.

[1611] Specific behavior:

[1612] The terminal displays the generated text in a text area, allowing the user to freely edit it.

[1613] Step 8: Submitting and forwarding inquiries

[1614] The user checks the final adjusted query sentence and clicks the send button. The revised query sentence is received as input. The query sentence is sent to the server as data processing. The query sentence received by the server is forwarded to customer support as output.

[1615] Specific behavior:

[1616] When the user clicks the submit button, the data is sent in a POST request to the server, which then forwards the data to the customer support system using a REST API.

[1617] Step 9: Emotional Escalation

[1618] The server analyzes the user's emotional data and automatically escalates queries when a certain threshold is exceeded. The input is the emotional data. The data calculation evaluates the emotional data that exceeds the threshold. The output is a query with a high priority.

[1619] Specific behavior:

[1620] The server uses a Python script to analyze the sentiment data and, if certain conditions are met, sets an escalation flag and notifies the support team.

[1621] (Application example 2)

[1622] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1623] Conventional customer support systems have had problems with users being unable to accurately and quickly communicate their problems, resulting in poor quality and efficiency of inquiries, especially when the problem is complex or the user is in an emotional state. Furthermore, the inability to respond appropriately to emotional users risks reducing customer satisfaction. To address these issues, the present invention aims to provide a system that makes it easy for users to visually select a problem and automatically generates appropriate inquiry sentences that take the user's emotional state into account.

[1624] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1625] In this invention, the server includes means for learning the content of a query and generating illustrations as problem parts for visually expressing the query, means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem, means for recognizing the user's emotional state and acquiring emotional data, and means for generating a query sentence based on the emotional data and illustration information, thereby enabling the user to visually select a problem and generating an appropriate query sentence according to the user's emotional state.

[1626] "Support" means the assistance provided to resolve any problems or questions regarding a service or product.

[1627] "Query" means the details of a question or problem submitted by a User to Support.

[1628] "Problem parts" are elements of a problem that are visually and symbolically represented so that users can select them visually.

[1629] An "illustration" is visual content created to visually represent an issue or situation.

[1630] "User terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[1631] A "visual question selection interface" is a user interface that allows a user to select a question from a visually displayed set of options.

[1632] "Emotional state" refers to the user's psychological and emotional state, which is determined through facial recognition and voice analysis.

[1633] "Emotional Data" refers to data about a user's emotional state, captured in real time.

[1634] "Automatic generation" refers to the process of using AI technology and algorithms to automatically generate query sentences based on pre-set rules and data.

[1635] "Customer support" refers to the support team or department that responds to customer inquiries.

[1636] This invention is a system that automatically generates query sentences that make it easier to visually select problems and take into account emotional states. This system operates by combining a user terminal, a server, and an emotion engine, and can improve the quality and efficiency of user queries.

[1637] System Configuration

[1638] The system consists of the following main components:

[1639] server

[1640] Learning and illustration generation

[1641] The server collects past inquiry data and uses an AI model to illustrate common problems as "problem parts," including information about network connection issues and product defects. The server uses an AI model (e.g., a GPT-3 model) to generate illustrations and stores them in a database.

[1642] Query generation

[1643] The server uses an AI model to automatically generate a query based on the illustration information selected by the user and the emotional data obtained by the emotion engine. For example, if a user selects the illustrations of "connection error" and "Wi-Fi router" and their emotional state is "irritated," the server generates the sentence, "Currently, your internet connection is very unstable, and there seems to be a problem with your Wi-Fi router."

[1644] User Device

[1645] Visual Interface

[1646] The device retrieves the illustrations sent from the server and displays them on a user interface that allows drag-and-drop operation, allowing users to intuitively select problems.

[1647] emotion recognition

[1648] The user device is equipped with an emotion engine and uses a camera and microphone to recognize the user's emotional state in real time, allowing it to collect emotional data while the user is operating the device.

[1649] Example of operation

[1650] Let's take the example of an unstable Wi-Fi connection in a store. A user uses their smartphone to report a problem with the store's Wi-Fi connection. They select illustrations of "Wi-Fi router" and "connection error" on the device, and these illustrations are sent to the server. Using the device's camera and microphone, the emotion engine recognizes that the user is frustrated. The server uses an AI model based on the illustration information and emotion data to generate a query.

[1651] Software and hardware used

[1652] AI model: GPT-3

[1653] Emotion recognition library: FER (Face Emotion Recognition)

[1654] Hardware: Smartphone, server (cloud environment)

[1655] Prompt Sentence Examples

[1656] Below are some examples of specific prompt sentences.

[1657] Select the illustration below and the emotion is as follows:

[1658] Illustration: ['Wi-Fi router', 'Connection error']

[1659] Emotion data: {'Anger': 0.8, 'Happiness': 0.1}

[1660] As described above, this system allows users to visually select a problem and generate an appropriate inquiry sentence based on their emotional state, thereby improving the efficiency and quality of customer support.

[1661] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1662] Step 1:

[1663] The server learns from past inquiry data and uses an AI model to illustrate common problem areas. Specifically, it collects data on network connection issues and product defects and generates illustrations to visually represent them. The generated illustrations are stored in a database.

[1664] Input: Past inquiry data

[1665] Output: Illustration of the problem part

[1666] Step 2:

[1667] The terminal retrieves illustrations delivered from the server and displays them on the user interface. Users visually select their own problems using drag-and-drop operations. This interface is designed to be intuitive.

[1668] Input: Illustration delivered from the server

[1669] Output: User's question selection information

[1670] Step 3:

[1671] The device transmits the illustration information visually selected by the user to the server. At the same time, the device uses a camera and microphone to recognize the user's emotional state in real time and acquire emotional data. This data is then analyzed by the emotion engine.

[1672] Input: User-selected illustration information, real-time emotion data

[1673] Output: Send illustration information and emotion data to the server

[1674] Step 4:

[1675] The server receives illustration information and emotion data sent by the user and uses the AI ​​model to automatically generate a query based on them. For example, if the illustrations of "Connection Error" and "Wi-Fi Router" are selected and the user is feeling irritated, the server will generate the sentence "Currently, your internet connection is very unstable and there seems to be a problem with your Wi-Fi router."

[1676] Input: illustration information, emotion data

[1677] Output: Generated query text

[1678] Step 5:

[1679] The device displays the generated query to the user and provides an interface for fine-tuning, allowing the user to modify the query as needed by adding specific router model numbers and status details.

[1680] Input: Generated query text

[1681] Output: User-adjusted query text

[1682] Step 6:

[1683] After the user has fine-tuned the query text, they send it to the server, and by pressing the send button on their device, the final query text is forwarded to the customer support team.

[1684] Input: Tweaked query text

[1685] Output: Send to customer support team

[1686] Step 7:

[1687] The server has the means to automatically escalate a query if the user's emotional state exceeds a certain threshold: for example, if the user is very annoyed, the query will be given high priority and dealt with quickly.

[1688] Input: User emotion data

[1689] Output: Escalation of tickets with high priority

[1690] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1691] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1692] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1693] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1694] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1695] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1696] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1697] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1698] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1699] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1700] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1701] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1702] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1703] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1704] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1705] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1706] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1707] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1708] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1709] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1710] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1711] The following is further disclosed regarding the above embodiment.

[1712] (Claim 1)

[1713] A method to learn the content of inquiries received by support and generate illustrations as problem parts to visually represent these,

[1714] A means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem;

[1715] A means for automatically generating an appropriate inquiry sentence based on illustration information selected by the user;

[1716] a means for providing an interface that allows a user to fine-tune the generated query text;

[1717] A means for the user to submit the final inquiry text to customer support;

[1718] A system including:

[1719] (Claim 2)

[1720] A means for analyzing the illustration information selected by the user and the text information added by the user and using the AI ​​model to generate a query sentence;

[1721] The system of claim 1 further comprising:

[1722] (Claim 3)

[1723] A means to provide an interface that allows users to visually represent their problems by dragging and dropping illustrations;

[1724] The system of claim 1 further comprising:

[1725] "Example 1"

[1726] (Claim 1)

[1727] means for learning query content and generating pictures to visually represent the query content;

[1728] A means for delivering the generated picture to a user terminal and providing an interface that allows the user to visually select a problem;

[1729] A means for automatically generating an appropriate inquiry sentence based on the picture information selected by the user;

[1730] a means for providing an interface that allows a user to fine-tune the generated query text;

[1731] A means for sending the final inquiry text adjusted by the user to the support center;

[1732] A system including:

[1733] (Claim 2)

[1734] means for analyzing the user-selected pictorial information and the user-added text information and using the generative model to generate a query sentence;

[1735] The system of claim 1 further comprising:

[1736] (Claim 3)

[1737] A means to provide an interface that allows users to visually represent their problem by dragging and dropping pictures;

[1738] The system of claim 1 further comprising:

[1739] "Application Example 1"

[1740] (Claim 1)

[1741] A means to learn the content of inquiries received by support and generate behavior as problem parts to visually express these, and

[1742] means for delivering the generated actions to a user device and providing an interface through which the user can visually select a problem;

[1743] A means for automatically generating an appropriate query sentence based on the operation information selected by the user;

[1744] a means for providing an interface that allows a user to fine-tune the generated query text;

[1745] A means for the user to send the final, adjusted inquiry text to the support staff;

[1746] a means for using an artificial intelligence (AI) model to generate a query;

[1747] A means for sending the query text generated by the artificial intelligence (AI) model to the appropriate person; and

[1748] In addition, a means for managing transmission history,

[1749] A system including:

[1750] (Claim 2)

[1751] means for analyzing the user-selected motion information and the user-added text information and using an artificial intelligence (AI) model to generate a query sentence;

[1752] A means for the generative AI model to use the prompt sentence in generating the query sentence;

[1753] The system of claim 1 further comprising:

[1754] (Claim 3)

[1755] A means to provide an interface that allows users to visually represent their problems by dragging and dropping illustrations;

[1756] Furthermore, the generated query text is displayed in real time, allowing users to check and edit it.

[1757] 10. The system of claim 1, comprising:

[1758] "Example 2: Combining Emotion Engines"

[1759] (Claim 1)

[1760] A method to learn the content of inquiries received by support and generate illustrations as problem parts to visually represent these,

[1761] A means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem;

[1762] A means for automatically generating an appropriate inquiry sentence based on illustration information selected by the user;

[1763] a means for providing an interface that allows a user to fine-tune the generated query text;

[1764] A means for recognizing a user's emotion and generating a query sentence including the emotion;

[1765] A means for the user to submit the final inquiry text to customer support;

[1766] A system including:

[1767] (Claim 2)

[1768] A means for analyzing the illustration information selected by the user and the user's emotion data and using the AI ​​model to generate a query sentence;

[1769] The system of claim 1 further comprising:

[1770] (Claim 3)

[1771] A means to provide an interface that allows users to visually represent their problems by dragging and dropping illustrations;

[1772] The system of claim 1 further comprising:

[1773] "Application example 2 when combining emotion engines"

[1774] (Claim 1)

[1775] A means to learn the content of inquiries and generate illustrations as problem parts to visually express them,

[1776] A means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem;

[1777] A means for automatically generating an appropriate inquiry sentence based on illustration information selected by the user;

[1778] means for recognizing a user's emotional state and obtaining emotional data;

[1779] A means for generating a query sentence based on emotion data and illustration information;

[1780] a means for providing an interface that allows a user to fine-tune the generated query text;

[1781] A means for the user to submit the final, adjusted inquiry to the support team;

[1782] A system including:

[1783] (Claim 2)

[1784] A means for analyzing the illustration information selected by the user and the text information added by the user and using the AI ​​model to generate a query sentence;

[1785] A means for formatting the selected illustration information and emotion data and inputting it into the generative AI model;

[1786] The system of claim 1 further comprising:

[1787] (Claim 3)

[1788] A means to provide an interface that allows users to visually represent their problems by dragging and dropping illustrations;

[1789] A means including an emotion engine that analyzes the user's emotions through voice and image recognition;

[1790] The system of claim 1 further comprising: [Explanation of symbols]

[1791] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A method to learn the content of inquiries received by support and generate illustrations as problem parts to visually represent these, A means for delivering the generated illustrations to a user terminal and providing an interface that allows the user to visually select a problem; A means for automatically generating an appropriate inquiry sentence based on illustration information selected by the user; a means for providing an interface that allows a user to fine-tune the generated query text; A means for the user to submit the final inquiry text to customer support; A system including:

2. A means for analyzing the illustration information selected by the user and the text information added by the user and using the AI ​​model to generate a query sentence; The system of claim 1 further comprising:

3. A means to provide an interface that allows users to visually represent their problems by dragging and dropping illustrations; The system of claim 1 further comprising:

Images