system

Abstract

We provide the system. [Solution] A means of learning operating procedures using generative artificial intelligence, A means of automatically generating a user manual, A means of automatically generating and providing answers to user inquiries, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] --- [[ID=۳۷]]

[0005] In corporate business, since multiple services are handled, there is a problem that it is difficult to be proficient in all services. Therefore, a large amount of labor is required to respond to inquiries from customers. In order to solve this problem, means for efficiently creating usage manuals for various services and automating inquiry responses are required.

Means for Solving the Problems

[0006] This invention relates to a system that uses generative artificial intelligence to learn operating procedures, automatically generates user manuals, and provides automatic answers to user inquiries. Specifically, it collects user operation logs in real time and analyzes and learns them using generative artificial intelligence to understand operating procedures. Subsequently, based on the learned content, it automatically generates a user manual including text explanations of operations, screenshots, and videos. Furthermore, the generative artificial intelligence also analyzes user inquiries and automatically generates and provides appropriate answers. This significantly reduces the workload involved in handling service-related inquiries.

[0007] ---

[0008] ---

[0009] "Generative artificial intelligence" refers to artificial intelligence that has the ability to automatically generate necessary information by analyzing user operation logs and inquiries.

[0010] An "operating procedure" is a sequence of specific steps and means for performing a particular task or operation.

[0011] A "user manual" is a document containing instructions, guidelines, and explanations to help users properly utilize a particular service or system.

[0012] An "operation log" is a record of operations and inputs performed by a user, containing data such as timestamps, operation details, and information about the target element.

[0013] "Analysis" is the process of thoroughly examining data and information and understanding or interpreting its content.

[0014] "Learning" is the process by which generative artificial intelligence acquires new knowledge and skills based on the data it is provided, and becomes able to apply them.

[0015] A "screenshot" is a visual capture image of the user's screen at a specific moment.

[0016] A "video" is a medium that includes continuous images and sounds and is used to visually show operation procedures and usage methods.

[0017] An "inquiry" refers to questions and requests sent by a user seeking specific information or support.

[0018] An "answer" is appropriate information and instructions provided by a generative artificial intelligence in response to a user's inquiry.

[0019] "Man-hours" is a unit of labor and time required for a specific task or project.

[0020] ---

[0021] Thus, definition sentences have been created for the important words included in the claims.

Brief Description of the Drawings

[0022] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7]It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0023] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0024] First, the language used in the following description will be explained.

[0025] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0026] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0027] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0028] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0030] [First Embodiment]

[0031] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0032] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0035] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0038] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0042] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0043] ---

[0044] This invention provides a system that enables the learning of operating procedures using generative artificial intelligence, the automatic generation of user manuals, and the automation of inquiry handling. The system of this invention operates as follows.

[0045] 1. Initial setup and retrieval of service list

[0046] The user starts the system on their terminal. After the terminal loads the system's initial settings, it connects to the server and retrieves a list of services provided. This list is sent back to the terminal from the server in JSON format.

[0047] 2. Collecting user operation logs

[0048] When a user interacts with a specific service in their browser, the device records an operation log in real time. This operation log includes information such as click locations, input content, page transitions, and error messages, and this information is recorded along with a timestamp.

[0049] 3. Learning Phase of Generative Artificial Intelligence

[0050] The terminal sends recorded operation logs to the server at regular intervals. The server uses generative artificial intelligence to analyze the transmitted operation logs and learn the operation procedures related to the service. Specifically, it understands the order and conditions under which specific operations are performed and uses that information to systematize the operation procedures.

[0051] 4. Automatic generation of user manuals

[0052] After learning the operating procedures for the service, the server automatically generates a user manual using generative artificial intelligence. The generated manual includes text explanations of the operating procedures, screenshots, and, if necessary, videos. For example, a manual on setting up campaigns for an email marketing tool would show steps such as "Step 1: Click 'New Campaign' from the dashboard" and "Step 2: Enter the campaign name."

[0053] 5. Automation of inquiry handling

[0054] When a user makes an inquiry about a specific service, the device sends the inquiry to the server. The server's generative artificial intelligence analyzes the inquiry and generates appropriate operating procedures or answers based on that analysis. The generated answers are sent back to the device, which then displays the answers to the user. For example, if a user asks, "I don't know how to set up a campaign," the server will provide an answer in the form of, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details."

[0055] This embodiment significantly streamlines the handling of inquiries regarding service operation methods. Furthermore, because the latest operating procedures and manuals are always provided through learning and automatic generation by generative artificial intelligence, an improved user experience can be expected.

[0056] The following describes the processing flow.

[0057] ---

[0058] Step 1: System Startup

[0059] The user starts the system on the terminal.

[0060] The terminal reads the system configuration file and performs an initialization process.

[0061] Step 2: Obtain the list of services

[0062] The device sends a REST API request to the server, requesting a list of the services it provides.

[0063] The server retrieves a list of currently available services from the database and sends it back to the terminal in JSON format.

[0064] The terminal displays a list of received services to the user.

[0065] Step 3: Start collecting user operation logs.

[0066] The user interacts with a specific service through their browser.

[0067] The device records user actions (clicks, input, page transitions, error messages, etc.) in real time.

[0068] These operation logs include a timestamp and information about the element that was manipulated.

[0069] Step 4: Sending the operation log

[0070] The device sends operation logs recorded at regular intervals to the server.

[0071] The server saves the received operation logs to a database for analysis.

[0072] Step 5: Learning Generative Artificial Intelligence

[0073] The server provides the stored operation logs to the generative artificial intelligence.

[0074] Generative artificial intelligence analyzes these operation logs and learns the operating procedures for each service.

[0075] The server stores the learning results in a database.

[0076] Step 6: Automatic generation of user manual

[0077] The server uses generative artificial intelligence to generate a user manual based on the operating procedures.

[0078] The manual includes textual instructions, screenshots, and, where necessary, videos of the operating procedures.

[0079] For example, for "How to set up a campaign in an email marketing tool," specific steps such as "Step 1: Click 'New Campaign' from the dashboard" are generated.

[0080] Step 7: Receiving the inquiry

[0081] A user sends an inquiry about a specific service from their device.

[0082] The terminal sends the entered query content to the server.

[0083] Step 8: Analyzing the inquiry content

[0084] The server's generation-based artificial intelligence analyzes the content of the received inquiry.

[0085] Based on the analysis results, the system searches for the information and operating procedures requested by the user and generates appropriate answers.

[0086] Step 9: Submit and view your answer

[0087] The server sends the generated answer back to the terminal.

[0088] The terminal displays the received answer to the user.

[0089] For example, in response to an inquiry such as "I don't know how to set up a campaign," the message displayed might be, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details."

[0090] ---

[0091] This clearly explains the specific processing steps, from initial setup and operation log collection to learning by generative artificial intelligence, automatic generation of user manuals, and handling inquiries, as a continuous flow.

[0092] (Example 1)

[0093] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0094] Currently, many users spend a considerable amount of time and effort understanding the procedures for using software and web services. Furthermore, the high volume of inquiries regarding operation procedures and settings places a significant burden on customer support. Traditional user manuals tend to be outdated, making it difficult to obtain the latest information. Therefore, there is a need for a system that allows users to easily learn operation procedures and quickly resolve problems.

[0095] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0096] In this invention, the server includes means for learning operating procedures using generative artificial intelligence, means for automatically generating a user manual, means for automatically generating and providing answers to user inquiries, means for collecting user operation logs in real time and transmitting them to the server, and means for generating text descriptions, screenshots, and videos of operations based on the learned operating procedures. This allows users to quickly obtain the latest and most accurate operating procedures, and enables more efficient handling of inquiries.

[0097] "Generative artificial intelligence" is a type of artificial intelligence that analyzes user operations and inquiries to generate optimal answers and procedures.

[0098] "Learning operating procedures" is the process of analyzing user operation logs to understand and systematize the steps and flows necessary for those operations.

[0099] A "user manual" is a guideline document that describes the procedures and settings that users should follow when using software or web services.

[0100] An "operation log" is data that records a series of operations performed by a user when using software or web services, in chronological order.

[0101] A "server" is a computer system that provides services to other terminals or systems on a network.

[0102] A "device" is a device that a user directly operates (for example, a personal computer, tablet, or smartphone).

[0103] A "prompt sentence" is an input sentence used to prompt a generative artificial intelligence to produce a specific answer or output.

[0104] "Text instructions" refer to instructions on how to use the device or answers to inquiries presented in written form.

[0105] A "screenshot" is an image captured from a computer screen at a specific point in time.

[0106] A "video" is a recording of operating procedures or settings methods in moving images to visually demonstrate them.

[0107] This invention provides a system that enables the learning of operating procedures using generative artificial intelligence, the automatic generation of user manuals, and the automation of inquiry handling. The system of this invention operates as follows.

[0108] Initial setup and retrieval of service list

[0109] The user starts the system on the terminal. The terminal reads a configuration file (such as config.json) as part of its initial setup. After reading the file, the terminal automatically connects to the server and sends a request to the server to retrieve a list of services. The server returns the service list to the terminal in JSON format, and the terminal displays that data on the screen. For example, when a user starts a new system for the first time, the configuration file is read, the connection to the server is made, and the service list is retrieved.

[0110] User operation log collection

[0111] When a user begins interacting with a specific service, the device starts recording that interaction in real time. The recorded activity log includes click locations, input content, page transitions, and error messages. This data is stored along with timestamps. For example, the user filling out a form and clicking the "Submit" button is recorded in the log.

[0112] Learning phase of generative artificial intelligence

[0113] At regular intervals, the terminal sends collected operation logs to the server. The server analyzes the received operation logs and learns operation procedures using generative artificial intelligence (for example, OpenAI®'s GPT-4®). Based on the analysis results, it determines whether a particular operation is important and systematizes the operation procedures accordingly. For example, it understands and structures the flow as "Step 1: Log in" and "Step 2: Go to dashboard".

[0114] Automatic generation of user manuals

[0115] The server automatically generates user manuals based on learned operating procedures. Generative artificial intelligence is used to create text explanations, screenshots, and, if necessary, videos of the operating procedures. For example, a manual on setting up campaigns for an email marketing tool would be structured as follows: "Step 1: Click 'New Campaign' from the dashboard" and "Step 2: Enter the campaign name."

[0116] Automating customer support

[0117] When a user makes an inquiry about a specific service, the device sends the user's inquiry to the server. The server's generative artificial intelligence analyzes the inquiry and generates appropriate operating procedures or answers. For example, if a user asks, "I don't know how to set up a new campaign," the server will provide an answer in the form of, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details."

[0118] Example of a prompt

[0119] For example, you can ask a generative AI model the following question:

[0120] "If a user contacts you because they don't know how to set up a new campaign, please provide them with appropriate instructions."

[0121] In response to this prompt, the server's generative artificial intelligence might generate the following answer:

[0122] "Step 1: Click 'New Campaign' from the dashboard. Step 2: Enter a campaign name and click 'Next'. Step 3: Set specific goals on the goal setting screen. Finally, click 'Save'."

[0123] Thus, the system of the present invention learns the user's operating procedures and automatically generates and updates the user manual based on them. Furthermore, it can quickly provide appropriate answers to user inquiries. As a result, the user experience is improved, and the efficiency of handling inquiries is enhanced.

[0124] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0125] Step 1:

[0126] System startup and initial setup

[0127] The user starts the system on the terminal. Specifically, the user double-clicks a desktop icon or selects an app from the Start menu. After startup, the terminal reads a configuration file (such as config.json) and performs initial setup. This configuration file contains server connection information and initial user interface settings. The input is the user's startup action, and the output is the status of the configuration loading completion.

[0128] Step 2:

[0129] Connecting to the server and retrieving a list of services

[0130] After initial setup, the device automatically connects to the server and sends a request to retrieve a list of services. The server receives this request and returns the service list in JSON format. For example, the service list might include "email marketing," "web analytics," and "ad delivery." The input is the connection request to the server, and the output is the service list in JSON format from the server.

[0131] Step 3:

[0132] User operation log collection started.

[0133] When a user selects a specific service and begins an operation, the terminal starts recording an operation log in real time. The recorded operation log includes click locations, input content, page transitions, error messages, and more. Input is the user's operation action, and output is the start status of the operation log.

[0134] Step 4:

[0135] Recording and saving operation logs

[0136] The terminal meticulously records each user action in chronological order. For example, the actions of a user filling out a form and clicking the "Submit" button are saved as logs. Inputs are the user's specific actions, and outputs are the recorded action logs.

[0137] Step 5:

[0138] Sending operation logs

[0139] The terminal sends the collected operation logs to the server in batch processing at regular intervals. For example, if the terminal is configured to send operation logs to the server every minute, batch processing will occur once every 60 seconds. The input is the collected operation logs, and the output is the operation logs sent to the server.

[0140] Step 6:

[0141] Operation log analysis using generative artificial intelligence

[0142] The server analyzes the received operation logs using generative artificial intelligence (e.g., OpenAI's GPT-4). Specifically, it analyzes each operation log to identify important operation procedures and flows. The input is the operation log, and the output is the analysis results.

[0143] Step 7:

[0144] Learning and systematizing operating procedures

[0145] The server learns the operating procedures using generative artificial intelligence based on the analysis results. In this process, it understands the order and conditions under which specific operations are performed and systematizes the operating procedures based on that. For example, specific procedures such as "Step 1: Log in" and "Step 2: Go to the dashboard" are generated. The input is the analysis results, and the output is the learned operating procedures.

[0146] Step 8:

[0147] Automatic generation of user manuals

[0148] The server automatically generates a user manual using generative artificial intelligence based on learned operating procedures. This manual includes text descriptions of the operating procedures, screenshots, and, if necessary, videos. The input is the learned operating procedures, and the output is the generated user manual.

[0149] Step 9:

[0150] User manual provided.

[0151] When a user wants to refer to a manual for a specific service, the terminal retrieves the manual generated from the server and displays it on the screen. The input is the user's manual reference request, and the output is the user manual displayed on the screen.

[0152] Step 10:

[0153] User inquiries accepted.

[0154] When a user makes an inquiry about a specific service, the device sends the inquiry details to the server. For example, a user might inquire, "I don't know how to set up a new campaign." The input is the user's inquiry, and the output is the inquiry data sent to the server.

[0155] Step 11:

[0156] Inquiry content analysis using generative artificial intelligence

[0157] The server's generative artificial intelligence analyzes the received inquiry and generates appropriate operating procedures and answers. For example, it might provide an answer in the form of, "Step 1: Click 'New Campaign' from the dashboard. Step 2: Enter the campaign name." The input is the user's inquiry, and the output is the generated answer.

[0158] Step 12:

[0159] Providing the answer

[0160] The server sends the generated answer back to the terminal, which then displays it to the user. The input is the generated answer, and the output is the answer displayed to the user.

[0161] As described above, the system of the present invention allows users to easily learn the operating procedures, quickly solve problems, and improve the efficiency of handling inquiries.

[0162] (Application Example 1)

[0163] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0164] Many current systems suffer from a lack of user manuals to help users understand operating procedures, and support for inquiries is often time-consuming and cumbersome, thus compromising user convenience. Therefore, there is a need for systems that automatically generate user manuals regarding operating procedures and streamline inquiry handling. Furthermore, it is crucial that the generated manuals always reflect the latest information, but this is difficult to achieve with conventional methods. Therefore, a system that addresses these challenges is necessary.

[0165] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0166] In this invention, the server includes means for learning operating procedures using generative artificial intelligence, means for automatically generating a user manual, means for automatically generating and providing answers to user inquiries, means for collecting and transmitting user operation logs in real time, means for including text explanations, images, and videos in the generated user manual, and means for providing operating procedures and answers through a smartphone application. This enables users to efficiently understand operating procedures and quickly obtain answers to their inquiries.

[0167] "Generative artificial intelligence" is an artificial intelligence technology that can generate and predict natural language based on vast amounts of data.

[0168] An "operating procedure" is a series of actions or steps necessary to accomplish a specific task or function.

[0169] A "user manual" is a document that describes the procedures and precautions that users should take when operating a system or service.

[0170] "Inquiry support" refers to the activity of providing appropriate information and solutions to questions and problems from users.

[0171] A "user operation log" is a series of action logs recorded when a user operates the system.

[0172] "Real-time" refers to operations and data processing being performed instantly.

[0173] "Text instructions" refer to written descriptions of operating procedures and precautions.

[0174] An "image" is a still image used to provide information visually.

[0175] A "video" is a visual representation of movement created by playing multiple images in sequence.

[0176] A "smartphone application" is a computer program that runs on a smartphone.

[0177] This invention provides a system that uses generative artificial intelligence to automate the learning of operating procedures, the automatic generation of user manuals, and the handling of inquiries. Specific embodiments of this system are described below.

[0178] 1. System Configuration

[0179] This system consists of the following main components:

[0180] Terminal: Smartphone application. It is responsible for collecting user operation logs, sending inquiries, and displaying generated user manuals and inquiry responses.

[0181] Server: Performs learning and analysis using generative artificial intelligence. It also saves user operation logs, generates user manuals based on analysis results, and generates appropriate answers for inquiries.

[0182] 2. Hardware and software to be used

[0183] Hardware: Smartphones (iOS, Android® compatible), Cloud servers (e.g., AWS®, Google Cloud, Azure®)

[0184] Software: Smartphone applications (frameworks: Flutter®, React Native, etc.), server-side applications (frameworks: Node.js, Django, etc.), generative artificial intelligence models (e.g., OpenAI GPT-4, GPT-3®)

[0185] 3. Data processing and calculations

[0186] Collection of user operation logs

[0187] Smartphone applications running on a device collect log data in real time as the user interacts with it. For example, they record when a user clicks a button, the text they enter, and information about page transitions. This data is sent to a server using methods such as WebSocket.

[0188] Analysis and learning using generative AI

[0189] On the server, collected operation logs are analyzed and learned from by generative artificial intelligence at regular intervals. The generative AI analyzes the operation logs and patterns and defines operation procedures. These learning results are stored in a database (e.g., MongoDB, MySQL®).

[0190] Automatic generation of user manuals

[0191] The server automatically generates user manuals based on operating procedures analyzed and learned using generative AI. The manuals include text explanations, images, and videos as needed. For example, a manual on how to order a product would include steps such as "Step 1: Click 'Order History' from the dashboard" and "Step 2: Select the order and click 'Cancel'," along with corresponding screenshots.

[0192] Automating customer support

[0193] When a user submits an inquiry within the smartphone application, the content is sent to the server. The server's generation AI analyzes the inquiry and generates an appropriate answer. This answer, including operating procedures and countermeasures, is sent back to the user in text format.

[0194] 4. Specific Examples and Prompts

[0195] Specific example:

[0196] If a user asks within the app, "I don't know how to cancel my order," the generative AI will create a prompt message like the following, analyze it, and then provide a response.

[0197] Example of a prompt:

[0198] A user has inquired about how to cancel an order. Please explain the order cancellation procedure. Include the following information: details of each step, necessary screenshots, and links.

[0199] Generated example answer:

[0200] Here's how to cancel your order:

[0201] 1. From the dashboard screen, click "Order History".

[0202] ![Screenshot of Step 1]()

[0203] 2. Select the order you wish to cancel and click "Cancel".

[0204] ![Screenshot of Step 2]()

[0205] 3. A confirmation message will appear; select "Yes".

[0206] ![Screenshot of Step 3]()

[0207] For detailed instructions, please refer to [this link](https: / / support.example.com / cancel-order).

[0208] In this way, the present invention provides a system that enables users to efficiently understand operating procedures and quickly obtain answers to their inquiries.

[0209] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0210] Step 1:

[0211] The device collects user activity logs in real time through a smartphone application. Input includes user actions (clicks, text input, page transitions), and this information is recorded along with a timestamp. Output is generated as activity log data, which is sent to the server using WebSocket.

[0212] Step 2:

[0213] The server saves the received operation log data to the database. The received operation log data is taken as input, and the saving process is performed to the database (e.g., MongoDB, MySQL). A status indicating successful saving is generated as output. This step involves specific actions to format the data according to data integrity and the structure of the destination table.

[0214] Step 3:

[0215] The server periodically retrieves operation log data from the database and uses the generative artificial intelligence model GPT-4 to analyze and train it. The retrieved operation log data is the input, and the generative AI model analyzes this data. As output, patterns of operation procedures as a result of training are generated and stored in the database. Specifically, data preprocessing (denoising, normalization, etc.) is performed, and the analysis results are defined as operation procedures.

[0216] Step 4:

[0217] The server uses generative artificial intelligence to automatically generate user manuals based on stored operating procedures. The input is a set of learned operating procedures, and the generative AI generates text explanations, images, and, if necessary, videos. The user manual is generated as output and stored on the server. Specifically, the operating procedures are converted into text in a sequential manner, and images and videos are captured and edited.

[0218] Step 5:

[0219] When a user submits an inquiry within a smartphone application, the device sends the content to the server. The input is the user's inquiry content, which is sent to the server in text format. The output is the text of the inquiry content reaching the server. Specifically, this includes text input in an input form and clicking the submit button.

[0220] Step 6:

[0221] The server's generative AI analyzes the query content, creates a prompt, and generates an appropriate answer. The input is the text of the query, and a prompt is generated based on it. The generative AI model analyzes this prompt and generates an appropriate answer. The output is the generated answer text. Specifically, prompt generation and subsequent answer generation are performed.

[0222] Step 7:

[0223] The server sends the generated response back to the device, which then displays it to the user. The input is the generated response text, which is sent to the device. The output is the response displayed on the user's smartphone screen. Specifically, text data is transmitted over the network, and the display component handles the screen display.

[0224] These steps enable users to efficiently understand the operating procedures and quickly obtain answers to their inquiries.

[0225] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0226] ---

[0227] This invention is a system that combines learning of operating procedures using generative artificial intelligence, automatic generation of user manuals, automation of inquiry handling, and an emotion engine that recognizes user emotions. This system further streamlines the handling of inquiries regarding the use of specific services and improves the user experience.

[0228] 1. Initial setup and retrieval of service list

[0229] The user starts the system on their terminal. The terminal reads the system's configuration file, then connects to the server to retrieve a list of services provided. This list is sent back to the terminal from the server in JSON format. The terminal then displays the received list of services to the user.

[0230] 2. Collection of user operation logs and sentiment data

[0231] When a user interacts with a specific service in their browser, the device records an operation log in real time. This operation log includes click locations, input content, page transitions, and error messages. The device also incorporates an emotion engine that collects emotional data from the user's facial expressions and voice. This emotional data is also recorded along with the operation log and stored with a timestamp.

[0232] 3. Sending operation logs and sentiment data

[0233] The device sends operation logs and emotion data recorded at regular intervals to the server. The server stores the received data in a database for analysis.

[0234] 4. Learning of Generative Artificial Intelligence

[0235] The server provides stored operation logs and emotion data to a generative artificial intelligence (AI). The AI ​​analyzes this data and learns not only the operation procedures for each service, but also makes adjustments based on the user's emotional state. For example, if the user is confused, it learns to explain the operation procedures more specifically and carefully. The server stores the learning results in a database.

[0236] 5. Automatic generation of user manuals

[0237] After learning the operating procedures for the service, the server automatically generates a user manual using generative artificial intelligence. The generated manual includes text explanations of the operating procedures, screenshots, and, if necessary, videos. It also includes content that is customized to the user's emotions. For example, if the user is nervous, it may include additional advice such as "Please relax and proceed."

[0238] 6. Automation of inquiry handling

[0239] When a user makes an inquiry about a specific service, the device sends the inquiry to the server. The server's generative artificial intelligence analyzes the received inquiry and sentiment data, and based on that, generates the optimal operating procedures and answers. For example, if a user asks, "I don't know how to set up a campaign," the server will consider the user's sentiment state and provide an answer such as, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details. Don't worry, it's easy to set up."

[0240] ---

[0241] This embodiment is expected to further streamline the handling of user inquiries regarding operation methods, and to significantly improve the user experience through the collaboration of generative artificial intelligence and an emotion engine.

[0242] The following describes the processing flow.

[0243] ---

[0244] Step 1: System Startup

[0245] The user starts the system on the terminal.

[0246] The terminal reads the system configuration file and performs an initialization process.

[0247] Step 2: Obtain the list of services

[0248] The device sends a REST API request to the server, requesting a list of the services it provides.

[0249] The server retrieves a list of currently available services from the database and sends it back to the terminal in JSON format.

[0250] The terminal displays a list of received services to the user.

[0251] Step 3: User begins operation

[0252] The user interacts with a specific service through their browser.

[0253] The device records user actions (clicks, input, page transitions, error messages, etc.) in real time.

[0254] Step 4: Collecting user sentiment data

[0255] The device's built-in emotion engine collects emotional data from the user's facial expressions and voice.

[0256] The collected emotional data is recorded along with a timestamp, similar to the operation log.

[0257] Step 5: Sending operation logs and emotion data

[0258] The device sends operation logs and emotion data recorded at regular intervals to the server.

[0259] The server stores the received data in a database for analysis.

[0260] Step 6: Learning Generative Artificial Intelligence

[0261] The server provides stored operation logs and emotion data to the generative artificial intelligence.

[0262] Generative artificial intelligence analyzes this data and learns the operating procedures for each service, as well as making adjustments based on the user's emotional state.

[0263] For example, if a user is confused, the system will learn to explain the operating procedures in more detail and with greater care.

[0264] The server stores the learning results in a database.

[0265] Step 7: User manual generation

[0266] The server uses generative artificial intelligence to generate a user manual based on the operating procedures.

[0267] The generated manual will include textual instructions, screenshots, and, if necessary, videos of the operating procedures.

[0268] It also includes content that is customized to the user's emotions. For example, if the user is feeling nervous, it may include additional advice such as, "Please relax and proceed."

[0269] Step 8: Inquiry received

[0270] A user sends an inquiry about a specific service from their device.

[0271] The terminal sends the entered query content to the server.

[0272] Step 9: Analysis of inquiry content and sentiment data

[0273] The server's generative artificial intelligence analyzes the received inquiry content and sentiment data.

[0274] Generative artificial intelligence generates optimal operating procedures and answers while taking into account the user's emotional state.

[0275] For example, if a user asks, "I don't know how to set up a campaign," the server will respond with something like, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details. Don't worry, it's easy to set up."

[0276] Step 10: Submit and view your answer

[0277] The server sends the generated answer back to the terminal.

[0278] The terminal displays the received answer to the user.

[0279] This allows users to receive appropriate information in a way that is sensitive to their emotions.

[0280] ---

[0281] In this way, the overall processing flow of the system is explained by breaking it down into specific steps.

[0282] (Example 2)

[0283] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart device 14 is referred to as a "terminal".

[0284] In a conventional system, since the learning of the user's operation procedure and the generation of the manual are performed manually, the efficiency is low, and it is particularly difficult to take into account the user's feelings. In addition, since the inquiry response also needs to be performed individually, there are likely to be variations in the speed and quality of the response. As a result, it is difficult to improve the user experience, and dissatisfaction may occur with the use of the service. Therefore, there has been a demand for a system that collects operation logs and emotion data in real time, learns and automatically generates operation procedures using a generative artificial intelligence based on the data, and improves the user experience.

[0285] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0286] In this invention, the server includes means for starting the system by the user terminal, reading the configuration file, and acquiring the service list, means for collecting and recording the operation logs and emotion data of the user in real time, and means for transmitting the collected operation logs and emotion data to the server at regular intervals and storing them in the database. As a result, it becomes possible to learn the operation procedure, automatically generate a user manual, and automate the inquiry response.

[0287] 1. The "user terminal" is a device such as a computer or a smartphone used by the user to operate the system.

[0288] 2. The "configuration file" is a file that describes various information necessary for the operation of the system, including initial settings and connection information.

[0289] 3. The "service list" is a list of various services provided by the server, and is a group of information indicating the service contents available to the user.

[0290] 4. An "operation log" is a record of the operations performed by a user on the system, and includes data such as click locations, input content, page transitions, and error messages.

[0291] 5. "Emotional data" refers to data that indicates the emotional state of a user, analyzed from their facial expressions and voice.

[0292] 6. "Real-time" refers to the processing and recording of data and operations almost simultaneously.

[0293] 7. A "server" is a computer system that provides services over a network, including data storage and analysis.

[0294] 8. A "database" is a system that systematically manages and stores operation logs, sentiment data, and other similar information.

[0295] 9. "Generative artificial intelligence" refers to an AI model that analyzes and learns from provided data to generate new information and procedures.

[0296] 10. "Learning" is the process of analyzing features from data and accumulating information internally that improves future predictions and judgments.

[0297] 11. A "user manual" is a guide that explains the procedures and methods for users to use a system.

[0298] 12. "Inquiry support" is the process of providing answers and support to questions and problems submitted by users.

[0299] 13. "Analysis" is the process of examining data in detail and understanding its meaning and relationships.

[0300] 14. A "procedure" refers to a series of steps or operations necessary to achieve a certain objective.

[0301] The present invention is a system that utilizes a user terminal, a server, and a generative artificial intelligence to perform learning of operation procedures, automatic generation of user manuals, and automation of inquiry responses. This improves the efficiency of inquiry responses regarding the user's operation method and significantly enhances the user experience.

[0302] Obtaining Initial Settings and Service List

[0303] When the user starts the system on the terminal, the terminal first loads the configuration file. This configuration file contains connection information and the URL for obtaining the service list. The terminal sends an HTTP request to the server based on the configuration file and obtains the service list. The server returns the service list in JSON format, and the terminal analyzes this and displays it to the user.

[0304] Specific Example:

[0305] The user starts the system on the browser of a personal computer and loads the configuration file (config.json). The terminal sends an HTTP request to the server and obtains a list of the provided services. The server returns JSON data containing information such as the name, description, and provider of each service in response to the request. The terminal parses this data and displays it on the user interface.

[0306] Example of Prompt Sentence for the Generative AI Model:

[0307] "Please teach me the procedure for loading the configuration file and obtaining data to get a list of the provided services."

[0308] Collection of User Operation Logs and Sentiment Data

[0309] When the user operates a specific service on the browser, the terminal records operation logs such as click position, input content, page transition, error message, etc. in real time. Furthermore, sentiment data is collected from the user's expressions and voices through a sentiment engine. These data are saved together with timestamps.

[0310] Specific example:

[0311] When a user fills out a form on a web system, the device records the buttons clicked and the contents of the entered text fields. An emotion engine is used to collect the user's facial expression data and analyze the user's emotions (e.g., happiness, anxiety, anger).

[0312] Examples of prompts for a generative AI model:

[0313] "Please provide a script for recording user activity logs on a webpage. Also, please explain how to use an emotion engine to extract emotions from a user's facial expressions."

[0314] Sending operation logs and sentiment data

[0315] The collected operation logs and emotion data are sent from the terminal to the server at regular intervals. The server stores the received data in a database.

[0316] Specific example:

[0317] The terminal sends operation logs and sentiment data to the server in batch processing every 10 minutes. The server receives the JSON data and saves it to a MySQL database.

[0318] Examples of prompts for a generative AI model:

[0319] "Please tell me the procedure for periodically sending locally recorded data to a server and saving it to a database."

[0320] Learning of generative artificial intelligence

[0321] The server provides stored operation logs and emotion data to a generative artificial intelligence (AI). The AI ​​analyzes this data and learns the operation procedures for each service. In particular, it also makes adjustments based on the user's emotional state.

[0322] Specific example:

[0323] The server uses stored operation logs and sentiment data to have a generative AI model perform analysis. The model learns to generate more specific explanations for operation procedures that frequently confuse users.

[0324] Examples of prompts for a generative AI model:

[0325] "How can we use operation logs and sentiment data to learn appropriate operating procedures based on the user's level of confusion?"

[0326] Automatic generation of user manuals

[0327] The server uses generative artificial intelligence to automatically generate user manuals based on what it has learned. The generated manuals include textual explanations of operating procedures, screenshots, and, if necessary, videos. Advice tailored to the user's emotions is also added.

[0328] Specific example:

[0329] The server generates a manual for specific operating procedures. The generated manual includes detailed explanations for each step, relevant images, and advice for confused users such as, "Don't worry, just click here."

[0330] Examples of prompts for a generative AI model:

[0331] "Please tell me how to automatically generate a user manual based on operating procedures. Please also include additional advice that takes sentiment data into consideration."

[0332] Automating customer support

[0333] When a user makes an inquiry about a specific service, the device sends the inquiry details to the server. Generative artificial intelligence analyzes this data and generates and provides the optimal operating procedures and answers.

[0334] Specific example:

[0335] When a user inquires that they "don't know how to reset their password," the server's generated AI model analyzes the relevant steps and provides guidance such as, "Go to your account settings page and click the password reset link."

[0336] Examples of prompts for a generative AI model:

[0337] "Please tell me how to automatically generate optimal operating procedures and answers based on user inquiries."

[0338] This system streamlines the handling of user inquiries regarding operation methods and significantly improves the user experience through the integration of generative artificial intelligence and an emotion engine.

[0339] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0340] Step 1:

[0341] The user starts the system on the terminal.

[0342] Specific operation: The user accesses the system URL using a browser on their PC or smartphone and starts the system.

[0343] Input: System URL

[0344] Output: System initial screen

[0345] Step 2:

[0346] The terminal reads the configuration file.

[0347] Specific operation: The terminal opens the configuration file (config.json) in the program directory and retrieves the necessary connection information and API key.

[0348] Input: Configuration file (config.json)

[0349] Output: Connection information, API key

[0350] Step 3:

[0351] The terminal sends an HTTP request to the server to retrieve a list of services.

[0352] Specific operation: The terminal sends a GET request to a URL read from the configuration file, requesting data for the list of services. The server receives the list of services in JSON format.

[0353] Input: Connection information (URL), API key

[0354] Output: List of services in JSON format

[0355] Step 4:

[0356] The terminal analyzes the list of services it has received and displays it to the user.

[0357] Specific operation: The terminal parses the received JSON data and displays the name, description, provider, etc. of each service in the user interface.

[0358] Input: List of services in JSON format

[0359] Output: List of services displayed in the user interface

[0360] Step 5:

[0361] The user interacts with a specific service through their browser.

[0362] Specific actions: Users fill out forms or click buttons to use the service.

[0363] Input: User actions

[0364] Output: System response to user interaction (page transitions, submission of input, etc.)

[0365] Step 6:

[0366] The device records user activity logs in real time.

[0367] Specific operation: Using JavaScript® and other client-side technologies, data such as click location, input content, page transitions, and error messages are captured in real time and saved to a log file or memory.

[0368] Input: User actions

[0369] Output: Operation Log

[0370] Step 7:

[0371] The device uses an emotion engine to collect emotional data from the user's facial expressions and voice.

[0372] Specific operation: The system uses a camera and microphone to capture the user's facial expressions and voice, and generates emotional data through an emotion analysis API. For example, it can determine emotions such as happiness, anxiety, and anger.

[0373] Input: User's facial expressions, voice data

[0374] Output: Sentiment data

[0375] Step 8:

[0376] The device sends operation logs and emotion data collected at regular intervals to the server.

[0377] Specific operation: Every 10 minutes, the terminal batch processes and collects operation logs and sentiment data, then sends them to the server via an HTTP POST request.

[0378] Input: Operation log, sentiment data

[0379] Output: Data sent to the server

[0380] Step 9:

[0381] The server saves the received operation logs and sentiment data to a database.

[0382] Specific operation: The server parses the received JSON data and writes it to a database such as MySQL or PostgreSQL.

[0383] Input: Received operation logs, sentiment data

[0384] Output: Data stored in the database

[0385] Step 10:

[0386] The server provides stored operation logs and emotion data to the generative artificial intelligence.

[0387] Specific operation: The server periodically extracts operation logs and sentiment data from the database and feeds them to the generative artificial intelligence.

[0388] Input: Operation log, sentiment data

[0389] Output: Data provided to the generative artificial intelligence.

[0390] Step 11:

[0391] Generative artificial intelligence analyzes the provided data and learns the operating procedures.

[0392] Specific operation: Generative artificial intelligence analyzes user interaction patterns and emotional states to learn the optimal operating procedures for each service. This includes generating detailed guides for operating procedures that frequently confuse users.

[0393] Input: Operation log, sentiment data

[0394] Output: Learned operating procedures

[0395] Step 12:

[0396] The server automatically generates a user manual based on what it has learned using generative artificial intelligence.

[0397] Specific operation: The server automatically generates a manual based on the operating procedures learned by the generative artificial intelligence. The generated manual includes text explanations, screenshots, and videos. It also adds advice tailored to the user's emotions.

[0398] Input: Learned operating procedures

[0399] Output: User Manual

[0400] Step 13:

[0401] A user makes an inquiry about a specific service.

[0402] Specific action: The user enters a question into the inquiry form and clicks the submit button.

[0403] Input: Inquiry details

[0404] Output: Query received by the server

[0405] Step 14:

[0406] The server receives the inquiry, and a generative artificial intelligence analyzes the inquiry content and sentiment data.

[0407] Specific operation: The generative artificial intelligence analyzes the received inquiry content and past sentiment data to generate the optimal operating procedure and answer.

[0408] Input: Inquiry details, sentiment data

[0409] Output: Generated operating procedures and solutions

[0410] Step 15:

[0411] The server provides the generated operating procedures and solutions.

[0412] Specific operation: The server returns the generated answer to the user. For example, in response to an inquiry such as "I don't know how to reset my password," it will return specific instructions such as "Go to the account settings page and click the password reset link."

[0413] Input: Generated operating procedures and answers

[0414] Output: Answers provided to the user

[0415] This system streamlines the handling of user inquiries regarding operation methods, and significantly improves the user experience through the integration of generative artificial intelligence and an emotion engine.

[0416] (Application Example 2)

[0417] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0418] In recent years, many service providers have aimed to improve the efficiency of user support, but traditional systems often failed to adequately address user inquiries about how to use the service. Furthermore, support methods that took user emotions into consideration were not provided, resulting in a lack of improvement in the user experience. This raised concerns that user dissatisfaction would increase, potentially leading to service interruptions or cancellations.

[0419] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0420] In this invention, the server includes means for learning operating procedures using generative artificial intelligence, means for automatically generating a user manual, means for automatically generating and providing answers to user inquiries, and means for recognizing the user's emotions in real time and adjusting the operating procedures and answers based on that state. This makes it possible to respond to user inquiries about how to operate the system efficiently and in a way that is considerate of the user's emotions.

[0421] "Generative artificial intelligence" refers to artificial intelligence that has the ability to learn the user's operating procedures and automatically generate appropriate answers or manuals.

[0422] "Operating procedures" refer to a series of actions or steps that a user must perform when using a particular service or system.

[0423] A "user manual" is a guide containing documents, images, videos, and other materials generated to explain how to use a particular service or system.

[0424] "Inquiry support" refers to the activity of providing appropriate answers and solutions to questions and problems from users.

[0425] An "emotion engine" is a technology that analyzes a user's facial expressions and voice to recognize their emotions in real time.

[0426] An "operation log" is data that records the specific actions and inputs a user makes while operating a system or service.

[0427] A "server" is a computer system that stores operation logs and emotional data, and performs analysis and manual generation using generative artificial intelligence.

[0428] "Text description" refers to information in a written format that explains operating procedures and methods to the user.

[0429] A "screenshot" refers to an image captured from the screen being operated by the user.

[0430] A "video" is video content used to dynamically explain operating procedures and methods to users.

[0431] "Customization" means adjusting or changing the content according to the user's specific situation or feelings.

[0432] Modes for carrying out the invention

[0433] This invention is a system that uses generative artificial intelligence and an emotion engine to automatically learn user procedures and generate user manuals and answers to inquiries. Specific embodiments of this system are described below.

[0434] 1. Initial setup and obtaining a list of services

[0435] When the terminal starts the system, it first performs initial setup. The terminal reads a configuration file and connects to the server to retrieve a list of services provided. This list is sent back to the terminal from the server in JSON format, and the terminal displays the received list of services to the user.

[0436] 2. Collection of user operation logs and sentiment data

[0437] When a user interacts with a specific service, the device records an operation log in real time. This operation log includes click locations, input content, page transitions, and error messages. The device also incorporates an EmotionEngine, which collects emotional data from the user's facial expressions and voice. This emotional data is also recorded along with the operation log and stored with a timestamp.

[0438] 3. Sending operation logs and emotion data

[0439] The device sends operation logs and emotion data recorded at regular intervals to the server. The server stores the received data in a database for analysis.

[0440] 4. Learning of Generative Artificial Intelligence

[0441] The server provides stored operation logs and sentiment data to a generative artificial intelligence (AIAssistant). The generative AI analyzes this data and not only learns the operation procedures for each service, but also makes adjustments based on the user's emotions. For example, if the user is confused, it learns to explain the operation procedures more specifically and carefully. The server stores the learning results in a database.

[0442] 5. Automatic generation of user manuals

[0443] After learning the operating procedures for the service, the server automatically generates a user manual using generative artificial intelligence. The generated manual includes text explanations of the operating procedures, screenshots, and videos. It also includes content that is customized to the user's emotions. For example, if the user is nervous, it may include additional advice such as, "Please relax and proceed."

[0444] 6. Automation of inquiry handling

[0445] When a user makes an inquiry about a specific service, the device sends the inquiry to the server. The server's generative artificial intelligence analyzes the received inquiry and sentiment data, and based on that, generates the optimal operating procedures and answers. For example, if a user makes an inquiry such as "The video streaming keeps stopping," the generative AI analyzes the user's level of confusion using its sentiment engine when providing operating procedures, and generates an answer in the form of "Step 1: Check your Wi-Fi connection. Then, Step 2: Restart the app. Don't worry, it should go back to normal."

[0446] Specific example

[0447] Example of a printed statement:

[0448] Inquiry: Video streaming stops

[0449] Emotion: Confused (60%)

[0450] Answer: Step 1: Check your Wi-Fi connection. Then, Step 2: Restart the app. Don't worry, this should fix it.

[0451] This is expected to significantly improve the user experience by enabling more efficient and user-friendly responses to inquiries about how to use the system.

[0452] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0453] Step 1:

[0454] Initial setup and retrieval of service list

[0455] input:

[0456] Terminal startup and configuration file loading

[0457] process:

[0458] The terminal starts the system and reads the configuration file. Then it connects to the server and retrieves a list of services provided. The server returns the service list in JSON format.

[0459] output:

[0460] A list of acquired services will be displayed on the device.

[0461] Step 2:

[0462] Collection of user operation logs and sentiment data

[0463] input:

[0464] User service operation, facial expressions, voice

[0465] process:

[0466] While a user interacts with a specific service, the device records operation logs (click locations, input content, page transitions, error messages) in real time. Additionally, an EmotionEngine collects emotional data from the user's facial expressions and voice. Both the operation logs and emotional data are stored along with timestamps.

[0467] output:

[0468] Recorded operation logs and emotion data are saved to a file or database.

[0469] Step 3:

[0470] Sending operation logs and sentiment data

[0471] input:

[0472] Recorded operation logs and sentiment data

[0473] process:

[0474] The device sends recorded operation logs and emotion data to the server at regular intervals. The server stores the received data in a database for analysis.

[0475] output:

[0476] Operation logs and sentiment data stored in the server's database.

[0477] Step 4:

[0478] Learning of generative artificial intelligence

[0479] input:

[0480] Operation logs and sentiment data stored on the server

[0481] process:

[0482] The server provides operation logs and sentiment data to a generative artificial intelligence (AIAssistant). The generative AI analyzes this data and learns the operating procedures for each service. If the user is confused, it adjusts the instructions to explain them in a specific and detailed manner.

[0483] output:

[0484] The learning results are stored in the server's database.

[0485] Step 5:

[0486] Automatic generation of user manuals

[0487] input:

[0488] Learning results from generative artificial intelligence

[0489] process:

[0490] Based on learning, the server generates text instructions, screenshots, and videos of the operating procedures. Furthermore, it customizes them according to the user's emotional state. For example, if the user is nervous, it includes additional advice such as, "Please relax and proceed."

[0491] output:

[0492] An automatically generated user manual.

[0493] Step 6:

[0494] Automating customer support

[0495] input:

[0496] User inquiries and sentiment data

[0497] process:

[0498] When a user submits an inquiry, the device sends the inquiry details to the server. Generative artificial intelligence analyzes the inquiry details and sentiment data to generate the optimal operating procedures and answers. For example, if a user submits an inquiry about "video streaming stopping," the sentiment engine analyzes the user's level of frustration and generates an answer in the form of, "Step 1: Check your Wi-Fi connection. Then, Step 2: Restart the app. Don't worry, this should fix it."

[0499] output:

[0500] A prompt message that provides appropriate operating procedures or answers.

[0501] This is expected to significantly improve the user experience by enabling more efficient and user-friendly responses to inquiries about how to use the system.

[0502] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0503] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0504] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0505] [Second Embodiment]

[0506] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0507] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0508] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0509] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0510] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0511] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0512] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0513] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0514] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0515] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0516] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0517] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0518] ---

[0519] This invention provides a system that enables the learning of operating procedures using generative artificial intelligence, the automatic generation of user manuals, and the automation of inquiry handling. The system of this invention operates as follows.

[0520] 1. Initial setup and retrieval of service list

[0521] The user starts the system on their terminal. After the terminal loads the system's initial settings, it connects to the server and retrieves a list of services provided. This list is sent back to the terminal from the server in JSON format.

[0522] 2. Collecting user operation logs

[0523] When a user interacts with a specific service in their browser, the device records an operation log in real time. This operation log includes information such as click locations, input content, page transitions, and error messages, and this information is recorded along with a timestamp.

[0524] 3. Learning Phase of Generative Artificial Intelligence

[0525] The terminal sends recorded operation logs to the server at regular intervals. The server uses generative artificial intelligence to analyze the transmitted operation logs and learn the operation procedures related to the service. Specifically, it understands the order and conditions under which specific operations are performed and uses that information to systematize the operation procedures.

[0526] 4. Automatic generation of user manuals

[0527] After learning the operating procedures for the service, the server automatically generates a user manual using generative artificial intelligence. The generated manual includes text explanations of the operating procedures, screenshots, and, if necessary, videos. For example, a manual on setting up campaigns for an email marketing tool would show steps such as "Step 1: Click 'New Campaign' from the dashboard" and "Step 2: Enter the campaign name."

[0528] 5. Automation of inquiry handling

[0529] When a user makes an inquiry about a specific service, the device sends the inquiry to the server. The server's generative artificial intelligence analyzes the inquiry and generates appropriate operating procedures or answers based on that analysis. The generated answers are sent back to the device, which then displays the answers to the user. For example, if a user asks, "I don't know how to set up a campaign," the server will provide an answer in the form of, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details."

[0530] This embodiment significantly streamlines the handling of inquiries regarding service operation methods. Furthermore, because the latest operating procedures and manuals are always provided through learning and automatic generation by generative artificial intelligence, an improved user experience can be expected.

[0531] The following describes the processing flow.

[0532] ---

[0533] Step 1: System Startup

[0534] The user starts the system on the terminal.

[0535] The terminal reads the system configuration file and performs an initialization process.

[0536] Step 2: Obtain the list of services

[0537] The device sends a REST API request to the server, requesting a list of the services it provides.

[0538] The server retrieves a list of currently available services from the database and sends it back to the terminal in JSON format.

[0539] The terminal displays a list of received services to the user.

[0540] Step 3: Start collecting user operation logs.

[0541] The user interacts with a specific service through their browser.

[0542] The device records user actions (clicks, input, page transitions, error messages, etc.) in real time.

[0543] These operation logs include a timestamp and information about the element that was manipulated.

[0544] Step 4: Sending the operation log

[0545] The device sends operation logs recorded at regular intervals to the server.

[0546] The server saves the received operation logs to a database for analysis.

[0547] Step 5: Learning Generative Artificial Intelligence

[0548] The server provides the stored operation logs to the generative artificial intelligence.

[0549] Generative artificial intelligence analyzes these operation logs and learns the operating procedures for each service.

[0550] The server stores the learning results in a database.

[0551] Step 6: Automatic generation of user manual

[0552] The server uses generative artificial intelligence to generate a user manual based on the operating procedures.

[0553] The manual includes textual instructions, screenshots, and, where necessary, videos of the operating procedures.

[0554] For example, for "How to set up a campaign in an email marketing tool," specific steps such as "Step 1: Click 'New Campaign' from the dashboard" are generated.

[0555] Step 7: Receiving the inquiry

[0556] A user sends an inquiry about a specific service from their device.

[0557] The terminal sends the entered query content to the server.

[0558] Step 8: Analyzing the inquiry content

[0559] The server's generation-based artificial intelligence analyzes the content of the received inquiry.

[0560] Based on the analysis results, the system searches for the information and operating procedures requested by the user and generates appropriate answers.

[0561] Step 9: Submit and view your answer

[0562] The server sends the generated answer back to the terminal.

[0563] The terminal displays the received answer to the user.

[0564] For example, in response to an inquiry such as "I don't know how to set up a campaign," the message displayed might be, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details."

[0565] ---

[0566] This clearly explains the specific processing steps, from initial setup and operation log collection to learning by generative artificial intelligence, automatic generation of user manuals, and handling inquiries, as a continuous flow.

[0567] (Example 1)

[0568] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0569] Currently, many users spend a considerable amount of time and effort understanding the procedures for using software and web services. Furthermore, the high volume of inquiries regarding operation procedures and settings places a significant burden on customer support. Traditional user manuals tend to be outdated, making it difficult to obtain the latest information. Therefore, there is a need for a system that allows users to easily learn operation procedures and quickly resolve problems.

[0570] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0571] In this invention, the server includes means for learning operating procedures using generative artificial intelligence, means for automatically generating a user manual, means for automatically generating and providing answers to user inquiries, means for collecting user operation logs in real time and transmitting them to the server, and means for generating text descriptions, screenshots, and videos of operations based on the learned operating procedures. This allows users to quickly obtain the latest and most accurate operating procedures, and enables more efficient handling of inquiries.

[0572] "Generative artificial intelligence" is a type of artificial intelligence that analyzes user operations and inquiries to generate optimal answers and procedures.

[0573] "Learning operating procedures" is the process of analyzing user operation logs to understand and systematize the steps and flows necessary for those operations.

[0574] A "user manual" is a guideline document that describes the procedures and settings that users should follow when using software or web services.

[0575] An "operation log" is data that records a series of operations performed by a user when using software or web services, in chronological order.

[0576] A "server" is a computer system that provides services to other terminals or systems on a network.

[0577] A "device" is a device that a user directly operates (for example, a personal computer, tablet, or smartphone).

[0578] A "prompt sentence" is an input sentence used to prompt a generative artificial intelligence to produce a specific answer or output.

[0579] "Text instructions" refer to instructions on how to use the device or answers to inquiries presented in written form.

[0580] A "screenshot" is an image captured from a computer screen at a specific point in time.

[0581] A "video" is a recording of operating procedures or settings methods in moving images to visually demonstrate them.

[0582] This invention provides a system that enables the learning of operating procedures using generative artificial intelligence, the automatic generation of user manuals, and the automation of inquiry handling. The system of this invention operates as follows.

[0583] Initial setup and retrieval of service list

[0584] The user starts the system on the terminal. The terminal reads a configuration file (such as config.json) as part of its initial setup. After reading the file, the terminal automatically connects to the server and sends a request to the server to retrieve a list of services. The server returns the service list to the terminal in JSON format, and the terminal displays that data on the screen. For example, when a user starts a new system for the first time, the configuration file is read, the connection to the server is made, and the service list is retrieved.

[0585] User operation log collection

[0586] When a user begins interacting with a specific service, the device starts recording that interaction in real time. The recorded activity log includes click locations, input content, page transitions, and error messages. This data is stored along with timestamps. For example, the user filling out a form and clicking the "Submit" button is recorded in the log.

[0587] Learning phase of generative artificial intelligence

[0588] At regular intervals, the terminal sends collected operation logs to the server. The server analyzes the received operation logs and learns operation procedures using generative artificial intelligence (e.g., OpenAI's GPT-4). Based on the analysis results, it determines whether a particular operation is important and systematizes the operation procedures accordingly. For example, it understands and structures the flow as "Step 1: Log in" and "Step 2: Go to dashboard".

[0589] Automatic generation of user manuals

[0590] The server automatically generates user manuals based on learned operating procedures. Generative artificial intelligence is used to create text explanations, screenshots, and, if necessary, videos of the operating procedures. For example, a manual on setting up campaigns for an email marketing tool would be structured as follows: "Step 1: Click 'New Campaign' from the dashboard" and "Step 2: Enter the campaign name."

[0591] Automating customer support

[0592] When a user makes an inquiry about a specific service, the device sends the user's inquiry to the server. The server's generative artificial intelligence analyzes the inquiry and generates appropriate operating procedures or answers. For example, if a user asks, "I don't know how to set up a new campaign," the server will provide an answer in the form of, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details."

[0593] Example of a prompt

[0594] For example, you can ask a generative AI model the following question:

[0595] "If a user contacts you because they don't know how to set up a new campaign, please provide them with appropriate instructions."

[0596] In response to this prompt, the server's generative artificial intelligence might generate the following answer:

[0597] "Step 1: Click 'New Campaign' from the dashboard. Step 2: Enter a campaign name and click 'Next'. Step 3: Set specific goals on the goal setting screen. Finally, click 'Save'."

[0598] Thus, the system of the present invention learns the user's operating procedures and automatically generates and updates the user manual based on them. Furthermore, it can quickly provide appropriate answers to user inquiries. As a result, the user experience is improved, and the efficiency of handling inquiries is enhanced.

[0599] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0600] Step 1:

[0601] System startup and initial setup

[0602] The user starts the system on the terminal. Specifically, the user double-clicks a desktop icon or selects an app from the Start menu. After startup, the terminal reads a configuration file (such as config.json) and performs initial setup. This configuration file contains server connection information and initial user interface settings. The input is the user's startup action, and the output is the status of the configuration loading completion.

[0603] Step 2:

[0604] Connecting to the server and retrieving a list of services

[0605] After initial setup, the device automatically connects to the server and sends a request to retrieve a list of services. The server receives this request and returns the service list in JSON format. For example, the service list might include "email marketing," "web analytics," and "ad delivery." The input is the connection request to the server, and the output is the service list in JSON format from the server.

[0606] Step 3:

[0607] User operation log collection started.

[0608] When a user selects a specific service and begins an operation, the terminal starts recording an operation log in real time. The recorded operation log includes click locations, input content, page transitions, error messages, and more. Input is the user's operation action, and output is the start status of the operation log.

[0609] Step 4:

[0610] Recording and saving operation logs

[0611] The terminal meticulously records each user action in chronological order. For example, the actions of a user filling out a form and clicking the "Submit" button are saved as logs. Inputs are the user's specific actions, and outputs are the recorded action logs.

[0612] Step 5:

[0613] Sending operation logs

[0614] The terminal sends the collected operation logs to the server in batch processing at regular intervals. For example, if the terminal is configured to send operation logs to the server every minute, batch processing will occur once every 60 seconds. The input is the collected operation logs, and the output is the operation logs sent to the server.

[0615] Step 6:

[0616] Operation log analysis using generative artificial intelligence

[0617] The server analyzes the received operation logs using generative artificial intelligence (e.g., OpenAI's GPT-4). Specifically, it analyzes each operation log to identify important operation procedures and flows. The input is the operation log, and the output is the analysis results.

[0618] Step 7:

[0619] Learning and systematizing operating procedures

[0620] The server learns the operating procedures using generative artificial intelligence based on the analysis results. In this process, it understands the order and conditions under which specific operations are performed and systematizes the operating procedures based on that. For example, specific procedures such as "Step 1: Log in" and "Step 2: Go to the dashboard" are generated. The input is the analysis results, and the output is the learned operating procedures.

[0621] Step 8:

[0622] Automatic generation of user manuals

[0623] The server automatically generates a user manual using generative artificial intelligence based on learned operating procedures. This manual includes text descriptions of the operating procedures, screenshots, and, if necessary, videos. The input is the learned operating procedures, and the output is the generated user manual.

[0624] Step 9:

[0625] User manual provided.

[0626] When a user wants to refer to a manual for a specific service, the terminal retrieves the manual generated from the server and displays it on the screen. The input is the user's manual reference request, and the output is the user manual displayed on the screen.

[0627] Step 10:

[0628] User inquiries accepted.

[0629] When a user makes an inquiry about a specific service, the device sends the inquiry details to the server. For example, a user might inquire, "I don't know how to set up a new campaign." The input is the user's inquiry, and the output is the inquiry data sent to the server.

[0630] Step 11:

[0631] Inquiry content analysis using generative artificial intelligence

[0632] The server's generative artificial intelligence analyzes the received inquiry and generates appropriate operating procedures and answers. For example, it might provide an answer in the form of, "Step 1: Click 'New Campaign' from the dashboard. Step 2: Enter the campaign name." The input is the user's inquiry, and the output is the generated answer.

[0633] Step 12:

[0634] Providing the answer

[0635] The server sends the generated answer back to the terminal, which then displays it to the user. The input is the generated answer, and the output is the answer displayed to the user.

[0636] As described above, the system of the present invention allows users to easily learn the operating procedures, quickly solve problems, and improve the efficiency of handling inquiries.

[0637] (Application Example 1)

[0638] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0639] Many current systems suffer from a lack of user manuals to help users understand operating procedures, and support for inquiries is often time-consuming and cumbersome, thus compromising user convenience. Therefore, there is a need for systems that automatically generate user manuals regarding operating procedures and streamline inquiry handling. Furthermore, it is crucial that the generated manuals always reflect the latest information, but this is difficult to achieve with conventional methods. Therefore, a system that addresses these challenges is necessary.

[0640] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0641] In this invention, the server includes means for learning operating procedures using generative artificial intelligence, means for automatically generating a user manual, means for automatically generating and providing answers to user inquiries, means for collecting and transmitting user operation logs in real time, means for including text explanations, images, and videos in the generated user manual, and means for providing operating procedures and answers through a smartphone application. This enables users to efficiently understand operating procedures and quickly obtain answers to their inquiries.

[0642] "Generative artificial intelligence" is an artificial intelligence technology that can generate and predict natural language based on vast amounts of data.

[0643] An "operating procedure" is a series of actions or steps necessary to accomplish a specific task or function.

[0644] A "user manual" is a document that describes the procedures and precautions that users should take when operating a system or service.

[0645] "Inquiry support" refers to the activity of providing appropriate information and solutions to questions and problems from users.

[0646] A "user operation log" is a series of action logs recorded when a user operates the system.

[0647] "Real-time" refers to operations and data processing being performed instantly.

[0648] "Text instructions" refer to written descriptions of operating procedures and precautions.

[0649] An "image" is a still image used to provide information visually.

[0650] A "video" is a visual representation of movement created by playing multiple images in sequence.

[0651] A "smartphone application" is a computer program that runs on a smartphone.

[0652] This invention provides a system that uses generative artificial intelligence to automate the learning of operating procedures, the automatic generation of user manuals, and the handling of inquiries. Specific embodiments of this system are described below.

[0653] 1. System Configuration

[0654] This system consists of the following main components:

[0655] Terminal: Smartphone application. It is responsible for collecting user operation logs, sending inquiries, and displaying generated user manuals and inquiry responses.

[0656] Server: Performs learning and analysis using generative artificial intelligence. It also saves user operation logs, generates user manuals based on analysis results, and generates appropriate answers for inquiries.

[0657] 2. Hardware and software to be used

[0658] Hardware: Smartphones (iOS, Android compatible), cloud servers (e.g., AWS, Google Cloud, Azure)

[0659] Software: Smartphone apps (frameworks: Flutter, React Native, etc.), server-side apps (frameworks: Node.js, Django, etc.), generative artificial intelligence models (e.g., OpenAI GPT-4, GPT-3)

[0660] 3. Data processing and calculations

[0661] Collection of user operation logs

[0662] Smartphone applications running on a device collect log data in real time as the user interacts with it. For example, they record when a user clicks a button, the text they enter, and information about page transitions. This data is sent to a server using methods such as WebSocket.

[0663] Analysis and learning using generative AI

[0664] On the server, collected operation logs are analyzed and learned from by generative artificial intelligence at regular intervals. The generative AI analyzes the operation logs and patterns and defines operation procedures. The learning results are stored in a database (e.g., MongoDB, MySQL).

[0665] Automatic generation of user manuals

[0666] The server automatically generates user manuals based on operating procedures analyzed and learned using generative AI. The manuals include text explanations, images, and videos as needed. For example, a manual on how to order a product would include steps such as "Step 1: Click 'Order History' from the dashboard" and "Step 2: Select the order and click 'Cancel'," along with corresponding screenshots.

[0667] Automating customer support

[0668] When a user submits an inquiry within the smartphone application, the content is sent to the server. The server's generation AI analyzes the inquiry and generates an appropriate answer. This answer, including operating procedures and countermeasures, is sent back to the user in text format.

[0669] 4. Specific Examples and Prompts

[0670] Specific example:

[0671] If a user asks within the app, "I don't know how to cancel my order," the generative AI will create a prompt message like the following, analyze it, and then provide a response.

[0672] Example of a prompt:

[0673] A user has inquired about how to cancel an order. Please explain the order cancellation procedure. Include the following information: details of each step, necessary screenshots, and links.

[0674] Generated example answer:

[0675] Here's how to cancel your order:

[0676] 1. From the dashboard screen, click "Order History".

[0677] ![Screenshot of Step 1]()

[0678] 2. Select the order you wish to cancel and click "Cancel".

[0679] ![Screenshot of Step 2]()

[0680] 3. A confirmation message will appear; select "Yes".

[0681] ![Screenshot of Step 3]()

[0682] For detailed instructions, please refer to [this link](https: / / support.example.com / cancel-order).

[0683] In this way, the present invention provides a system that enables users to efficiently understand operating procedures and quickly obtain answers to their inquiries.

[0684] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0685] Step 1:

[0686] The device collects user activity logs in real time through a smartphone application. Input includes user actions (clicks, text input, page transitions), and this information is recorded along with a timestamp. Output is generated as activity log data, which is sent to the server using WebSocket.

[0687] Step 2:

[0688] The server saves the received operation log data to the database. The received operation log data is taken as input, and the saving process is performed to the database (e.g., MongoDB, MySQL). A status indicating successful saving is generated as output. This step involves specific actions to format the data according to data integrity and the structure of the destination table.

[0689] Step 3:

[0690] The server periodically retrieves operation log data from the database and uses the generative artificial intelligence model GPT-4 to analyze and train it. The retrieved operation log data is the input, and the generative AI model analyzes this data. As output, patterns of operation procedures as a result of training are generated and stored in the database. Specifically, data preprocessing (denoising, normalization, etc.) is performed, and the analysis results are defined as operation procedures.

[0691] Step 4:

[0692] The server uses generative artificial intelligence to automatically generate user manuals based on stored operating procedures. The input is a set of learned operating procedures, and the generative AI generates text explanations, images, and, if necessary, videos. The user manual is generated as output and stored on the server. Specifically, the operating procedures are converted into text in a sequential manner, and images and videos are captured and edited.

[0693] Step 5:

[0694] When a user submits an inquiry within a smartphone application, the device sends the content to the server. The input is the user's inquiry content, which is sent to the server in text format. The output is the text of the inquiry content reaching the server. Specifically, this includes text input in an input form and clicking the submit button.

[0695] Step 6:

[0696] The server's generative AI analyzes the query content, creates a prompt, and generates an appropriate answer. The input is the text of the query, and a prompt is generated based on it. The generative AI model analyzes this prompt and generates an appropriate answer. The output is the generated answer text. Specifically, prompt generation and subsequent answer generation are performed.

[0697] Step 7:

[0698] The server sends the generated response back to the device, which then displays it to the user. The input is the generated response text, which is sent to the device. The output is the response displayed on the user's smartphone screen. Specifically, text data is transmitted over the network, and the display component handles the screen display.

[0699] These steps enable users to efficiently understand the operating procedures and quickly obtain answers to their inquiries.

[0700] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0701] ---

[0702] This invention is a system that combines learning of operating procedures using generative artificial intelligence, automatic generation of user manuals, automation of inquiry handling, and an emotion engine that recognizes user emotions. This system further streamlines the handling of inquiries regarding the use of specific services and improves the user experience.

[0703] 1. Initial setup and retrieval of service list

[0704] The user starts the system on their terminal. The terminal reads the system's configuration file, then connects to the server to retrieve a list of services provided. This list is sent back to the terminal from the server in JSON format. The terminal then displays the received list of services to the user.

[0705] 2. Collection of user operation logs and sentiment data

[0706] When a user interacts with a specific service in their browser, the device records an operation log in real time. This operation log includes click locations, input content, page transitions, and error messages. The device also incorporates an emotion engine that collects emotional data from the user's facial expressions and voice. This emotional data is also recorded along with the operation log and stored with a timestamp.

[0707] 3. Sending operation logs and sentiment data

[0708] The device sends operation logs and emotion data recorded at regular intervals to the server. The server stores the received data in a database for analysis.

[0709] 4. Learning of Generative Artificial Intelligence

[0710] The server provides stored operation logs and emotion data to a generative artificial intelligence (AI). The AI ​​analyzes this data and learns not only the operation procedures for each service, but also makes adjustments based on the user's emotional state. For example, if the user is confused, it learns to explain the operation procedures more specifically and carefully. The server stores the learning results in a database.

[0711] 5. Automatic generation of user manuals

[0712] After learning the operating procedures for the service, the server automatically generates a user manual using generative artificial intelligence. The generated manual includes text explanations of the operating procedures, screenshots, and, if necessary, videos. It also includes content that is customized to the user's emotions. For example, if the user is nervous, it may include additional advice such as "Please relax and proceed."

[0713] 6. Automation of inquiry handling

[0714] When a user makes an inquiry about a specific service, the device sends the inquiry to the server. The server's generative artificial intelligence analyzes the received inquiry and sentiment data, and based on that, generates the optimal operating procedures and answers. For example, if a user asks, "I don't know how to set up a campaign," the server will consider the user's sentiment state and provide an answer such as, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details. Don't worry, it's easy to set up."

[0715] ---

[0716] This embodiment is expected to further streamline the handling of user inquiries regarding operation methods, and to significantly improve the user experience through the collaboration of generative artificial intelligence and an emotion engine.

[0717] The following describes the processing flow.

[0718] ---

[0719] Step 1: System Startup

[0720] The user starts the system on the terminal.

[0721] The terminal reads the system configuration file and performs an initialization process.

[0722] Step 2: Obtain the list of services

[0723] The device sends a REST API request to the server, requesting a list of the services it provides.

[0724] The server retrieves a list of currently available services from the database and sends it back to the terminal in JSON format.

[0725] The terminal displays a list of received services to the user.

[0726] Step 3: User begins operation

[0727] The user interacts with a specific service through their browser.

[0728] The device records user actions (clicks, input, page transitions, error messages, etc.) in real time.

[0729] Step 4: Collecting user sentiment data

[0730] The device's built-in emotion engine collects emotional data from the user's facial expressions and voice.

[0731] The collected emotional data is recorded along with a timestamp, similar to the operation log.

[0732] Step 5: Sending operation logs and emotion data

[0733] The device sends operation logs and emotion data recorded at regular intervals to the server.

[0734] The server stores the received data in a database for analysis.

[0735] Step 6: Learning Generative Artificial Intelligence

[0736] The server provides stored operation logs and emotion data to the generative artificial intelligence.

[0737] Generative artificial intelligence analyzes this data and learns the operating procedures for each service, as well as making adjustments based on the user's emotional state.

[0738] For example, if a user is confused, the system will learn to explain the operating procedures in more detail and with greater care.

[0739] The server stores the learning results in a database.

[0740] Step 7: User manual generation

[0741] The server uses generative artificial intelligence to generate a user manual based on the operating procedures.

[0742] The generated manual will include textual instructions, screenshots, and, if necessary, videos of the operating procedures.

[0743] It also includes content that is customized to the user's emotions. For example, if the user is feeling nervous, it may include additional advice such as, "Please relax and proceed."

[0744] Step 8: Inquiry received

[0745] A user sends an inquiry about a specific service from their device.

[0746] The terminal sends the entered query content to the server.

[0747] Step 9: Analysis of inquiry content and sentiment data

[0748] The server's generative artificial intelligence analyzes the received inquiry content and sentiment data.

[0749] Generative artificial intelligence generates optimal operating procedures and answers while taking into account the user's emotional state.

[0750] For example, if a user asks, "I don't know how to set up a campaign," the server will respond with something like, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details. Don't worry, it's easy to set up."

[0751] Step 10: Submit and view your answer

[0752] The server sends the generated answer back to the terminal.

[0753] The terminal displays the received answer to the user.

[0754] This allows users to receive appropriate information in a way that is sensitive to their emotions.

[0755] ---

[0756] In this way, the overall processing flow of the system is explained by breaking it down into specific steps.

[0757] (Example 2)

[0758] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0759] In conventional systems, learning user operation procedures and generating manuals were done manually, which was inefficient and made it particularly difficult to respond while considering user emotions. Furthermore, inquiries had to be handled individually, leading to inconsistencies in response speed and quality. This made it difficult to improve the user experience and sometimes resulted in dissatisfaction with the service. Therefore, there was a need for a system that could collect operation logs and emotion data in real time, and then use generative artificial intelligence to learn and automatically generate operation procedures based on this data, thereby improving the user experience.

[0760] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0761] In this invention, the server includes means for starting the system via a user terminal, reading configuration files, and obtaining a list of services; means for collecting and recording user operation logs and sentiment data in real time; and means for sending the collected operation logs and sentiment data to the server at regular intervals and storing them in a database. This makes it possible to learn operating procedures, automatically generate user manuals, and automate inquiry handling.

[0762] 1. A "user terminal" refers to a device, such as a computer or smartphone, used by a user to operate the system.

[0763] 2. A "configuration file" is a file that contains various information necessary for the system to operate, including initial settings and connection information.

[0764] 3. The "Service List" is a list of various services provided by the server, and is a set of information that shows the services available to the user.

[0765] 4. An "operation log" is a record of the operations performed by a user on the system, and includes data such as click locations, input content, page transitions, and error messages.

[0766] 5. "Emotional data" refers to data that indicates the emotional state of a user, analyzed from their facial expressions and voice.

[0767] 6. "Real-time" refers to the processing and recording of data and operations almost simultaneously.

[0768] 7. A "server" is a computer system that provides services over a network, including data storage and analysis.

[0769] 8. A "database" is a system that systematically manages and stores operation logs, sentiment data, and other similar information.

[0770] 9. "Generative artificial intelligence" refers to an AI model that analyzes and learns from provided data to generate new information and procedures.

[0771] 10. "Learning" is the process of analyzing features from data and accumulating information internally that improves future predictions and judgments.

[0772] 11. A "user manual" is a guide that explains the procedures and methods for users to use a system.

[0773] 12. "Inquiry support" is the process of providing answers and support to questions and problems submitted by users.

[0774] 13. "Analysis" is the process of examining data in detail and understanding its meaning and relationships.

[0775] 14. A "procedure" refers to a series of steps or operations necessary to achieve a certain objective.

[0776] This invention provides a system that utilizes a user terminal, a server, and generative artificial intelligence to learn operating procedures, automatically generate user manuals, and automate inquiry handling. This streamlines the handling of user inquiries regarding operation methods and significantly improves the user experience.

[0777] Initial setup and retrieval of service list

[0778] When a user starts the system on their terminal, the terminal first loads a configuration file. This configuration file contains URLs for retrieving connection information and a list of services. Based on the configuration file, the terminal sends an HTTP request to the server to retrieve the service list. The server returns the service list in JSON format, which the terminal parses and displays to the user.

[0779] Specific example:

[0780] The user launches the system in their computer's browser and loads the configuration file (config.json). The terminal sends an HTTP request to the server to retrieve a list of available services. The server returns JSON data in response to the request, containing information such as the name, description, and provider of each service. The terminal parses this data and displays it on the user interface.

[0781] Examples of prompts for a generative AI model:

[0782] "Please explain the procedure for reading the configuration file and retrieving data to obtain a list of services provided."

[0783] Collection of user operation logs and sentiment data

[0784] When a user interacts with a specific service in their browser, the device records real-time logs of their actions, including click locations, input content, page transitions, and error messages. Furthermore, it collects emotional data from the user's facial expressions and voice through an emotion engine. This data is stored along with timestamps.

[0785] Specific example:

[0786] When a user fills out a form on a web system, the device records the buttons clicked and the contents of the entered text fields. An emotion engine is used to collect the user's facial expression data and analyze the user's emotions (e.g., happiness, anxiety, anger).

[0787] Examples of prompts for a generative AI model:

[0788] "Please provide a script for recording user activity logs on a webpage. Also, please explain how to use an emotion engine to extract emotions from a user's facial expressions."

[0789] Sending operation logs and sentiment data

[0790] The collected operation logs and emotion data are sent from the terminal to the server at regular intervals. The server stores the received data in a database.

[0791] Specific example:

[0792] The terminal sends operation logs and sentiment data to the server in batch processing every 10 minutes. The server receives the JSON data and saves it to a MySQL database.

[0793] Examples of prompts for a generative AI model:

[0794] "Please tell me the procedure for periodically sending locally recorded data to a server and saving it to a database."

[0795] Learning of generative artificial intelligence

[0796] The server provides stored operation logs and emotion data to a generative artificial intelligence (AI). The AI ​​analyzes this data and learns the operation procedures for each service. In particular, it also makes adjustments based on the user's emotional state.

[0797] Specific example:

[0798] The server uses stored operation logs and sentiment data to have a generative AI model perform analysis. The model learns to generate more specific explanations for operation procedures that frequently confuse users.

[0799] Examples of prompts for a generative AI model:

[0800] "How can we use operation logs and sentiment data to learn appropriate operating procedures based on the user's level of confusion?"

[0801] Automatic generation of user manuals

[0802] The server uses generative artificial intelligence to automatically generate user manuals based on what it has learned. The generated manuals include textual explanations of operating procedures, screenshots, and, if necessary, videos. Advice tailored to the user's emotions is also added.

[0803] Specific example:

[0804] The server generates a manual for specific operating procedures. The generated manual includes detailed explanations for each step, relevant images, and advice for confused users such as, "Don't worry, just click here."

[0805] Examples of prompts for a generative AI model:

[0806] "Please tell me how to automatically generate a user manual based on operating procedures. Please also include additional advice that takes sentiment data into consideration."

[0807] Automating customer support

[0808] When a user makes an inquiry about a specific service, the device sends the inquiry details to the server. Generative artificial intelligence analyzes this data and generates and provides the optimal operating procedures and answers.

[0809] Specific example:

[0810] When a user inquires that they "don't know how to reset their password," the server's generated AI model analyzes the relevant steps and provides guidance such as, "Go to your account settings page and click the password reset link."

[0811] Examples of prompts for a generative AI model:

[0812] "Please tell me how to automatically generate optimal operating procedures and answers based on user inquiries."

[0813] This system streamlines the handling of user inquiries regarding operation methods and significantly improves the user experience through the integration of generative artificial intelligence and an emotion engine.

[0814] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0815] Step 1:

[0816] The user starts the system on the terminal.

[0817] Specific operation: The user accesses the system URL using a browser on their PC or smartphone and starts the system.

[0818] Input: System URL

[0819] Output: System initial screen

[0820] Step 2:

[0821] The terminal reads the configuration file.

[0822] Specific operation: The terminal opens the configuration file (config.json) in the program directory and retrieves the necessary connection information and API key.

[0823] Input: Configuration file (config.json)

[0824] Output: Connection information, API key

[0825] Step 3:

[0826] The terminal sends an HTTP request to the server to retrieve a list of services.

[0827] Specific operation: The terminal sends a GET request to a URL read from the configuration file, requesting data for the list of services. The server receives the list of services in JSON format.

[0828] Input: Connection information (URL), API key

[0829] Output: List of services in JSON format

[0830] Step 4:

[0831] The terminal analyzes the list of services it has received and displays it to the user.

[0832] Specific operation: The terminal parses the received JSON data and displays the name, description, provider, etc. of each service in the user interface.

[0833] Input: List of services in JSON format

[0834] Output: List of services displayed in the user interface

[0835] Step 5:

[0836] The user interacts with a specific service through their browser.

[0837] Specific actions: Users fill out forms or click buttons to use the service.

[0838] Input: User actions

[0839] Output: System response to user interaction (page transitions, submission of input, etc.)

[0840] Step 6:

[0841] The device records user activity logs in real time.

[0842] Specific operation: Using JavaScript and other client-side technologies, data such as click location, input content, page transitions, and error messages are captured in real time and saved to a log file or memory.

[0843] Input: User actions

[0844] Output: Operation Log

[0845] Step 7:

[0846] The device uses an emotion engine to collect emotional data from the user's facial expressions and voice.

[0847] Specific operation: The system uses a camera and microphone to capture the user's facial expressions and voice, and generates emotional data through an emotion analysis API. For example, it can determine emotions such as happiness, anxiety, and anger.

[0848] Input: User's facial expressions, voice data

[0849] Output: Sentiment data

[0850] Step 8:

[0851] The device sends operation logs and emotion data collected at regular intervals to the server.

[0852] Specific operation: Every 10 minutes, the terminal batch processes and collects operation logs and sentiment data, then sends them to the server via an HTTP POST request.

[0853] Input: Operation log, sentiment data

[0854] Output: Data sent to the server

[0855] Step 9:

[0856] The server saves the received operation logs and sentiment data to a database.

[0857] Specific operation: The server parses the received JSON data and writes it to a database such as MySQL or PostgreSQL.

[0858] Input: Received operation logs, sentiment data

[0859] Output: Data stored in the database

[0860] Step 10:

[0861] The server provides stored operation logs and emotion data to the generative artificial intelligence.

[0862] Specific operation: The server periodically extracts operation logs and sentiment data from the database and feeds them to the generative artificial intelligence.

[0863] Input: Operation log, sentiment data

[0864] Output: Data provided to the generative artificial intelligence.

[0865] Step 11:

[0866] Generative artificial intelligence analyzes the provided data and learns the operating procedures.

[0867] Specific operation: Generative artificial intelligence analyzes user interaction patterns and emotional states to learn the optimal operating procedures for each service. This includes generating detailed guides for operating procedures that frequently confuse users.

[0868] Input: Operation log, sentiment data

[0869] Output: Learned operating procedures

[0870] Step 12:

[0871] The server automatically generates a user manual based on what it has learned using generative artificial intelligence.

[0872] Specific operation: The server automatically generates a manual based on the operating procedures learned by the generative artificial intelligence. The generated manual includes text explanations, screenshots, and videos. It also adds advice tailored to the user's emotions.

[0873] Input: Learned operating procedures

[0874] Output: User Manual

[0875] Step 13:

[0876] A user makes an inquiry about a specific service.

[0877] Specific action: The user enters a question into the inquiry form and clicks the submit button.

[0878] Input: Inquiry details

[0879] Output: Query received by the server

[0880] Step 14:

[0881] The server receives the inquiry, and a generative artificial intelligence analyzes the inquiry content and sentiment data.

[0882] Specific operation: The generative artificial intelligence analyzes the received inquiry content and past sentiment data to generate the optimal operating procedure and answer.

[0883] Input: Inquiry details, sentiment data

[0884] Output: Generated operating procedures and solutions

[0885] Step 15:

[0886] The server provides the generated operating procedures and solutions.

[0887] Specific operation: The server returns the generated answer to the user. For example, in response to an inquiry such as "I don't know how to reset my password," it will return specific instructions such as "Go to the account settings page and click the password reset link."

[0888] Input: Generated operating procedures and answers

[0889] Output: Answers provided to the user

[0890] This system streamlines the handling of user inquiries regarding operation methods, and significantly improves the user experience through the integration of generative artificial intelligence and an emotion engine.

[0891] (Application Example 2)

[0892] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0893] In recent years, many service providers have aimed to improve the efficiency of user support, but traditional systems often failed to adequately address user inquiries about how to use the service. Furthermore, support methods that took user emotions into consideration were not provided, resulting in a lack of improvement in the user experience. This raised concerns that user dissatisfaction would increase, potentially leading to service interruptions or cancellations.

[0894] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0895] In this invention, the server includes means for learning operating procedures using generative artificial intelligence, means for automatically generating a user manual, means for automatically generating and providing answers to user inquiries, and means for recognizing the user's emotions in real time and adjusting the operating procedures and answers based on that state. This makes it possible to respond to user inquiries about how to operate the system efficiently and in a way that is considerate of the user's emotions.

[0896] "Generative artificial intelligence" refers to artificial intelligence that has the ability to learn the user's operating procedures and automatically generate appropriate answers or manuals.

[0897] "Operating procedures" refer to a series of actions or steps that a user must perform when using a particular service or system.

[0898] A "user manual" is a guide containing documents, images, videos, and other materials generated to explain how to use a particular service or system.

[0899] "Inquiry support" refers to the activity of providing appropriate answers and solutions to questions and problems from users.

[0900] An "emotion engine" is a technology that analyzes a user's facial expressions and voice to recognize their emotions in real time.

[0901] An "operation log" is data that records the specific actions and inputs a user makes while operating a system or service.

[0902] A "server" is a computer system that stores operation logs and emotional data, and performs analysis and manual generation using generative artificial intelligence.

[0903] "Text description" refers to information in a written format that explains operating procedures and methods to the user.

[0904] A "screenshot" refers to an image captured from the screen being operated by the user.

[0905] A "video" is video content used to dynamically explain operating procedures and methods to users.

[0906] "Customization" means adjusting or changing the content according to the user's specific situation or feelings.

[0907] Modes for carrying out the invention

[0908] This invention is a system that uses generative artificial intelligence and an emotion engine to automatically learn user procedures and generate user manuals and answers to inquiries. Specific embodiments of this system are described below.

[0909] 1. Initial setup and obtaining a list of services

[0910] When the terminal starts the system, it first performs initial setup. The terminal reads a configuration file and connects to the server to retrieve a list of services provided. This list is sent back to the terminal from the server in JSON format, and the terminal displays the received list of services to the user.

[0911] 2. Collection of user operation logs and sentiment data

[0912] When a user interacts with a specific service, the device records an operation log in real time. This operation log includes click locations, input content, page transitions, and error messages. The device also incorporates an EmotionEngine, which collects emotional data from the user's facial expressions and voice. This emotional data is also recorded along with the operation log and stored with a timestamp.

[0913] 3. Sending operation logs and emotion data

[0914] The device sends operation logs and emotion data recorded at regular intervals to the server. The server stores the received data in a database for analysis.

[0915] 4. Learning of Generative Artificial Intelligence

[0916] The server provides stored operation logs and sentiment data to a generative artificial intelligence (AIAssistant). The generative AI analyzes this data and not only learns the operation procedures for each service, but also makes adjustments based on the user's emotions. For example, if the user is confused, it learns to explain the operation procedures more specifically and carefully. The server stores the learning results in a database.

[0917] 5. Automatic generation of user manuals

[0918] After learning the operating procedures for the service, the server automatically generates a user manual using generative artificial intelligence. The generated manual includes text explanations of the operating procedures, screenshots, and videos. It also includes content that is customized to the user's emotions. For example, if the user is nervous, it may include additional advice such as, "Please relax and proceed."

[0919] 6. Automation of inquiry handling

[0920] When a user makes an inquiry about a specific service, the device sends the inquiry to the server. The server's generative artificial intelligence analyzes the received inquiry and sentiment data, and based on that, generates the optimal operating procedures and answers. For example, if a user makes an inquiry such as "The video streaming keeps stopping," the generative AI analyzes the user's level of confusion using its sentiment engine when providing operating procedures, and generates an answer in the form of "Step 1: Check your Wi-Fi connection. Then, Step 2: Restart the app. Don't worry, it should go back to normal."

[0921] Specific example

[0922] Example of a printed statement:

[0923] Inquiry: Video streaming stops

[0924] Emotion: Confused (60%)

[0925] Answer: Step 1: Check your Wi-Fi connection. Then, Step 2: Restart the app. Don't worry, this should fix it.

[0926] This is expected to significantly improve the user experience by enabling more efficient and user-friendly responses to inquiries about how to use the system.

[0927] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0928] Step 1:

[0929] Initial setup and retrieval of service list

[0930] input:

[0931] Terminal startup and configuration file loading

[0932] process:

[0933] The terminal starts the system and reads the configuration file. Then it connects to the server and retrieves a list of services provided. The server returns the service list in JSON format.

[0934] output:

[0935] A list of acquired services will be displayed on the device.

[0936] Step 2:

[0937] Collection of user operation logs and sentiment data

[0938] input:

[0939] User service operation, facial expressions, voice

[0940] process:

[0941] While a user interacts with a specific service, the device records operation logs (click locations, input content, page transitions, error messages) in real time. Additionally, an EmotionEngine collects emotional data from the user's facial expressions and voice. Both the operation logs and emotional data are stored along with timestamps.

[0942] output:

[0943] Recorded operation logs and emotion data are saved to a file or database.

[0944] Step 3:

[0945] Sending operation logs and sentiment data

[0946] input:

[0947] Recorded operation logs and sentiment data

[0948] process:

[0949] The device sends recorded operation logs and emotion data to the server at regular intervals. The server stores the received data in a database for analysis.

[0950] output:

[0951] Operation logs and sentiment data stored in the server's database.

[0952] Step 4:

[0953] Learning of generative artificial intelligence

[0954] input:

[0955] Operation logs and sentiment data stored on the server

[0956] process:

[0957] The server provides operation logs and sentiment data to a generative artificial intelligence (AIAssistant). The generative AI analyzes this data and learns the operating procedures for each service. If the user is confused, it adjusts the instructions to explain them in a specific and detailed manner.

[0958] output:

[0959] The learning results are stored in the server's database.

[0960] Step 5:

[0961] Automatic generation of user manuals

[0962] input:

[0963] Learning results from generative artificial intelligence

[0964] process:

[0965] Based on learning, the server generates text instructions, screenshots, and videos of the operating procedures. Furthermore, it customizes them according to the user's emotional state. For example, if the user is nervous, it includes additional advice such as, "Please relax and proceed."

[0966] output:

[0967] An automatically generated user manual.

[0968] Step 6:

[0969] Automating customer support

[0970] input:

[0971] User inquiries and sentiment data

[0972] process:

[0973] When a user submits an inquiry, the device sends the inquiry details to the server. Generative artificial intelligence analyzes the inquiry details and sentiment data to generate the optimal operating procedures and answers. For example, if a user submits an inquiry about "video streaming stopping," the sentiment engine analyzes the user's level of frustration and generates an answer in the form of, "Step 1: Check your Wi-Fi connection. Then, Step 2: Restart the app. Don't worry, this should fix it."

[0974] output:

[0975] A prompt message that provides appropriate operating procedures or answers.

[0976] This is expected to significantly improve the user experience by enabling more efficient and user-friendly responses to inquiries about how to use the system.

[0977] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0978] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0979] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0980] [Third Embodiment]

[0981] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0982] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0983] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0984] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0985] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0986] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0987] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0988] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0989] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0990] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0991] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0992] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0993] ---

[0994] This invention provides a system that enables the learning of operating procedures using generative artificial intelligence, the automatic generation of user manuals, and the automation of inquiry handling. The system of this invention operates as follows.

[0995] 1. Initial setup and retrieval of service list

[0996] The user starts the system on their terminal. After the terminal loads the system's initial settings, it connects to the server and retrieves a list of services provided. This list is sent back to the terminal from the server in JSON format.

[0997] 2. Collecting user operation logs

[0998] When a user interacts with a specific service in their browser, the device records an operation log in real time. This operation log includes information such as click locations, input content, page transitions, and error messages, and this information is recorded along with a timestamp.

[0999] 3. Learning Phase of Generative Artificial Intelligence

[1000] The terminal sends recorded operation logs to the server at regular intervals. The server uses generative artificial intelligence to analyze the transmitted operation logs and learn the operation procedures related to the service. Specifically, it understands the order and conditions under which specific operations are performed and uses that information to systematize the operation procedures.

[1001] 4. Automatic generation of user manuals

[1002] After learning the operating procedures for the service, the server automatically generates a user manual using generative artificial intelligence. The generated manual includes text explanations of the operating procedures, screenshots, and, if necessary, videos. For example, a manual on setting up campaigns for an email marketing tool would show steps such as "Step 1: Click 'New Campaign' from the dashboard" and "Step 2: Enter the campaign name."

[1003] 5. Automation of inquiry handling

[1004] When a user makes an inquiry about a specific service, the device sends the inquiry to the server. The server's generative artificial intelligence analyzes the inquiry and generates appropriate operating procedures or answers based on that analysis. The generated answers are sent back to the device, which then displays the answers to the user. For example, if a user asks, "I don't know how to set up a campaign," the server will provide an answer in the form of, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details."

[1005] This embodiment significantly streamlines the handling of inquiries regarding service operation methods. Furthermore, because the latest operating procedures and manuals are always provided through learning and automatic generation by generative artificial intelligence, an improved user experience can be expected.

[1006] The following describes the processing flow.

[1007] ---

[1008] Step 1: System Startup

[1009] The user starts the system on the terminal.

[1010] The terminal reads the system configuration file and performs an initialization process.

[1011] Step 2: Obtain the list of services

[1012] The device sends a REST API request to the server, requesting a list of the services it provides.

[1013] The server retrieves a list of currently available services from the database and sends it back to the terminal in JSON format.

[1014] The terminal displays a list of received services to the user.

[1015] Step 3: Start collecting user operation logs.

[1016] The user interacts with a specific service through their browser.

[1017] The device records user actions (clicks, input, page transitions, error messages, etc.) in real time.

[1018] These operation logs include a timestamp and information about the element that was manipulated.

[1019] Step 4: Sending the operation log

[1020] The device sends operation logs recorded at regular intervals to the server.

[1021] The server saves the received operation logs to a database for analysis.

[1022] Step 5: Learning Generative Artificial Intelligence

[1023] The server provides the stored operation logs to the generative artificial intelligence.

[1024] Generative artificial intelligence analyzes these operation logs and learns the operating procedures for each service.

[1025] The server stores the learning results in a database.

[1026] Step 6: Automatic generation of user manual

[1027] The server uses generative artificial intelligence to generate a user manual based on the operating procedures.

[1028] The manual includes textual instructions, screenshots, and, where necessary, videos of the operating procedures.

[1029] For example, for "How to set up a campaign in an email marketing tool," specific steps such as "Step 1: Click 'New Campaign' from the dashboard" are generated.

[1030] Step 7: Receiving the inquiry

[1031] A user sends an inquiry about a specific service from their device.

[1032] The terminal sends the entered query content to the server.

[1033] Step 8: Analyzing the inquiry content

[1034] The server's generation-based artificial intelligence analyzes the content of the received inquiry.

[1035] Based on the analysis results, the system searches for the information and operating procedures requested by the user and generates appropriate answers.

[1036] Step 9: Submit and view your answer

[1037] The server sends the generated answer back to the terminal.

[1038] The terminal displays the received answer to the user.

[1039] For example, in response to an inquiry such as "I don't know how to set up a campaign," the message displayed might be, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details."

[1040] ---

[1041] This clearly explains the specific processing steps, from initial setup and operation log collection to learning by generative artificial intelligence, automatic generation of user manuals, and handling inquiries, as a continuous flow.

[1042] (Example 1)

[1043] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1044] Currently, many users spend a considerable amount of time and effort understanding the procedures for using software and web services. Furthermore, the high volume of inquiries regarding operation procedures and settings places a significant burden on customer support. Traditional user manuals tend to be outdated, making it difficult to obtain the latest information. Therefore, there is a need for a system that allows users to easily learn operation procedures and quickly resolve problems.

[1045] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1046] In this invention, the server includes means for learning operating procedures using generative artificial intelligence, means for automatically generating a user manual, means for automatically generating and providing answers to user inquiries, means for collecting user operation logs in real time and transmitting them to the server, and means for generating text descriptions, screenshots, and videos of operations based on the learned operating procedures. This allows users to quickly obtain the latest and most accurate operating procedures, and enables more efficient handling of inquiries.

[1047] "Generative artificial intelligence" is a type of artificial intelligence that analyzes user operations and inquiries to generate optimal answers and procedures.

[1048] "Learning operating procedures" is the process of analyzing user operation logs to understand and systematize the steps and flows necessary for those operations.

[1049] A "user manual" is a guideline document that describes the procedures and settings that users should follow when using software or web services.

[1050] An "operation log" is data that records a series of operations performed by a user when using software or web services, in chronological order.

[1051] A "server" is a computer system that provides services to other terminals or systems on a network.

[1052] A "device" is a device that a user directly operates (for example, a personal computer, tablet, or smartphone).

[1053] A "prompt sentence" is an input sentence used to prompt a generative artificial intelligence to produce a specific answer or output.

[1054] "Text instructions" refer to instructions on how to use the device or answers to inquiries presented in written form.

[1055] A "screenshot" is an image captured from a computer screen at a specific point in time.

[1056] A "video" is a recording of operating procedures or settings methods in moving images to visually demonstrate them.

[1057] This invention provides a system that enables the learning of operating procedures using generative artificial intelligence, the automatic generation of user manuals, and the automation of inquiry handling. The system of this invention operates as follows.

[1058] Initial setup and retrieval of service list

[1059] The user starts the system on the terminal. The terminal reads a configuration file (such as config.json) as part of its initial setup. After reading the file, the terminal automatically connects to the server and sends a request to the server to retrieve a list of services. The server returns the service list to the terminal in JSON format, and the terminal displays that data on the screen. For example, when a user starts a new system for the first time, the configuration file is read, the connection to the server is made, and the service list is retrieved.

[1060] User operation log collection

[1061] When a user begins interacting with a specific service, the device starts recording that interaction in real time. The recorded activity log includes click locations, input content, page transitions, and error messages. This data is stored along with timestamps. For example, the user filling out a form and clicking the "Submit" button is recorded in the log.

[1062] Learning phase of generative artificial intelligence

[1063] At regular intervals, the terminal sends collected operation logs to the server. The server analyzes the received operation logs and learns operation procedures using generative artificial intelligence (e.g., OpenAI's GPT-4). Based on the analysis results, it determines whether a particular operation is important and systematizes the operation procedures accordingly. For example, it understands and structures the flow as "Step 1: Log in" and "Step 2: Go to dashboard".

[1064] Automatic generation of user manuals

[1065] The server automatically generates user manuals based on learned operating procedures. Generative artificial intelligence is used to create text explanations, screenshots, and, if necessary, videos of the operating procedures. For example, a manual on setting up campaigns for an email marketing tool would be structured as follows: "Step 1: Click 'New Campaign' from the dashboard" and "Step 2: Enter the campaign name."

[1066] Automating customer support

[1067] When a user makes an inquiry about a specific service, the device sends the user's inquiry to the server. The server's generative artificial intelligence analyzes the inquiry and generates appropriate operating procedures or answers. For example, if a user asks, "I don't know how to set up a new campaign," the server will provide an answer in the form of, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details."

[1068] Example of a prompt

[1069] For example, you can ask a generative AI model the following question:

[1070] "If a user contacts you because they don't know how to set up a new campaign, please provide them with appropriate instructions."

[1071] In response to this prompt, the server's generative artificial intelligence might generate the following answer:

[1072] "Step 1: Click 'New Campaign' from the dashboard. Step 2: Enter a campaign name and click 'Next'. Step 3: Set specific goals on the goal setting screen. Finally, click 'Save'."

[1073] Thus, the system of the present invention learns the user's operating procedures and automatically generates and updates the user manual based on them. Furthermore, it can quickly provide appropriate answers to user inquiries. As a result, the user experience is improved, and the efficiency of handling inquiries is enhanced.

[1074] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1075] Step 1:

[1076] System startup and initial setup

[1077] The user starts the system on the terminal. Specifically, the user double-clicks a desktop icon or selects an app from the Start menu. After startup, the terminal reads a configuration file (such as config.json) and performs initial setup. This configuration file contains server connection information and initial user interface settings. The input is the user's startup action, and the output is the status of the configuration loading completion.

[1078] Step 2:

[1079] Connecting to the server and retrieving a list of services

[1080] After initial setup, the device automatically connects to the server and sends a request to retrieve a list of services. The server receives this request and returns the service list in JSON format. For example, the service list might include "email marketing," "web analytics," and "ad delivery." The input is the connection request to the server, and the output is the service list in JSON format from the server.

[1081] Step 3:

[1082] User operation log collection started.

[1083] When a user selects a specific service and begins an operation, the terminal starts recording an operation log in real time. The recorded operation log includes click locations, input content, page transitions, error messages, and more. Input is the user's operation action, and output is the start status of the operation log.

[1084] Step 4:

[1085] Recording and saving operation logs

[1086] The terminal meticulously records each user action in chronological order. For example, the actions of a user filling out a form and clicking the "Submit" button are saved as logs. Inputs are the user's specific actions, and outputs are the recorded action logs.

[1087] Step 5:

[1088] Sending operation logs

[1089] The terminal sends the collected operation logs to the server in batch processing at regular intervals. For example, if the terminal is configured to send operation logs to the server every minute, batch processing will occur once every 60 seconds. The input is the collected operation logs, and the output is the operation logs sent to the server.

[1090] Step 6:

[1091] Operation log analysis using generative artificial intelligence

[1092] The server analyzes the received operation logs using generative artificial intelligence (e.g., OpenAI's GPT-4). Specifically, it analyzes each operation log to identify important operation procedures and flows. The input is the operation log, and the output is the analysis results.

[1093] Step 7:

[1094] Learning and systematizing operating procedures

[1095] The server learns the operating procedures using generative artificial intelligence based on the analysis results. In this process, it understands the order and conditions under which specific operations are performed and systematizes the operating procedures based on that. For example, specific procedures such as "Step 1: Log in" and "Step 2: Go to the dashboard" are generated. The input is the analysis results, and the output is the learned operating procedures.

[1096] Step 8:

[1097] Automatic generation of user manuals

[1098] The server automatically generates a user manual using generative artificial intelligence based on learned operating procedures. This manual includes text descriptions of the operating procedures, screenshots, and, if necessary, videos. The input is the learned operating procedures, and the output is the generated user manual.

[1099] Step 9:

[1100] User manual provided.

[1101] When a user wants to refer to a manual for a specific service, the terminal retrieves the manual generated from the server and displays it on the screen. The input is the user's manual reference request, and the output is the user manual displayed on the screen.

[1102] Step 10:

[1103] User inquiries accepted.

[1104] When a user makes an inquiry about a specific service, the device sends the inquiry details to the server. For example, a user might inquire, "I don't know how to set up a new campaign." The input is the user's inquiry, and the output is the inquiry data sent to the server.

[1105] Step 11:

[1106] Inquiry content analysis using generative artificial intelligence

[1107] The server's generative artificial intelligence analyzes the received inquiry and generates appropriate operating procedures and answers. For example, it might provide an answer in the form of, "Step 1: Click 'New Campaign' from the dashboard. Step 2: Enter the campaign name." The input is the user's inquiry, and the output is the generated answer.

[1108] Step 12:

[1109] Providing the answer

[1110] The server sends the generated answer back to the terminal, which then displays it to the user. The input is the generated answer, and the output is the answer displayed to the user.

[1111] As described above, the system of the present invention allows users to easily learn the operating procedures, quickly solve problems, and improve the efficiency of handling inquiries.

[1112] (Application Example 1)

[1113] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1114] Many current systems suffer from a lack of user manuals to help users understand operating procedures, and support for inquiries is often time-consuming and cumbersome, thus compromising user convenience. Therefore, there is a need for systems that automatically generate user manuals regarding operating procedures and streamline inquiry handling. Furthermore, it is crucial that the generated manuals always reflect the latest information, but this is difficult to achieve with conventional methods. Therefore, a system that addresses these challenges is necessary.

[1115] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1116] In this invention, the server includes means for learning operating procedures using generative artificial intelligence, means for automatically generating a user manual, means for automatically generating and providing answers to user inquiries, means for collecting and transmitting user operation logs in real time, means for including text explanations, images, and videos in the generated user manual, and means for providing operating procedures and answers through a smartphone application. This enables users to efficiently understand operating procedures and quickly obtain answers to their inquiries.

[1117] "Generative artificial intelligence" is an artificial intelligence technology that can generate and predict natural language based on vast amounts of data.

[1118] An "operating procedure" is a series of actions or steps necessary to accomplish a specific task or function.

[1119] A "user manual" is a document that describes the procedures and precautions that users should take when operating a system or service.

[1120] "Inquiry support" refers to the activity of providing appropriate information and solutions to questions and problems from users.

[1121] A "user operation log" is a series of action logs recorded when a user operates the system.

[1122] "Real-time" refers to operations and data processing being performed instantly.

[1123] "Text instructions" refer to written descriptions of operating procedures and precautions.

[1124] An "image" is a still image used to provide information visually.

[1125] A "video" is a visual representation of movement created by playing multiple images in sequence.

[1126] A "smartphone application" is a computer program that runs on a smartphone.

[1127] This invention provides a system that uses generative artificial intelligence to automate the learning of operating procedures, the automatic generation of user manuals, and the handling of inquiries. Specific embodiments of this system are described below.

[1128] 1. System Configuration

[1129] This system consists of the following main components:

[1130] Terminal: Smartphone application. It is responsible for collecting user operation logs, sending inquiries, and displaying generated user manuals and inquiry responses.

[1131] Server: Performs learning and analysis using generative artificial intelligence. It also saves user operation logs, generates user manuals based on analysis results, and generates appropriate answers for inquiries.

[1132] 2. Hardware and software to be used

[1133] Hardware: Smartphones (iOS, Android compatible), cloud servers (e.g., AWS, Google Cloud, Azure)

[1134] Software: Smartphone apps (frameworks: Flutter, React Native, etc.), server-side apps (frameworks: Node.js, Django, etc.), generative artificial intelligence models (e.g., OpenAI GPT-4, GPT-3)

[1135] 3. Data processing and calculations

[1136] Collection of user operation logs

[1137] Smartphone applications running on a device collect log data in real time as the user interacts with it. For example, they record when a user clicks a button, the text they enter, and information about page transitions. This data is sent to a server using methods such as WebSocket.

[1138] Analysis and learning using generative AI

[1139] On the server, collected operation logs are analyzed and learned from by generative artificial intelligence at regular intervals. The generative AI analyzes the operation logs and patterns and defines operation procedures. The learning results are stored in a database (e.g., MongoDB, MySQL).

[1140] Automatic generation of user manuals

[1141] The server automatically generates user manuals based on operating procedures analyzed and learned using generative AI. The manuals include text explanations, images, and videos as needed. For example, a manual on how to order a product would include steps such as "Step 1: Click 'Order History' from the dashboard" and "Step 2: Select the order and click 'Cancel'," along with corresponding screenshots.

[1142] Automating customer support

[1143] When a user submits an inquiry within the smartphone application, the content is sent to the server. The server's generation AI analyzes the inquiry and generates an appropriate answer. This answer, including operating procedures and countermeasures, is sent back to the user in text format.

[1144] 4. Specific Examples and Prompts

[1145] Specific example:

[1146] If a user asks within the app, "I don't know how to cancel my order," the generative AI will create a prompt message like the following, analyze it, and then provide a response.

[1147] Example of a prompt:

[1148] A user has inquired about how to cancel an order. Please explain the order cancellation procedure. Include the following information: details of each step, necessary screenshots, and links.

[1149] Generated example answer:

[1150] Here's how to cancel your order:

[1151] 1. From the dashboard screen, click "Order History".

[1152] ![Screenshot of Step 1]()

[1153] 2. Select the order you wish to cancel and click "Cancel".

[1154] ![Screenshot of Step 2]()

[1155] 3. A confirmation message will appear; select "Yes".

[1156] ![Screenshot of Step 3]()

[1157] For detailed instructions, please refer to [this link](https: / / support.example.com / cancel-order).

[1158] In this way, the present invention provides a system that enables users to efficiently understand operating procedures and quickly obtain answers to their inquiries.

[1159] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1160] Step 1:

[1161] The device collects user activity logs in real time through a smartphone application. Input includes user actions (clicks, text input, page transitions), and this information is recorded along with a timestamp. Output is generated as activity log data, which is sent to the server using WebSocket.

[1162] Step 2:

[1163] The server saves the received operation log data to the database. The received operation log data is taken as input, and the saving process is performed to the database (e.g., MongoDB, MySQL). A status indicating successful saving is generated as output. This step involves specific actions to format the data according to data integrity and the structure of the destination table.

[1164] Step 3:

[1165] The server periodically retrieves operation log data from the database and uses the generative artificial intelligence model GPT-4 to analyze and train it. The retrieved operation log data is the input, and the generative AI model analyzes this data. As output, patterns of operation procedures as a result of training are generated and stored in the database. Specifically, data preprocessing (denoising, normalization, etc.) is performed, and the analysis results are defined as operation procedures.

[1166] Step 4:

[1167] The server uses generative artificial intelligence to automatically generate user manuals based on stored operating procedures. The input is a set of learned operating procedures, and the generative AI generates text explanations, images, and, if necessary, videos. The user manual is generated as output and stored on the server. Specifically, the operating procedures are converted into text in a sequential manner, and images and videos are captured and edited.

[1168] Step 5:

[1169] When a user submits an inquiry within a smartphone application, the device sends the content to the server. The input is the user's inquiry content, which is sent to the server in text format. The output is the text of the inquiry content reaching the server. Specifically, this includes text input in an input form and clicking the submit button.

[1170] Step 6:

[1171] The server's generative AI analyzes the query content, creates a prompt, and generates an appropriate answer. The input is the text of the query, and a prompt is generated based on it. The generative AI model analyzes this prompt and generates an appropriate answer. The output is the generated answer text. Specifically, prompt generation and subsequent answer generation are performed.

[1172] Step 7:

[1173] The server sends the generated response back to the device, which then displays it to the user. The input is the generated response text, which is sent to the device. The output is the response displayed on the user's smartphone screen. Specifically, text data is transmitted over the network, and the display component handles the screen display.

[1174] These steps enable users to efficiently understand the operating procedures and quickly obtain answers to their inquiries.

[1175] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1176] ---

[1177] This invention is a system that combines learning of operating procedures using generative artificial intelligence, automatic generation of user manuals, automation of inquiry handling, and an emotion engine that recognizes user emotions. This system further streamlines the handling of inquiries regarding the use of specific services and improves the user experience.

[1178] 1. Initial setup and retrieval of service list

[1179] The user starts the system on their terminal. The terminal reads the system's configuration file, then connects to the server to retrieve a list of services provided. This list is sent back to the terminal from the server in JSON format. The terminal then displays the received list of services to the user.

[1180] 2. Collection of user operation logs and sentiment data

[1181] When a user interacts with a specific service in their browser, the device records an operation log in real time. This operation log includes click locations, input content, page transitions, and error messages. The device also incorporates an emotion engine that collects emotional data from the user's facial expressions and voice. This emotional data is also recorded along with the operation log and stored with a timestamp.

[1182] 3. Sending operation logs and sentiment data

[1183] The device sends operation logs and emotion data recorded at regular intervals to the server. The server stores the received data in a database for analysis.

[1184] 4. Learning of Generative Artificial Intelligence

[1185] The server provides stored operation logs and emotion data to a generative artificial intelligence (AI). The AI ​​analyzes this data and learns not only the operation procedures for each service, but also makes adjustments based on the user's emotional state. For example, if the user is confused, it learns to explain the operation procedures more specifically and carefully. The server stores the learning results in a database.

[1186] 5. Automatic generation of user manuals

[1187] After learning the operating procedures for the service, the server automatically generates a user manual using generative artificial intelligence. The generated manual includes text explanations of the operating procedures, screenshots, and, if necessary, videos. It also includes content that is customized to the user's emotions. For example, if the user is nervous, it may include additional advice such as "Please relax and proceed."

[1188] 6. Automation of inquiry handling

[1189] When a user makes an inquiry about a specific service, the device sends the inquiry to the server. The server's generative artificial intelligence analyzes the received inquiry and sentiment data, and based on that, generates the optimal operating procedures and answers. For example, if a user asks, "I don't know how to set up a campaign," the server will consider the user's sentiment state and provide an answer such as, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details. Don't worry, it's easy to set up."

[1190] ---

[1191] This embodiment is expected to further streamline the handling of user inquiries regarding operation methods, and to significantly improve the user experience through the collaboration of generative artificial intelligence and an emotion engine.

[1192] The following describes the processing flow.

[1193] ---

[1194] Step 1: System Startup

[1195] The user starts the system on the terminal.

[1196] The terminal reads the system configuration file and performs an initialization process.

[1197] Step 2: Obtain the list of services

[1198] The device sends a REST API request to the server, requesting a list of the services it provides.

[1199] The server retrieves a list of currently available services from the database and sends it back to the terminal in JSON format.

[1200] The terminal displays a list of received services to the user.

[1201] Step 3: User begins operation

[1202] The user interacts with a specific service through their browser.

[1203] The device records user actions (clicks, input, page transitions, error messages, etc.) in real time.

[1204] Step 4: Collecting user sentiment data

[1205] The device's built-in emotion engine collects emotional data from the user's facial expressions and voice.

[1206] The collected emotional data is recorded along with a timestamp, similar to the operation log.

[1207] Step 5: Sending operation logs and emotion data

[1208] The device sends operation logs and emotion data recorded at regular intervals to the server.

[1209] The server stores the received data in a database for analysis.

[1210] Step 6: Learning Generative Artificial Intelligence

[1211] The server provides stored operation logs and emotion data to the generative artificial intelligence.

[1212] Generative artificial intelligence analyzes this data and learns the operating procedures for each service, as well as making adjustments based on the user's emotional state.

[1213] For example, if a user is confused, the system will learn to explain the operating procedures in more detail and with greater care.

[1214] The server stores the learning results in a database.

[1215] Step 7: User manual generation

[1216] The server uses generative artificial intelligence to generate a user manual based on the operating procedures.

[1217] The generated manual will include textual instructions, screenshots, and, if necessary, videos of the operating procedures.

[1218] It also includes content that is customized to the user's emotions. For example, if the user is feeling nervous, it may include additional advice such as, "Please relax and proceed."

[1219] Step 8: Inquiry received

[1220] A user sends an inquiry about a specific service from their device.

[1221] The terminal sends the entered query content to the server.

[1222] Step 9: Analysis of inquiry content and sentiment data

[1223] The server's generative artificial intelligence analyzes the received inquiry content and sentiment data.

[1224] Generative artificial intelligence generates optimal operating procedures and answers while taking into account the user's emotional state.

[1225] For example, if a user asks, "I don't know how to set up a campaign," the server will respond with something like, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details. Don't worry, it's easy to set up."

[1226] Step 10: Submit and view your answer

[1227] The server sends the generated answer back to the terminal.

[1228] The terminal displays the received answer to the user.

[1229] This allows users to receive appropriate information in a way that is sensitive to their emotions.

[1230] ---

[1231] In this way, the overall processing flow of the system is explained by breaking it down into specific steps.

[1232] (Example 2)

[1233] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1234] In conventional systems, learning user operation procedures and generating manuals were done manually, which was inefficient and made it particularly difficult to respond while considering user emotions. Furthermore, inquiries had to be handled individually, leading to inconsistencies in response speed and quality. This made it difficult to improve the user experience and sometimes resulted in dissatisfaction with the service. Therefore, there was a need for a system that could collect operation logs and emotion data in real time, and then use generative artificial intelligence to learn and automatically generate operation procedures based on this data, thereby improving the user experience.

[1235] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1236] In this invention, the server includes means for starting the system via a user terminal, reading configuration files, and obtaining a list of services; means for collecting and recording user operation logs and sentiment data in real time; and means for sending the collected operation logs and sentiment data to the server at regular intervals and storing them in a database. This makes it possible to learn operating procedures, automatically generate user manuals, and automate inquiry handling.

[1237] 1. A "user terminal" refers to a device, such as a computer or smartphone, used by a user to operate the system.

[1238] 2. A "configuration file" is a file that contains various information necessary for the system to operate, including initial settings and connection information.

[1239] 3. The "Service List" is a list of various services provided by the server, and is a set of information that shows the services available to the user.

[1240] 4. An "operation log" is a record of the operations performed by a user on the system, and includes data such as click locations, input content, page transitions, and error messages.

[1241] 5. "Emotional data" refers to data that indicates the emotional state of a user, analyzed from their facial expressions and voice.

[1242] 6. "Real-time" refers to the processing and recording of data and operations almost simultaneously.

[1243] 7. A "server" is a computer system that provides services over a network, including data storage and analysis.

[1244] 8. A "database" is a system that systematically manages and stores operation logs, sentiment data, and other similar information.

[1245] 9. "Generative artificial intelligence" refers to an AI model that analyzes and learns from provided data to generate new information and procedures.

[1246] 10. "Learning" is the process of analyzing features from data and accumulating information internally that improves future predictions and judgments.

[1247] 11. A "user manual" is a guide that explains the procedures and methods for users to use a system.

[1248] 12. "Inquiry support" is the process of providing answers and support to questions and problems submitted by users.

[1249] 13. "Analysis" is the process of examining data in detail and understanding its meaning and relationships.

[1250] 14. A "procedure" refers to a series of steps or operations necessary to achieve a certain objective.

[1251] This invention provides a system that utilizes a user terminal, a server, and generative artificial intelligence to learn operating procedures, automatically generate user manuals, and automate inquiry handling. This streamlines the handling of user inquiries regarding operation methods and significantly improves the user experience.

[1252] Initial setup and retrieval of service list

[1253] When a user starts the system on their terminal, the terminal first loads a configuration file. This configuration file contains URLs for retrieving connection information and a list of services. Based on the configuration file, the terminal sends an HTTP request to the server to retrieve the service list. The server returns the service list in JSON format, which the terminal parses and displays to the user.

[1254] Specific example:

[1255] The user launches the system in their computer's browser and loads the configuration file (config.json). The terminal sends an HTTP request to the server to retrieve a list of available services. The server returns JSON data in response to the request, containing information such as the name, description, and provider of each service. The terminal parses this data and displays it on the user interface.

[1256] Examples of prompts for a generative AI model:

[1257] "Please explain the procedure for reading the configuration file and retrieving data to obtain a list of services provided."

[1258] Collection of user operation logs and sentiment data

[1259] When a user interacts with a specific service in their browser, the device records real-time logs of their actions, including click locations, input content, page transitions, and error messages. Furthermore, it collects emotional data from the user's facial expressions and voice through an emotion engine. This data is stored along with timestamps.

[1260] Specific example:

[1261] When a user fills out a form on a web system, the device records the buttons clicked and the contents of the entered text fields. An emotion engine is used to collect the user's facial expression data and analyze the user's emotions (e.g., happiness, anxiety, anger).

[1262] Examples of prompts for a generative AI model:

[1263] "Please provide a script for recording user activity logs on a webpage. Also, please explain how to use an emotion engine to extract emotions from a user's facial expressions."

[1264] Sending operation logs and sentiment data

[1265] The collected operation logs and emotion data are sent from the terminal to the server at regular intervals. The server stores the received data in a database.

[1266] Specific example:

[1267] The terminal sends operation logs and sentiment data to the server in batch processing every 10 minutes. The server receives the JSON data and saves it to a MySQL database.

[1268] Examples of prompts for a generative AI model:

[1269] "Please tell me the procedure for periodically sending locally recorded data to a server and saving it to a database."

[1270] Learning of generative artificial intelligence

[1271] The server provides stored operation logs and emotion data to a generative artificial intelligence (AI). The AI ​​analyzes this data and learns the operation procedures for each service. In particular, it also makes adjustments based on the user's emotional state.

[1272] Specific example:

[1273] The server uses stored operation logs and sentiment data to have a generative AI model perform analysis. The model learns to generate more specific explanations for operation procedures that frequently confuse users.

[1274] Examples of prompts for a generative AI model:

[1275] "How can we use operation logs and sentiment data to learn appropriate operating procedures based on the user's level of confusion?"

[1276] Automatic generation of user manuals

[1277] The server uses generative artificial intelligence to automatically generate user manuals based on what it has learned. The generated manuals include textual explanations of operating procedures, screenshots, and, if necessary, videos. Advice tailored to the user's emotions is also added.

[1278] Specific example:

[1279] The server generates a manual for specific operating procedures. The generated manual includes detailed explanations for each step, relevant images, and advice for confused users such as, "Don't worry, just click here."

[1280] Examples of prompts for a generative AI model:

[1281] "Please tell me how to automatically generate a user manual based on operating procedures. Please also include additional advice that takes sentiment data into consideration."

[1282] Automating customer support

[1283] When a user makes an inquiry about a specific service, the device sends the inquiry details to the server. Generative artificial intelligence analyzes this data and generates and provides the optimal operating procedures and answers.

[1284] Specific example:

[1285] When a user inquires that they "don't know how to reset their password," the server's generated AI model analyzes the relevant steps and provides guidance such as, "Go to your account settings page and click the password reset link."

[1286] Examples of prompts for a generative AI model:

[1287] "Please tell me how to automatically generate optimal operating procedures and answers based on user inquiries."

[1288] This system streamlines the handling of user inquiries regarding operation methods and significantly improves the user experience through the integration of generative artificial intelligence and an emotion engine.

[1289] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1290] Step 1:

[1291] The user starts the system on the terminal.

[1292] Specific operation: The user accesses the system URL using a browser on their PC or smartphone and starts the system.

[1293] Input: System URL

[1294] Output: System initial screen

[1295] Step 2:

[1296] The terminal reads the configuration file.

[1297] Specific operation: The terminal opens the configuration file (config.json) in the program directory and retrieves the necessary connection information and API key.

[1298] Input: Configuration file (config.json)

[1299] Output: Connection information, API key

[1300] Step 3:

[1301] The terminal sends an HTTP request to the server to retrieve a list of services.

[1302] Specific operation: The terminal sends a GET request to a URL read from the configuration file, requesting data for the list of services. The server receives the list of services in JSON format.

[1303] Input: Connection information (URL), API key

[1304] Output: List of services in JSON format

[1305] Step 4:

[1306] The terminal analyzes the list of services it has received and displays it to the user.

[1307] Specific operation: The terminal parses the received JSON data and displays the name, description, provider, etc. of each service in the user interface.

[1308] Input: List of services in JSON format

[1309] Output: List of services displayed in the user interface

[1310] Step 5:

[1311] The user interacts with a specific service through their browser.

[1312] Specific actions: Users fill out forms or click buttons to use the service.

[1313] Input: User actions

[1314] Output: System response to user interaction (page transitions, submission of input, etc.)

[1315] Step 6:

[1316] The device records user activity logs in real time.

[1317] Specific operation: Using JavaScript and other client-side technologies, data such as click location, input content, page transitions, and error messages are captured in real time and saved to a log file or memory.

[1318] Input: User actions

[1319] Output: Operation Log

[1320] Step 7:

[1321] The device uses an emotion engine to collect emotional data from the user's facial expressions and voice.

[1322] Specific operation: The system uses a camera and microphone to capture the user's facial expressions and voice, and generates emotional data through an emotion analysis API. For example, it can determine emotions such as happiness, anxiety, and anger.

[1323] Input: User's facial expressions, voice data

[1324] Output: Sentiment data

[1325] Step 8:

[1326] The device sends operation logs and emotion data collected at regular intervals to the server.

[1327] Specific operation: Every 10 minutes, the terminal batch processes and collects operation logs and sentiment data, then sends them to the server via an HTTP POST request.

[1328] Input: Operation log, sentiment data

[1329] Output: Data sent to the server

[1330] Step 9:

[1331] The server saves the received operation logs and sentiment data to a database.

[1332] Specific operation: The server parses the received JSON data and writes it to a database such as MySQL or PostgreSQL.

[1333] Input: Received operation logs, sentiment data

[1334] Output: Data stored in the database

[1335] Step 10:

[1336] The server provides stored operation logs and emotion data to the generative artificial intelligence.

[1337] Specific operation: The server periodically extracts operation logs and sentiment data from the database and feeds them to the generative artificial intelligence.

[1338] Input: Operation log, sentiment data

[1339] Output: Data provided to the generative artificial intelligence.

[1340] Step 11:

[1341] Generative artificial intelligence analyzes the provided data and learns the operating procedures.

[1342] Specific operation: Generative artificial intelligence analyzes user interaction patterns and emotional states to learn the optimal operating procedures for each service. This includes generating detailed guides for operating procedures that frequently confuse users.

[1343] Input: Operation log, sentiment data

[1344] Output: Learned operating procedures

[1345] Step 12:

[1346] The server automatically generates a user manual based on what it has learned using generative artificial intelligence.

[1347] Specific operation: The server automatically generates a manual based on the operating procedures learned by the generative artificial intelligence. The generated manual includes text explanations, screenshots, and videos. It also adds advice tailored to the user's emotions.

[1348] Input: Learned operating procedures

[1349] Output: User Manual

[1350] Step 13:

[1351] A user makes an inquiry about a specific service.

[1352] Specific action: The user enters a question into the inquiry form and clicks the submit button.

[1353] Input: Inquiry details

[1354] Output: Query received by the server

[1355] Step 14:

[1356] The server receives the inquiry, and a generative artificial intelligence analyzes the inquiry content and sentiment data.

[1357] Specific operation: The generative artificial intelligence analyzes the received inquiry content and past sentiment data to generate the optimal operating procedure and answer.

[1358] Input: Inquiry details, sentiment data

[1359] Output: Generated operating procedures and solutions

[1360] Step 15:

[1361] The server provides the generated operating procedures and solutions.

[1362] Specific operation: The server returns the generated answer to the user. For example, in response to an inquiry such as "I don't know how to reset my password," it will return specific instructions such as "Go to the account settings page and click the password reset link."

[1363] Input: Generated operating procedures and answers

[1364] Output: Answers provided to the user

[1365] This system streamlines the handling of user inquiries regarding operation methods, and significantly improves the user experience through the integration of generative artificial intelligence and an emotion engine.

[1366] (Application Example 2)

[1367] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1368] In recent years, many service providers have aimed to improve the efficiency of user support, but traditional systems often failed to adequately address user inquiries about how to use the service. Furthermore, support methods that took user emotions into consideration were not provided, resulting in a lack of improvement in the user experience. This raised concerns that user dissatisfaction would increase, potentially leading to service interruptions or cancellations.

[1369] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1370] In this invention, the server includes means for learning operating procedures using generative artificial intelligence, means for automatically generating a user manual, means for automatically generating and providing answers to user inquiries, and means for recognizing the user's emotions in real time and adjusting the operating procedures and answers based on that state. This makes it possible to respond to user inquiries about how to operate the system efficiently and in a way that is considerate of the user's emotions.

[1371] "Generative artificial intelligence" refers to artificial intelligence that has the ability to learn the user's operating procedures and automatically generate appropriate answers or manuals.

[1372] "Operating procedures" refer to a series of actions or steps that a user must perform when using a particular service or system.

[1373] A "user manual" is a guide containing documents, images, videos, and other materials generated to explain how to use a particular service or system.

[1374] "Inquiry support" refers to the activity of providing appropriate answers and solutions to questions and problems from users.

[1375] An "emotion engine" is a technology that analyzes a user's facial expressions and voice to recognize their emotions in real time.

[1376] An "operation log" is data that records the specific actions and inputs a user makes while operating a system or service.

[1377] A "server" is a computer system that stores operation logs and emotional data, and performs analysis and manual generation using generative artificial intelligence.

[1378] "Text description" refers to information in a written format that explains operating procedures and methods to the user.

[1379] A "screenshot" refers to an image captured from the screen being operated by the user.

[1380] A "video" is video content used to dynamically explain operating procedures and methods to users.

[1381] "Customization" means adjusting or changing the content according to the user's specific situation or feelings.

[1382] Modes for carrying out the invention

[1383] This invention is a system that uses generative artificial intelligence and an emotion engine to automatically learn user procedures and generate user manuals and answers to inquiries. Specific embodiments of this system are described below.

[1384] 1. Initial setup and obtaining a list of services

[1385] When the terminal starts the system, it first performs initial setup. The terminal reads a configuration file and connects to the server to retrieve a list of services provided. This list is sent back to the terminal from the server in JSON format, and the terminal displays the received list of services to the user.

[1386] 2. Collection of user operation logs and sentiment data

[1387] When a user interacts with a specific service, the device records an operation log in real time. This operation log includes click locations, input content, page transitions, and error messages. The device also incorporates an EmotionEngine, which collects emotional data from the user's facial expressions and voice. This emotional data is also recorded along with the operation log and stored with a timestamp.

[1388] 3. Sending operation logs and emotion data

[1389] The device sends operation logs and emotion data recorded at regular intervals to the server. The server stores the received data in a database for analysis.

[1390] 4. Learning of Generative Artificial Intelligence

[1391] The server provides stored operation logs and sentiment data to a generative artificial intelligence (AIAssistant). The generative AI analyzes this data and not only learns the operation procedures for each service, but also makes adjustments based on the user's emotions. For example, if the user is confused, it learns to explain the operation procedures more specifically and carefully. The server stores the learning results in a database.

[1392] 5. Automatic generation of user manuals

[1393] After learning the operating procedures for the service, the server automatically generates a user manual using generative artificial intelligence. The generated manual includes text explanations of the operating procedures, screenshots, and videos. It also includes content that is customized to the user's emotions. For example, if the user is nervous, it may include additional advice such as, "Please relax and proceed."

[1394] 6. Automation of inquiry handling

[1395] When a user makes an inquiry about a specific service, the device sends the inquiry to the server. The server's generative artificial intelligence analyzes the received inquiry and sentiment data, and based on that, generates the optimal operating procedures and answers. For example, if a user makes an inquiry such as "The video streaming keeps stopping," the generative AI analyzes the user's level of confusion using its sentiment engine when providing operating procedures, and generates an answer in the form of "Step 1: Check your Wi-Fi connection. Then, Step 2: Restart the app. Don't worry, it should go back to normal."

[1396] Specific example

[1397] Example of a printed statement:

[1398] Inquiry: Video streaming stops

[1399] Emotion: Confused (60%)

[1400] Answer: Step 1: Check your Wi-Fi connection. Then, Step 2: Restart the app. Don't worry, this should fix it.

[1401] This is expected to significantly improve the user experience by enabling more efficient and user-friendly responses to inquiries about how to use the system.

[1402] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1403] Step 1:

[1404] Initial setup and retrieval of service list

[1405] input:

[1406] Terminal startup and configuration file loading

[1407] process:

[1408] The terminal starts the system and reads the configuration file. Then it connects to the server and retrieves a list of services provided. The server returns the service list in JSON format.

[1409] output:

[1410] A list of acquired services will be displayed on the device.

[1411] Step 2:

[1412] Collection of user operation logs and sentiment data

[1413] input:

[1414] User service operation, facial expressions, voice

[1415] process:

[1416] While a user interacts with a specific service, the device records operation logs (click locations, input content, page transitions, error messages) in real time. Additionally, an EmotionEngine collects emotional data from the user's facial expressions and voice. Both the operation logs and emotional data are stored along with timestamps.

[1417] output:

[1418] Recorded operation logs and emotion data are saved to a file or database.

[1419] Step 3:

[1420] Sending operation logs and sentiment data

[1421] input:

[1422] Recorded operation logs and sentiment data

[1423] process:

[1424] The device sends recorded operation logs and emotion data to the server at regular intervals. The server stores the received data in a database for analysis.

[1425] output:

[1426] Operation logs and sentiment data stored in the server's database.

[1427] Step 4:

[1428] Learning of generative artificial intelligence

[1429] input:

[1430] Operation logs and sentiment data stored on the server

[1431] process:

[1432] The server provides operation logs and sentiment data to a generative artificial intelligence (AIAssistant). The generative AI analyzes this data and learns the operating procedures for each service. If the user is confused, it adjusts the instructions to explain them in a specific and detailed manner.

[1433] output:

[1434] The learning results are stored in the server's database.

[1435] Step 5:

[1436] Automatic generation of user manuals

[1437] input:

[1438] Learning results from generative artificial intelligence

[1439] process:

[1440] Based on learning, the server generates text instructions, screenshots, and videos of the operating procedures. Furthermore, it customizes them according to the user's emotional state. For example, if the user is nervous, it includes additional advice such as, "Please relax and proceed."

[1441] output:

[1442] An automatically generated user manual.

[1443] Step 6:

[1444] Automating customer support

[1445] input:

[1446] User inquiries and sentiment data

[1447] process:

[1448] When a user submits an inquiry, the device sends the inquiry details to the server. Generative artificial intelligence analyzes the inquiry details and sentiment data to generate the optimal operating procedures and answers. For example, if a user submits an inquiry about "video streaming stopping," the sentiment engine analyzes the user's level of frustration and generates an answer in the form of, "Step 1: Check your Wi-Fi connection. Then, Step 2: Restart the app. Don't worry, this should fix it."

[1449] output:

[1450] A prompt message that provides appropriate operating procedures or answers.

[1451] This is expected to significantly improve the user experience by enabling more efficient and user-friendly responses to inquiries about how to use the system.

[1452] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1453] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1454] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1455] [Fourth Embodiment]

[1456] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1457] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1458] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1459] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1460] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1461] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1462] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1463] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1464] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1465] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1466] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1467] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1468] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1469] ---

[1470] This invention provides a system that enables the learning of operating procedures using generative artificial intelligence, the automatic generation of user manuals, and the automation of inquiry handling. The system of this invention operates as follows.

[1471] 1. Initial setup and retrieval of service list

[1472] The user starts the system on their terminal. After the terminal loads the system's initial settings, it connects to the server and retrieves a list of services provided. This list is sent back to the terminal from the server in JSON format.

[1473] 2. Collecting user operation logs

[1474] When a user interacts with a specific service in their browser, the device records an operation log in real time. This operation log includes information such as click locations, input content, page transitions, and error messages, and this information is recorded along with a timestamp.

[1475] 3. Learning Phase of Generative Artificial Intelligence

[1476] The terminal sends recorded operation logs to the server at regular intervals. The server uses generative artificial intelligence to analyze the transmitted operation logs and learn the operation procedures related to the service. Specifically, it understands the order and conditions under which specific operations are performed and uses that information to systematize the operation procedures.

[1477] 4. Automatic generation of user manuals

[1478] After learning the operating procedures for the service, the server automatically generates a user manual using generative artificial intelligence. The generated manual includes text explanations of the operating procedures, screenshots, and, if necessary, videos. For example, a manual on setting up campaigns for an email marketing tool would show steps such as "Step 1: Click 'New Campaign' from the dashboard" and "Step 2: Enter the campaign name."

[1479] 5. Automation of inquiry handling

[1480] When a user makes an inquiry about a specific service, the device sends the inquiry to the server. The server's generative artificial intelligence analyzes the inquiry and generates appropriate operating procedures or answers based on that analysis. The generated answers are sent back to the device, which then displays the answers to the user. For example, if a user asks, "I don't know how to set up a campaign," the server will provide an answer in the form of, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details."

[1481] This embodiment significantly streamlines the handling of inquiries regarding service operation methods. Furthermore, because the latest operating procedures and manuals are always provided through learning and automatic generation by generative artificial intelligence, an improved user experience can be expected.

[1482] The following describes the processing flow.

[1483] ---

[1484] Step 1: System Startup

[1485] The user starts the system on the terminal.

[1486] The terminal reads the system configuration file and performs an initialization process.

[1487] Step 2: Obtain the list of services

[1488] The device sends a REST API request to the server, requesting a list of the services it provides.

[1489] The server retrieves a list of currently available services from the database and sends it back to the terminal in JSON format.

[1490] The terminal displays a list of received services to the user.

[1491] Step 3: Start collecting user operation logs.

[1492] The user interacts with a specific service through their browser.

[1493] The device records user actions (clicks, input, page transitions, error messages, etc.) in real time.

[1494] These operation logs include a timestamp and information about the element that was manipulated.

[1495] Step 4: Sending the operation log

[1496] The device sends operation logs recorded at regular intervals to the server.

[1497] The server saves the received operation logs to a database for analysis.

[1498] Step 5: Learning Generative Artificial Intelligence

[1499] The server provides the stored operation logs to the generative artificial intelligence.

[1500] Generative artificial intelligence analyzes these operation logs and learns the operating procedures for each service.

[1501] The server stores the learning results in a database.

[1502] Step 6: Automatic generation of user manual

[1503] The server uses generative artificial intelligence to generate a user manual based on the operating procedures.

[1504] The manual includes textual instructions, screenshots, and, where necessary, videos of the operating procedures.

[1505] For example, for "How to set up a campaign in an email marketing tool," specific steps such as "Step 1: Click 'New Campaign' from the dashboard" are generated.

[1506] Step 7: Receiving the inquiry

[1507] A user sends an inquiry about a specific service from their device.

[1508] The terminal sends the entered query content to the server.

[1509] Step 8: Analyzing the inquiry content

[1510] The server's generation-based artificial intelligence analyzes the content of the received inquiry.

[1511] Based on the analysis results, the system searches for the information and operating procedures requested by the user and generates appropriate answers.

[1512] Step 9: Submit and view your answer

[1513] The server sends the generated answer back to the terminal.

[1514] The terminal displays the received answer to the user.

[1515] For example, in response to an inquiry such as "I don't know how to set up a campaign," the message displayed might be, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details."

[1516] ---

[1517] This clearly explains the specific processing steps, from initial setup and operation log collection to learning by generative artificial intelligence, automatic generation of user manuals, and handling inquiries, as a continuous flow.

[1518] (Example 1)

[1519] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1520] Currently, many users spend a considerable amount of time and effort understanding the procedures for using software and web services. Furthermore, the high volume of inquiries regarding operation procedures and settings places a significant burden on customer support. Traditional user manuals tend to be outdated, making it difficult to obtain the latest information. Therefore, there is a need for a system that allows users to easily learn operation procedures and quickly resolve problems.

[1521] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1522] In this invention, the server includes means for learning operating procedures using generative artificial intelligence, means for automatically generating a user manual, means for automatically generating and providing answers to user inquiries, means for collecting user operation logs in real time and transmitting them to the server, and means for generating text descriptions, screenshots, and videos of operations based on the learned operating procedures. This allows users to quickly obtain the latest and most accurate operating procedures, and enables more efficient handling of inquiries.

[1523] "Generative artificial intelligence" is a type of artificial intelligence that analyzes user operations and inquiries to generate optimal answers and procedures.

[1524] "Learning operating procedures" is the process of analyzing user operation logs to understand and systematize the steps and flows necessary for those operations.

[1525] A "user manual" is a guideline document that describes the procedures and settings that users should follow when using software or web services.

[1526] An "operation log" is data that records a series of operations performed by a user when using software or web services, in chronological order.

[1527] A "server" is a computer system that provides services to other terminals or systems on a network.

[1528] A "device" is a device that a user directly operates (for example, a personal computer, tablet, or smartphone).

[1529] A "prompt sentence" is an input sentence used to prompt a generative artificial intelligence to produce a specific answer or output.

[1530] "Text instructions" refer to instructions on how to use the device or answers to inquiries presented in written form.

[1531] A "screenshot" is an image captured from a computer screen at a specific point in time.

[1532] A "video" is a recording of operating procedures or settings methods in moving images to visually demonstrate them.

[1533] This invention provides a system that enables the learning of operating procedures using generative artificial intelligence, the automatic generation of user manuals, and the automation of inquiry handling. The system of this invention operates as follows.

[1534] Initial setup and retrieval of service list

[1535] The user starts the system on the terminal. The terminal reads a configuration file (such as config.json) as part of its initial setup. After reading the file, the terminal automatically connects to the server and sends a request to the server to retrieve a list of services. The server returns the service list to the terminal in JSON format, and the terminal displays that data on the screen. For example, when a user starts a new system for the first time, the configuration file is read, the connection to the server is made, and the service list is retrieved.

[1536] User operation log collection

[1537] When a user begins interacting with a specific service, the device starts recording that interaction in real time. The recorded activity log includes click locations, input content, page transitions, and error messages. This data is stored along with timestamps. For example, the user filling out a form and clicking the "Submit" button is recorded in the log.

[1538] Learning phase of generative artificial intelligence

[1539] At regular intervals, the terminal sends collected operation logs to the server. The server analyzes the received operation logs and learns operation procedures using generative artificial intelligence (e.g., OpenAI's GPT-4). Based on the analysis results, it determines whether a particular operation is important and systematizes the operation procedures accordingly. For example, it understands and structures the flow as "Step 1: Log in" and "Step 2: Go to dashboard".

[1540] Automatic generation of user manuals

[1541] The server automatically generates user manuals based on learned operating procedures. Generative artificial intelligence is used to create text explanations, screenshots, and, if necessary, videos of the operating procedures. For example, a manual on setting up campaigns for an email marketing tool would be structured as follows: "Step 1: Click 'New Campaign' from the dashboard" and "Step 2: Enter the campaign name."

[1542] Automating customer support

[1543] When a user makes an inquiry about a specific service, the device sends the user's inquiry to the server. The server's generative artificial intelligence analyzes the inquiry and generates appropriate operating procedures or answers. For example, if a user asks, "I don't know how to set up a new campaign," the server will provide an answer in the form of, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details."

[1544] Example of a prompt

[1545] For example, you can ask a generative AI model the following question:

[1546] "If a user contacts you because they don't know how to set up a new campaign, please provide them with appropriate instructions."

[1547] In response to this prompt, the server's generative artificial intelligence might generate the following answer:

[1548] "Step 1: Click 'New Campaign' from the dashboard. Step 2: Enter a campaign name and click 'Next'. Step 3: Set specific goals on the goal setting screen. Finally, click 'Save'."

[1549] Thus, the system of the present invention learns the user's operating procedures and automatically generates and updates the user manual based on them. Furthermore, it can quickly provide appropriate answers to user inquiries. As a result, the user experience is improved, and the efficiency of handling inquiries is enhanced.

[1550] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1551] Step 1:

[1552] System startup and initial setup

[1553] The user starts the system on the terminal. Specifically, the user double-clicks a desktop icon or selects an app from the Start menu. After startup, the terminal reads a configuration file (such as config.json) and performs initial setup. This configuration file contains server connection information and initial user interface settings. The input is the user's startup action, and the output is the status of the configuration loading completion.

[1554] Step 2:

[1555] Connecting to the server and retrieving a list of services

[1556] After initial setup, the device automatically connects to the server and sends a request to retrieve a list of services. The server receives this request and returns the service list in JSON format. For example, the service list might include "email marketing," "web analytics," and "ad delivery." The input is the connection request to the server, and the output is the service list in JSON format from the server.

[1557] Step 3:

[1558] User operation log collection started.

[1559] When a user selects a specific service and begins an operation, the terminal starts recording an operation log in real time. The recorded operation log includes click locations, input content, page transitions, error messages, and more. Input is the user's operation action, and output is the start status of the operation log.

[1560] Step 4:

[1561] Recording and saving operation logs

[1562] The terminal meticulously records each user action in chronological order. For example, the actions of a user filling out a form and clicking the "Submit" button are saved as logs. Inputs are the user's specific actions, and outputs are the recorded action logs.

[1563] Step 5:

[1564] Sending operation logs

[1565] The terminal sends the collected operation logs to the server in batch processing at regular intervals. For example, if the terminal is configured to send operation logs to the server every minute, batch processing will occur once every 60 seconds. The input is the collected operation logs, and the output is the operation logs sent to the server.

[1566] Step 6:

[1567] Operation log analysis using generative artificial intelligence

[1568] The server analyzes the received operation logs using generative artificial intelligence (e.g., OpenAI's GPT-4). Specifically, it analyzes each operation log to identify important operation procedures and flows. The input is the operation log, and the output is the analysis results.

[1569] Step 7:

[1570] Learning and systematizing operating procedures

[1571] The server learns the operating procedures using generative artificial intelligence based on the analysis results. In this process, it understands the order and conditions under which specific operations are performed and systematizes the operating procedures based on that. For example, specific procedures such as "Step 1: Log in" and "Step 2: Go to the dashboard" are generated. The input is the analysis results, and the output is the learned operating procedures.

[1572] Step 8:

[1573] Automatic generation of user manuals

[1574] The server automatically generates a user manual using generative artificial intelligence based on learned operating procedures. This manual includes text descriptions of the operating procedures, screenshots, and, if necessary, videos. The input is the learned operating procedures, and the output is the generated user manual.

[1575] Step 9:

[1576] User manual provided.

[1577] When a user wants to refer to a manual for a specific service, the terminal retrieves the manual generated from the server and displays it on the screen. The input is the user's manual reference request, and the output is the user manual displayed on the screen.

[1578] Step 10:

[1579] User inquiries accepted.

[1580] When a user makes an inquiry about a specific service, the device sends the inquiry details to the server. For example, a user might inquire, "I don't know how to set up a new campaign." The input is the user's inquiry, and the output is the inquiry data sent to the server.

[1581] Step 11:

[1582] Inquiry content analysis using generative artificial intelligence

[1583] The server's generative artificial intelligence analyzes the received inquiry and generates appropriate operating procedures and answers. For example, it might provide an answer in the form of, "Step 1: Click 'New Campaign' from the dashboard. Step 2: Enter the campaign name." The input is the user's inquiry, and the output is the generated answer.

[1584] Step 12:

[1585] Providing the answer

[1586] The server sends the generated answer back to the terminal, which then displays it to the user. The input is the generated answer, and the output is the answer displayed to the user.

[1587] As described above, the system of the present invention allows users to easily learn the operating procedures, quickly solve problems, and improve the efficiency of handling inquiries.

[1588] (Application Example 1)

[1589] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1590] Many current systems suffer from a lack of user manuals to help users understand operating procedures, and support for inquiries is often time-consuming and cumbersome, thus compromising user convenience. Therefore, there is a need for systems that automatically generate user manuals regarding operating procedures and streamline inquiry handling. Furthermore, it is crucial that the generated manuals always reflect the latest information, but this is difficult to achieve with conventional methods. Therefore, a system that addresses these challenges is necessary.

[1591] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1592] In this invention, the server includes means for learning operating procedures using generative artificial intelligence, means for automatically generating a user manual, means for automatically generating and providing answers to user inquiries, means for collecting and transmitting user operation logs in real time, means for including text explanations, images, and videos in the generated user manual, and means for providing operating procedures and answers through a smartphone application. This enables users to efficiently understand operating procedures and quickly obtain answers to their inquiries.

[1593] "Generative artificial intelligence" is an artificial intelligence technology that can generate and predict natural language based on vast amounts of data.

[1594] An "operating procedure" is a series of actions or steps necessary to accomplish a specific task or function.

[1595] A "user manual" is a document that describes the procedures and precautions that users should take when operating a system or service.

[1596] "Inquiry support" refers to the activity of providing appropriate information and solutions to questions and problems from users.

[1597] A "user operation log" is a series of action logs recorded when a user operates the system.

[1598] "Real-time" refers to operations and data processing being performed instantly.

[1599] "Text instructions" refer to written descriptions of operating procedures and precautions.

[1600] An "image" is a still image used to provide information visually.

[1601] A "video" is a visual representation of movement created by playing multiple images in sequence.

[1602] A "smartphone application" is a computer program that runs on a smartphone.

[1603] This invention provides a system that uses generative artificial intelligence to automate the learning of operating procedures, the automatic generation of user manuals, and the handling of inquiries. Specific embodiments of this system are described below.

[1604] 1. System Configuration

[1605] This system consists of the following main components:

[1606] Terminal: Smartphone application. It is responsible for collecting user operation logs, sending inquiries, and displaying generated user manuals and inquiry responses.

[1607] Server: Performs learning and analysis using generative artificial intelligence. It also saves user operation logs, generates user manuals based on analysis results, and generates appropriate answers for inquiries.

[1608] 2. Hardware and software to be used

[1609] Hardware: Smartphones (iOS, Android compatible), cloud servers (e.g., AWS, Google Cloud, Azure)

[1610] Software: Smartphone apps (frameworks: Flutter, React Native, etc.), server-side apps (frameworks: Node.js, Django, etc.), generative artificial intelligence models (e.g., OpenAI GPT-4, GPT-3)

[1611] 3. Data processing and calculations

[1612] Collection of user operation logs

[1613] Smartphone applications running on a device collect log data in real time as the user interacts with it. For example, they record when a user clicks a button, the text they enter, and information about page transitions. This data is sent to a server using methods such as WebSocket.

[1614] Analysis and learning using generative AI

[1615] On the server, collected operation logs are analyzed and learned from by generative artificial intelligence at regular intervals. The generative AI analyzes the operation logs and patterns and defines operation procedures. The learning results are stored in a database (e.g., MongoDB, MySQL).

[1616] Automatic generation of user manuals

[1617] The server automatically generates user manuals based on operating procedures analyzed and learned using generative AI. The manuals include text explanations, images, and videos as needed. For example, a manual on how to order a product would include steps such as "Step 1: Click 'Order History' from the dashboard" and "Step 2: Select the order and click 'Cancel'," along with corresponding screenshots.

[1618] Automating customer support

[1619] When a user submits an inquiry within the smartphone application, the content is sent to the server. The server's generation AI analyzes the inquiry and generates an appropriate answer. This answer, including operating procedures and countermeasures, is sent back to the user in text format.

[1620] 4. Specific Examples and Prompts

[1621] Specific example:

[1622] If a user asks within the app, "I don't know how to cancel my order," the generative AI will create a prompt message like the following, analyze it, and then provide a response.

[1623] Example of a prompt:

[1624] A user has inquired about how to cancel an order. Please explain the order cancellation procedure. Include the following information: details of each step, necessary screenshots, and links.

[1625] Generated example answer:

[1626] Here's how to cancel your order:

[1627] 1. From the dashboard screen, click "Order History".

[1628] ![Screenshot of Step 1]()

[1629] 2. Select the order you wish to cancel and click "Cancel".

[1630] ![Screenshot of Step 2]()

[1631] 3. A confirmation message will appear; select "Yes".

[1632] ![Screenshot of Step 3]()

[1633] For detailed instructions, please refer to [this link](https: / / support.example.com / cancel-order).

[1634] In this way, the present invention provides a system that enables users to efficiently understand operating procedures and quickly obtain answers to their inquiries.

[1635] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1636] Step 1:

[1637] The device collects user activity logs in real time through a smartphone application. Input includes user actions (clicks, text input, page transitions), and this information is recorded along with a timestamp. Output is generated as activity log data, which is sent to the server using WebSocket.

[1638] Step 2:

[1639] The server saves the received operation log data to the database. The received operation log data is taken as input, and the saving process is performed to the database (e.g., MongoDB, MySQL). A status indicating successful saving is generated as output. This step involves specific actions to format the data according to data integrity and the structure of the destination table.

[1640] Step 3:

[1641] The server periodically retrieves operation log data from the database and uses the generative artificial intelligence model GPT-4 to analyze and train it. The retrieved operation log data is the input, and the generative AI model analyzes this data. As output, patterns of operation procedures as a result of training are generated and stored in the database. Specifically, data preprocessing (denoising, normalization, etc.) is performed, and the analysis results are defined as operation procedures.

[1642] Step 4:

[1643] The server uses generative artificial intelligence to automatically generate user manuals based on stored operating procedures. The input is a set of learned operating procedures, and the generative AI generates text explanations, images, and, if necessary, videos. The user manual is generated as output and stored on the server. Specifically, the operating procedures are converted into text in a sequential manner, and images and videos are captured and edited.

[1644] Step 5:

[1645] When a user submits an inquiry within a smartphone application, the device sends the content to the server. The input is the user's inquiry content, which is sent to the server in text format. The output is the text of the inquiry content reaching the server. Specifically, this includes text input in an input form and clicking the submit button.

[1646] Step 6:

[1647] The server's generative AI analyzes the query content, creates a prompt, and generates an appropriate answer. The input is the text of the query, and a prompt is generated based on it. The generative AI model analyzes this prompt and generates an appropriate answer. The output is the generated answer text. Specifically, prompt generation and subsequent answer generation are performed.

[1648] Step 7:

[1649] The server sends the generated response back to the device, which then displays it to the user. The input is the generated response text, which is sent to the device. The output is the response displayed on the user's smartphone screen. Specifically, text data is transmitted over the network, and the display component handles the screen display.

[1650] These steps enable users to efficiently understand the operating procedures and quickly obtain answers to their inquiries.

[1651] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1652] ---

[1653] This invention is a system that combines learning of operating procedures using generative artificial intelligence, automatic generation of user manuals, automation of inquiry handling, and an emotion engine that recognizes user emotions. This system further streamlines the handling of inquiries regarding the use of specific services and improves the user experience.

[1654] 1. Initial setup and retrieval of service list

[1655] The user starts the system on their terminal. The terminal reads the system's configuration file, then connects to the server to retrieve a list of services provided. This list is sent back to the terminal from the server in JSON format. The terminal then displays the received list of services to the user.

[1656] 2. Collection of user operation logs and sentiment data

[1657] When a user interacts with a specific service in their browser, the device records an operation log in real time. This operation log includes click locations, input content, page transitions, and error messages. The device also incorporates an emotion engine that collects emotional data from the user's facial expressions and voice. This emotional data is also recorded along with the operation log and stored with a timestamp.

[1658] 3. Sending operation logs and sentiment data

[1659] The device sends operation logs and emotion data recorded at regular intervals to the server. The server stores the received data in a database for analysis.

[1660] 4. Learning of Generative Artificial Intelligence

[1661] The server provides stored operation logs and emotion data to a generative artificial intelligence (AI). The AI ​​analyzes this data and learns not only the operation procedures for each service, but also makes adjustments based on the user's emotional state. For example, if the user is confused, it learns to explain the operation procedures more specifically and carefully. The server stores the learning results in a database.

[1662] 5. Automatic generation of user manuals

[1663] After learning the operating procedures for the service, the server automatically generates a user manual using generative artificial intelligence. The generated manual includes text explanations of the operating procedures, screenshots, and, if necessary, videos. It also includes content that is customized to the user's emotions. For example, if the user is nervous, it may include additional advice such as "Please relax and proceed."

[1664] 6. Automation of inquiry handling

[1665] When a user makes an inquiry about a specific service, the device sends the inquiry to the server. The server's generative artificial intelligence analyzes the received inquiry and sentiment data, and based on that, generates the optimal operating procedures and answers. For example, if a user asks, "I don't know how to set up a campaign," the server will consider the user's sentiment state and provide an answer such as, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details. Don't worry, it's easy to set up."

[1666] ---

[1667] This embodiment is expected to further streamline the handling of user inquiries regarding operation methods, and to significantly improve the user experience through the collaboration of generative artificial intelligence and an emotion engine.

[1668] The following describes the processing flow.

[1669] ---

[1670] Step 1: System Startup

[1671] The user starts the system on the terminal.

[1672] The terminal reads the system configuration file and performs an initialization process.

[1673] Step 2: Obtain the list of services

[1674] The device sends a REST API request to the server, requesting a list of the services it provides.

[1675] The server retrieves a list of currently available services from the database and sends it back to the terminal in JSON format.

[1676] The terminal displays a list of received services to the user.

[1677] Step 3: User begins operation

[1678] The user interacts with a specific service through their browser.

[1679] The device records user actions (clicks, input, page transitions, error messages, etc.) in real time.

[1680] Step 4: Collecting user sentiment data

[1681] The device's built-in emotion engine collects emotional data from the user's facial expressions and voice.

[1682] The collected emotional data is recorded along with a timestamp, similar to the operation log.

[1683] Step 5: Sending operation logs and emotion data

[1684] The device sends operation logs and emotion data recorded at regular intervals to the server.

[1685] The server stores the received data in a database for analysis.

[1686] Step 6: Learning Generative Artificial Intelligence

[1687] The server provides stored operation logs and emotion data to the generative artificial intelligence.

[1688] Generative artificial intelligence analyzes this data and learns the operating procedures for each service, as well as making adjustments based on the user's emotional state.

[1689] For example, if a user is confused, the system will learn to explain the operating procedures in more detail and with greater care.

[1690] The server stores the learning results in a database.

[1691] Step 7: User manual generation

[1692] The server uses generative artificial intelligence to generate a user manual based on the operating procedures.

[1693] The generated manual will include textual instructions, screenshots, and, if necessary, videos of the operating procedures.

[1694] It also includes content that is customized to the user's emotions. For example, if the user is feeling nervous, it may include additional advice such as, "Please relax and proceed."

[1695] Step 8: Inquiry received

[1696] A user sends an inquiry about a specific service from their device.

[1697] The terminal sends the entered query content to the server.

[1698] Step 9: Analysis of inquiry content and sentiment data

[1699] The server's generative artificial intelligence analyzes the received inquiry content and sentiment data.

[1700] Generative artificial intelligence generates optimal operating procedures and answers while taking into account the user's emotional state.

[1701] For example, if a user asks, "I don't know how to set up a campaign," the server will respond with something like, "Step 1: Click 'New Campaign' from the dashboard. See the link below for details. Don't worry, it's easy to set up."

[1702] Step 10: Submit and view your answer

[1703] The server sends the generated answer back to the terminal.

[1704] The terminal displays the received answer to the user.

[1705] This allows users to receive appropriate information in a way that is sensitive to their emotions.

[1706] ---

[1707] In this way, the overall processing flow of the system is explained by breaking it down into specific steps.

[1708] (Example 2)

[1709] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1710] In conventional systems, learning user operation procedures and generating manuals were done manually, which was inefficient and made it particularly difficult to respond while considering user emotions. Furthermore, inquiries had to be handled individually, leading to inconsistencies in response speed and quality. This made it difficult to improve the user experience and sometimes resulted in dissatisfaction with the service. Therefore, there was a need for a system that could collect operation logs and emotion data in real time, and then use generative artificial intelligence to learn and automatically generate operation procedures based on this data, thereby improving the user experience.

[1711] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1712] In this invention, the server includes means for starting the system via a user terminal, reading configuration files, and obtaining a list of services; means for collecting and recording user operation logs and sentiment data in real time; and means for sending the collected operation logs and sentiment data to the server at regular intervals and storing them in a database. This makes it possible to learn operating procedures, automatically generate user manuals, and automate inquiry handling.

[1713] 1. A "user terminal" refers to a device, such as a computer or smartphone, used by a user to operate the system.

[1714] 2. A "configuration file" is a file that contains various information necessary for the system to operate, including initial settings and connection information.

[1715] 3. The "Service List" is a list of various services provided by the server, and is a set of information that shows the services available to the user.

[1716] 4. An "operation log" is a record of the operations performed by a user on the system, and includes data such as click locations, input content, page transitions, and error messages.

[1717] 5. "Emotional data" refers to data that indicates the emotional state of a user, analyzed from their facial expressions and voice.

[1718] 6. "Real-time" refers to the processing and recording of data and operations almost simultaneously.

[1719] 7. A "server" is a computer system that provides services over a network, including data storage and analysis.

[1720] 8. A "database" is a system that systematically manages and stores operation logs, sentiment data, and other similar information.

[1721] 9. "Generative artificial intelligence" refers to an AI model that analyzes and learns from provided data to generate new information and procedures.

[1722] 10. "Learning" is the process of analyzing features from data and accumulating information internally that improves future predictions and judgments.

[1723] 11. A "user manual" is a guide that explains the procedures and methods for users to use a system.

[1724] 12. "Inquiry support" is the process of providing answers and support to questions and problems submitted by users.

[1725] 13. "Analysis" is the process of examining data in detail and understanding its meaning and relationships.

[1726] 14. A "procedure" refers to a series of steps or operations necessary to achieve a certain objective.

[1727] This invention provides a system that utilizes a user terminal, a server, and generative artificial intelligence to learn operating procedures, automatically generate user manuals, and automate inquiry handling. This streamlines the handling of user inquiries regarding operation methods and significantly improves the user experience.

[1728] Initial setup and retrieval of service list

[1729] When a user starts the system on their terminal, the terminal first loads a configuration file. This configuration file contains URLs for retrieving connection information and a list of services. Based on the configuration file, the terminal sends an HTTP request to the server to retrieve the service list. The server returns the service list in JSON format, which the terminal parses and displays to the user.

[1730] Specific example:

[1731] The user launches the system in their computer's browser and loads the configuration file (config.json). The terminal sends an HTTP request to the server to retrieve a list of available services. The server returns JSON data in response to the request, containing information such as the name, description, and provider of each service. The terminal parses this data and displays it on the user interface.

[1732] Examples of prompts for a generative AI model:

[1733] "Please explain the procedure for reading the configuration file and retrieving data to obtain a list of services provided."

[1734] Collection of user operation logs and sentiment data

[1735] When a user interacts with a specific service in their browser, the device records real-time logs of their actions, including click locations, input content, page transitions, and error messages. Furthermore, it collects emotional data from the user's facial expressions and voice through an emotion engine. This data is stored along with timestamps.

[1736] Specific example:

[1737] When a user fills out a form on a web system, the device records the buttons clicked and the contents of the entered text fields. An emotion engine is used to collect the user's facial expression data and analyze the user's emotions (e.g., happiness, anxiety, anger).

[1738] Examples of prompts for a generative AI model:

[1739] "Please provide a script for recording user activity logs on a webpage. Also, please explain how to use an emotion engine to extract emotions from a user's facial expressions."

[1740] Sending operation logs and sentiment data

[1741] The collected operation logs and emotion data are sent from the terminal to the server at regular intervals. The server stores the received data in a database.

[1742] Specific example:

[1743] The terminal sends operation logs and sentiment data to the server in batch processing every 10 minutes. The server receives the JSON data and saves it to a MySQL database.

[1744] Examples of prompts for a generative AI model:

[1745] "Please tell me the procedure for periodically sending locally recorded data to a server and saving it to a database."

[1746] Learning of generative artificial intelligence

[1747] The server provides stored operation logs and emotion data to a generative artificial intelligence (AI). The AI ​​analyzes this data and learns the operation procedures for each service. In particular, it also makes adjustments based on the user's emotional state.

[1748] Specific example:

[1749] The server uses stored operation logs and sentiment data to have a generative AI model perform analysis. The model learns to generate more specific explanations for operation procedures that frequently confuse users.

[1750] Examples of prompts for a generative AI model:

[1751] "How can we use operation logs and sentiment data to learn appropriate operating procedures based on the user's level of confusion?"

[1752] Automatic generation of user manuals

[1753] The server uses generative artificial intelligence to automatically generate user manuals based on what it has learned. The generated manuals include textual explanations of operating procedures, screenshots, and, if necessary, videos. Advice tailored to the user's emotions is also added.

[1754] Specific example:

[1755] The server generates a manual for specific operating procedures. The generated manual includes detailed explanations for each step, relevant images, and advice for confused users such as, "Don't worry, just click here."

[1756] Examples of prompts for a generative AI model:

[1757] "Please tell me how to automatically generate a user manual based on operating procedures. Please also include additional advice that takes sentiment data into consideration."

[1758] Automating customer support

[1759] When a user makes an inquiry about a specific service, the device sends the inquiry details to the server. Generative artificial intelligence analyzes this data and generates and provides the optimal operating procedures and answers.

[1760] Specific example:

[1761] When a user inquires that they "don't know how to reset their password," the server's generated AI model analyzes the relevant steps and provides guidance such as, "Go to your account settings page and click the password reset link."

[1762] Examples of prompts for a generative AI model:

[1763] "Please tell me how to automatically generate optimal operating procedures and answers based on user inquiries."

[1764] This system streamlines the handling of user inquiries regarding operation methods and significantly improves the user experience through the integration of generative artificial intelligence and an emotion engine.

[1765] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1766] Step 1:

[1767] The user starts the system on the terminal.

[1768] Specific operation: The user accesses the system URL using a browser on their PC or smartphone and starts the system.

[1769] Input: System URL

[1770] Output: System initial screen

[1771] Step 2:

[1772] The terminal reads the configuration file.

[1773] Specific operation: The terminal opens the configuration file (config.json) in the program directory and retrieves the necessary connection information and API key.

[1774] Input: Configuration file (config.json)

[1775] Output: Connection information, API key

[1776] Step 3:

[1777] The terminal sends an HTTP request to the server to retrieve a list of services.

[1778] Specific operation: The terminal sends a GET request to a URL read from the configuration file, requesting data for the list of services. The server receives the list of services in JSON format.

[1779] Input: Connection information (URL), API key

[1780] Output: List of services in JSON format

[1781] Step 4:

[1782] The terminal analyzes the list of services it has received and displays it to the user.

[1783] Specific operation: The terminal parses the received JSON data and displays the name, description, provider, etc. of each service in the user interface.

[1784] Input: List of services in JSON format

[1785] Output: List of services displayed in the user interface

[1786] Step 5:

[1787] The user interacts with a specific service through their browser.

[1788] Specific actions: Users fill out forms or click buttons to use the service.

[1789] Input: User actions

[1790] Output: System response to user interaction (page transitions, submission of input, etc.)

[1791] Step 6:

[1792] The device records user activity logs in real time.

[1793] Specific operation: Using JavaScript and other client-side technologies, data such as click location, input content, page transitions, and error messages are captured in real time and saved to a log file or memory.

[1794] Input: User actions

[1795] Output: Operation Log

[1796] Step 7:

[1797] The device uses an emotion engine to collect emotional data from the user's facial expressions and voice.

[1798] Specific operation: The system uses a camera and microphone to capture the user's facial expressions and voice, and generates emotional data through an emotion analysis API. For example, it can determine emotions such as happiness, anxiety, and anger.

[1799] Input: User's facial expressions, voice data

[1800] Output: Sentiment data

[1801] Step 8:

[1802] The device sends operation logs and emotion data collected at regular intervals to the server.

[1803] Specific operation: Every 10 minutes, the terminal batch processes and collects operation logs and sentiment data, then sends them to the server via an HTTP POST request.

[1804] Input: Operation log, sentiment data

[1805] Output: Data sent to the server

[1806] Step 9:

[1807] The server saves the received operation logs and sentiment data to a database.

[1808] Specific operation: The server parses the received JSON data and writes it to a database such as MySQL or PostgreSQL.

[1809] Input: Received operation logs, sentiment data

[1810] Output: Data stored in the database

[1811] Step 10:

[1812] The server provides stored operation logs and emotion data to the generative artificial intelligence.

[1813] Specific operation: The server periodically extracts operation logs and sentiment data from the database and feeds them to the generative artificial intelligence.

[1814] Input: Operation log, sentiment data

[1815] Output: Data provided to the generative artificial intelligence.

[1816] Step 11:

[1817] Generative artificial intelligence analyzes the provided data and learns the operating procedures.

[1818] Specific operation: Generative artificial intelligence analyzes user interaction patterns and emotional states to learn the optimal operating procedures for each service. This includes generating detailed guides for operating procedures that frequently confuse users.

[1819] Input: Operation log, sentiment data

[1820] Output: Learned operating procedures

[1821] Step 12:

[1822] The server automatically generates a user manual based on what it has learned using generative artificial intelligence.

[1823] Specific operation: The server automatically generates a manual based on the operating procedures learned by the generative artificial intelligence. The generated manual includes text explanations, screenshots, and videos. It also adds advice tailored to the user's emotions.

[1824] Input: Learned operating procedures

[1825] Output: User Manual

[1826] Step 13:

[1827] A user makes an inquiry about a specific service.

[1828] Specific action: The user enters a question into the inquiry form and clicks the submit button.

[1829] Input: Inquiry details

[1830] Output: Query received by the server

[1831] Step 14:

[1832] The server receives the inquiry, and a generative artificial intelligence analyzes the inquiry content and sentiment data.

[1833] Specific operation: The generative artificial intelligence analyzes the received inquiry content and past sentiment data to generate the optimal operating procedure and answer.

[1834] Input: Inquiry details, sentiment data

[1835] Output: Generated operating procedures and solutions

[1836] Step 15:

[1837] The server provides the generated operating procedures and solutions.

[1838] Specific operation: The server returns the generated answer to the user. For example, in response to an inquiry such as "I don't know how to reset my password," it will return specific instructions such as "Go to the account settings page and click the password reset link."

[1839] Input: Generated operating procedures and answers

[1840] Output: Answers provided to the user

[1841] This system streamlines the handling of user inquiries regarding operation methods, and significantly improves the user experience through the integration of generative artificial intelligence and an emotion engine.

[1842] (Application Example 2)

[1843] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1844] In recent years, many service providers have aimed to improve the efficiency of user support, but traditional systems often failed to adequately address user inquiries about how to use the service. Furthermore, support methods that took user emotions into consideration were not provided, resulting in a lack of improvement in the user experience. This raised concerns that user dissatisfaction would increase, potentially leading to service interruptions or cancellations.

[1845] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1846] In this invention, the server includes means for learning operating procedures using generative artificial intelligence, means for automatically generating a user manual, means for automatically generating and providing answers to user inquiries, and means for recognizing the user's emotions in real time and adjusting the operating procedures and answers based on that state. This makes it possible to respond to user inquiries about how to operate the system efficiently and in a way that is considerate of the user's emotions.

[1847] "Generative artificial intelligence" refers to artificial intelligence that has the ability to learn the user's operating procedures and automatically generate appropriate answers or manuals.

[1848] "Operating procedures" refer to a series of actions or steps that a user must perform when using a particular service or system.

[1849] A "user manual" is a guide containing documents, images, videos, and other materials generated to explain how to use a particular service or system.

[1850] "Inquiry support" refers to the activity of providing appropriate answers and solutions to questions and problems from users.

[1851] An "emotion engine" is a technology that analyzes a user's facial expressions and voice to recognize their emotions in real time.

[1852] An "operation log" is data that records the specific actions and inputs a user makes while operating a system or service.

[1853] A "server" is a computer system that stores operation logs and emotional data, and performs analysis and manual generation using generative artificial intelligence.

[1854] "Text description" refers to information in a written format that explains operating procedures and methods to the user.

[1855] A "screenshot" refers to an image captured from the screen being operated by the user.

[1856] A "video" is video content used to dynamically explain operating procedures and methods to users.

[1857] "Customization" means adjusting or changing the content according to the user's specific situation or feelings.

[1858] Modes for carrying out the invention

[1859] This invention is a system that uses generative artificial intelligence and an emotion engine to automatically learn user procedures and generate user manuals and answers to inquiries. Specific embodiments of this system are described below.

[1860] 1. Initial setup and obtaining a list of services

[1861] When the terminal starts the system, it first performs initial setup. The terminal reads a configuration file and connects to the server to retrieve a list of services provided. This list is sent back to the terminal from the server in JSON format, and the terminal displays the received list of services to the user.

[1862] 2. Collection of user operation logs and sentiment data

[1863] When a user interacts with a specific service, the device records an operation log in real time. This operation log includes click locations, input content, page transitions, and error messages. The device also incorporates an EmotionEngine, which collects emotional data from the user's facial expressions and voice. This emotional data is also recorded along with the operation log and stored with a timestamp.

[1864] 3. Sending operation logs and emotion data

[1865] The device sends operation logs and emotion data recorded at regular intervals to the server. The server stores the received data in a database for analysis.

[1866] 4. Learning of Generative Artificial Intelligence

[1867] The server provides stored operation logs and sentiment data to a generative artificial intelligence (AIAssistant). The generative AI analyzes this data and not only learns the operation procedures for each service, but also makes adjustments based on the user's emotions. For example, if the user is confused, it learns to explain the operation procedures more specifically and carefully. The server stores the learning results in a database.

[1868] 5. Automatic generation of user manuals

[1869] After learning the operating procedures for the service, the server automatically generates a user manual using generative artificial intelligence. The generated manual includes text explanations of the operating procedures, screenshots, and videos. It also includes content that is customized to the user's emotions. For example, if the user is nervous, it may include additional advice such as, "Please relax and proceed."

[1870] 6. Automation of inquiry handling

[1871] When a user makes an inquiry about a specific service, the device sends the inquiry to the server. The server's generative artificial intelligence analyzes the received inquiry and sentiment data, and based on that, generates the optimal operating procedures and answers. For example, if a user makes an inquiry such as "The video streaming keeps stopping," the generative AI analyzes the user's level of confusion using its sentiment engine when providing operating procedures, and generates an answer in the form of "Step 1: Check your Wi-Fi connection. Then, Step 2: Restart the app. Don't worry, it should go back to normal."

[1872] Specific example

[1873] Example of a printed statement:

[1874] Inquiry: Video streaming stops

[1875] Emotion: Confused (60%)

[1876] Answer: Step 1: Check your Wi-Fi connection. Then, Step 2: Restart the app. Don't worry, this should fix it.

[1877] This is expected to significantly improve the user experience by enabling more efficient and user-friendly responses to inquiries about how to use the system.

[1878] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1879] Step 1:

[1880] Initial setup and retrieval of service list

[1881] input:

[1882] Terminal startup and configuration file loading

[1883] process:

[1884] The terminal starts the system and reads the configuration file. Then it connects to the server and retrieves a list of services provided. The server returns the service list in JSON format.

[1885] output:

[1886] A list of acquired services will be displayed on the device.

[1887] Step 2:

[1888] Collection of user operation logs and sentiment data

[1889] input:

[1890] User service operation, facial expressions, voice

[1891] process:

[1892] While a user interacts with a specific service, the device records operation logs (click locations, input content, page transitions, error messages) in real time. Additionally, an EmotionEngine collects emotional data from the user's facial expressions and voice. Both the operation logs and emotional data are stored along with timestamps.

[1893] output:

[1894] Recorded operation logs and emotion data are saved to a file or database.

[1895] Step 3:

[1896] Sending operation logs and sentiment data

[1897] input:

[1898] Recorded operation logs and sentiment data

[1899] process:

[1900] The device sends recorded operation logs and emotion data to the server at regular intervals. The server stores the received data in a database for analysis.

[1901] output:

[1902] Operation logs and sentiment data stored in the server's database.

[1903] Step 4:

[1904] Learning of generative artificial intelligence

[1905] input:

[1906] Operation logs and sentiment data stored on the server

[1907] process:

[1908] The server provides operation logs and sentiment data to a generative artificial intelligence (AIAssistant). The generative AI analyzes this data and learns the operating procedures for each service. If the user is confused, it adjusts the instructions to explain them in a specific and detailed manner.

[1909] output:

[1910] The learning results are stored in the server's database.

[1911] Step 5:

[1912] Automatic generation of user manuals

[1913] input:

[1914] Learning results from generative artificial intelligence

[1915] process:

[1916] Based on learning, the server generates text instructions, screenshots, and videos of the operating procedures. Furthermore, it customizes them according to the user's emotional state. For example, if the user is nervous, it includes additional advice such as, "Please relax and proceed."

[1917] output:

[1918] An automatically generated user manual.

[1919] Step 6:

[1920] Automating customer support

[1921] input:

[1922] User inquiries and sentiment data

[1923] process:

[1924] When a user submits an inquiry, the device sends the inquiry details to the server. Generative artificial intelligence analyzes the inquiry details and sentiment data to generate the optimal operating procedures and answers. For example, if a user submits an inquiry about "video streaming stopping," the sentiment engine analyzes the user's level of frustration and generates an answer in the form of, "Step 1: Check your Wi-Fi connection. Then, Step 2: Restart the app. Don't worry, this should fix it."

[1925] output:

[1926] A prompt message that provides appropriate operating procedures or answers.

[1927] This is expected to significantly improve the user experience by enabling more efficient and user-friendly responses to inquiries about how to use the system.

[1928] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1929] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1930] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1931] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1932] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1933] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1934] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1935] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1936] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1937] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1938] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1939] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1940] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1941] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1942] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1943] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1944] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1945] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1946] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1947] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1948] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1949] The following is further disclosed regarding the embodiments described above.

[1950] ---

[1951] (Claim 1)

[1952] A means of learning operating procedures using generative artificial intelligence,

[1953] A means of automatically generating a user manual,

[1954] A means of automatically generating and providing answers to user inquiries,

[1955] A system that includes this.

[1956] (Claim 2)

[1957] The system according to claim 1, which collects user operation logs in real time and transmits them to a server.

[1958] (Claim 3)

[1959] The system according to claim 1, which generates text descriptions, screenshots, and videos of operations based on learning the operating procedures.

[1960] ---

[1961] In this way, each claim specifically describes the characteristic features of the invention.

[1962] "Example 1"

[1963] (Claim 1)

[1964] A means of learning operating procedures using generative artificial intelligence,

[1965] A means of automatically generating a user manual,

[1966] A means of automatically generating and providing answers to user inquiries,

[1967] A means of collecting user operation logs in real time and sending them to a server,

[1968] A means for generating text descriptions, screenshots, and videos of operations based on learning the operating procedures,

[1969] A system that includes this.

[1970] (Claim 2)

[1971] The system according to claim 1, wherein the user starts the system on a terminal, reads the initial settings, connects to a server, and obtains a list of services.

[1972] (Claim 3)

[1973] The system according to claim 1, wherein operation logs collected in real time are sent to a server at regular intervals, and a generative artificial intelligence learns the operation procedures.

[1974] "Application Example 1"

[1975] (Claim 1)

[1976] A means of learning operating procedures using generative artificial intelligence,

[1977] A means of automatically generating a user manual,

[1978] A means of automatically generating and providing answers to user inquiries,

[1979] A means of collecting and transmitting user operation logs in real time,

[1980] The generated user manual includes means such as text descriptions, images, and videos,

[1981] A means of providing operating procedures and answers through a smartphone application,

[1982] A system that includes this.

[1983] (Claim 2)

[1984] The system according to claim 1, which defines operating procedures using generative artificial intelligence based on analysis and learning by generative artificial intelligence.

[1985] (Claim 3)

[1986] The system according to claim 1, which collects operation logs in real time, analyzes the content of inquiries using generative artificial intelligence, and generates appropriate answers.

[1987] "Example 2 of combining an emotion engine"

[1988] (Claim 1)

[1989] A means for the user terminal to start the system, load configuration files, and retrieve a list of services,

[1990] A means of collecting and recording user operation logs and sentiment data in real time,

[1991] A means of sending collected operation logs and emotion data to a server at regular intervals and storing them in a database,

[1992] A means of learning operation procedures by analyzing saved operation logs and emotion data using generative artificial intelligence,

[1993] A method for automatically generating user manuals using generative artificial intelligence and adding advice tailored to the user's emotions,

[1994] A means to analyze user inquiries and automatically generate and provide optimal operating procedures and answers,

[1995] A system that includes this.

[1996] (Claim 2)

[1997] The system according to claim 1, which collects user operation logs in real time and collects emotional data from the user's facial expressions and voice using an emotion engine.

[1998] (Claim 3)

[1999] The system according to claim 1, which generates text descriptions, images, and a voice assistant for an operation based on learning the operation procedure.

[2000] "Application example 2 when combining with an emotional engine"

[2001] (Claim 1)

[2002] A means of learning operating procedures using generative artificial intelligence,

[2003] A means of automatically generating a user manual,

[2004] A means of automatically generating and providing answers to user inquiries,

[2005] A means of recognizing the user's emotions in real time and adjusting the operating procedures and answers based on that state,

[2006] A system that includes this.

[2007] (Claim 2)

[2008] The system according to claim 1, which collects user operation logs and sentiment data in real time and transmits them to a server.

[2009] (Claim 3)

[2010] The system according to claim 1, which generates text descriptions, screenshots, and videos of operations based on learning of the operation procedures, and further customizes them according to the user's emotional state. [Explanation of Symbols]

[2011] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of learning operating procedures using generative artificial intelligence, A means of automatically generating a user manual, A means of automatically generating and providing answers to user inquiries, A system that includes this.

2. The system according to claim 1, which collects user operation logs in real time and transmits them to a server.

3. The system according to claim 1, which generates text descriptions, screenshots, and videos of operations based on learning the operation procedures.

Images