system

Abstract

Provide a system. The method includes: a means for a user to input the final output in text form into a software application containing a dataset to be manipulated; A means for the terminal to transmit user input data to the server; A means for the server to use the generated AI model to generate specific operation procedures based on a user's request; A means to convert the specific operation procedures generated by the server into a GUI format; A means for the server to send specific operation procedures in GUI format to the terminal; A means for displaying the specific operation procedure received by the terminal on a GUI; means for a user to operate the software application according to operating procedures displayed on the GUI; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, data analysis using Excel or Google® Spreadsheets requires users to manually research and perform many individual steps. These steps are complex, time-consuming, and dependent on the user's skills, which often results in inconsistent analysis accuracy and consistency. As a result, work efficiency is reduced and erroneous analysis results may be obtained. There is a need to solve this problem and improve the efficiency of data analysis work for both individuals and companies. [Means for solving the problem]

[0005] The present invention provides a system including: a means for a user to input a final output in text format in a software application including a dataset to be operated; a means for a terminal to transmit the user's input data to a server; a means for the server to generate specific operating procedures based on the user's request using a generative AI model; a means for the server to convert the generated specific operating procedures into a GUI format; a means for the server to transmit the specific operating procedures in the GUI format to the terminal; a means for the terminal to display the specific operating procedures received on the GUI; and a means for the user to operate the software application according to the operating procedures displayed on the GUI. This reduces the time the user spends researching the procedures and allows the user to intuitively perform the operating procedures, thereby achieving consistent and accurate analysis results.

[0006] "User" refers to any individual or organization that uses software applications, including Excel and Google Sheets, to perform data analysis.

[0007] A "dataset to be manipulated" is a collection of data contained within a software application, such as Excel or Google Sheets, that is to be analyzed and manipulated.

[0008] "Final output" refers to the final result or goal that the user wants to achieve through data analysis.

[0009] "Textual input means" refers to an interface or method that allows a user to input text or strings of characters that represent the final output they desire.

[0010] "Device" refers to a device such as a computer, smartphone, or tablet that a user uses to analyze data.

[0011] "Server" refers to a central computer system that receives data sent from a user's device and processes the data using a generative AI model.

[0012] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to automatically generate optimal procedures based on input data.

[0013] "Specific operating procedures" refer to specific step-by-step instructions that a user should follow when analyzing data.

[0014] "Means for converting into GUI format" refers to methods or technologies for converting the specific operating procedures generated by a generative AI model into a format that can be displayed as a graphical user interface.

[0015] "Means for displaying on a GUI" refers to methods and technologies for visually displaying specific operating procedures on the terminal display.

[0016] "Feedback" refers to information such as opinions, evaluations, and areas for improvement provided by a user after performing an operation.

[0017] "Means for improving generative AI models" refers to methods and technologies that improve the algorithms and rules of generative AI models based on collected feedback, thereby enabling more accurate procedure generation.

[0018] "Software Applications" refers to programs and platforms for conducting data analysis, including Excel and Google Sheets. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention is a system in which a generative AI model automatically presents specific operational procedures when a user simply inputs the desired final output in data analysis work using software applications such as Excel or Google Spreadsheets. The following describes in detail an embodiment of this system.

[0041] Overall flow of the system program

[0042] User Input

[0043] The user enters the desired output in text format into an input field in Excel or Google Spreadsheet. For example, they can enter a specific request such as "aggregate sales data and create a monthly sales graph."

[0044] Sending input data

[0045] The terminal sends the user's input data to the server using an HTTP request. The input data is sent in JSON format.

[0046] Using generative AI models

[0047] The server analyzes the received user input data and activates a generative AI model. This generative AI model generates optimal operation procedures based on the user's request. For example, it generates specific instructions such as how to select a data range, how to create a pivot table, and how to insert a graph.

[0048] Converting the steps to a GUI format

[0049] The server converts the generated specific operation procedures into a GUI format, which provides the procedures in a format that is visually easy for the user to understand.

[0050] Sending procedure data

[0051] The server then sends the procedure data converted into GUI format to the terminal, again using an HTTP response, with the procedure data being sent in JSON format.

[0052] Displaying procedures on the GUI

[0053] The terminal analyzes the received instruction data and displays specific operation steps on the GUI, such as "Select a range" or "Click the Insert tab and select a pivot table."

[0054] User operations

[0055] Users follow the instructions displayed on the device's GUI to perform operations in Excel or Google Spreadsheets, allowing them to easily perform highly accurate data analysis.

[0056] Specific examples

[0057] Example: Summarizing and graphing sales data

[0058] 1. User Input

[0059] The user enters "aggregate sales data and create a monthly sales graph."

[0060] 2. Sending input data

[0061] The terminal transmits this input data to the server.

[0062] 3. Use of generative AI models

[0063] The server uses the generative AI model to generate the following specific steps:

[0064] Select the data range with dates in column A and sales amounts in column B.

[0065] Click the Insert tab and select PivotTable

[0066] Drag Date to Rows and Sales Amount to Values

[0067] Use the aggregated data to create a monthly bar chart

[0068] 4. Converting the procedure to GUI format

[0069] The server translates these steps into a GUI format.

[0070] 5. Sending procedure data

[0071] The server transmits the procedure data converted into GUI format to the terminal.

[0072] 6. Displaying procedures on the GUI

[0073] The terminal analyzes the received procedure data and displays the procedure on the GUI.

[0074] "Please select a data range with dates in column A and sales amounts in column B."

[0075] Click the Insert tab and select PivotTable.

[0076] "Drag the dates to the rows and the sales amounts to the values."

[0077] "Create a bar graph for each month."

[0078] 7. User Operations

[0079] The user follows the displayed steps to perform operations in Excel and complete a monthly sales graph.

[0080] In this way, the present invention is a system that improves the efficiency and accuracy of data analysis by automatically generating the procedures required for a user to perform data analysis and allowing intuitive operation.

[0081] The processing flow will be explained below.

[0082] Step 1:

[0083] The user enters the final output they want to achieve using Excel or Google Spreadsheets (e.g., "aggregate sales data and create a monthly sales graph") in text format into the tool's input field.

[0084] Step 2:

[0085] The device receives the user's input data and sends it to the server in JSON format via an HTTP request.

[0086] Step 3:

[0087] The server analyzes the received user input data and activates a generative AI model, which generates optimal operating procedures based on the user's requests.

[0088] Step 4:

[0089] The server converts the generated specific operation procedures into a GUI format, where the procedures output by the generative AI model are converted into a visually displayable format.

[0090] Step 5:

[0091] The server sends the procedure data converted into GUI format to the terminal. The data is sent in JSON format via an HTTP response.

[0092] Step 6:

[0093] The device analyzes the received procedure data and visually displays specific operation steps on the GUI, such as "Select a data range" or "Click the Insert tab and select a pivot table."

[0094] Step 7:

[0095] The user follows the instructions on the GUI to perform operations on Excel or Google Spreadsheets. The user follows the instructions displayed step by step to select a data range, create a pivot table, insert a chart, and so on.

[0096] Step 8:

[0097] The device collects feedback from the user during or after the operation is completed. The feedback is collected in the form of a questionnaire about whether the procedure was useful or not, what improvements can be made, etc.

[0098] Step 9:

[0099] The device sends the collected feedback to the server, which sends the feedback information in JSON format via an HTTP request.

[0100] Step 10:

[0101] The server analyzes the collected feedback and uses it to improve the generative AI model. Based on the feedback, the algorithms and rules of the generative AI model are improved, resulting in more accurate procedure generation.

[0102] Example 1

[0103] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0104] In traditional data analysis work, users often have to understand complex operating procedures in applications such as Excel or Google Spreadsheets and perform the operations manually. This tends to result in inefficient data analysis and low accuracy. Furthermore, users spend a lot of time understanding the necessary operating procedures, which reduces productivity. Furthermore, errors in operating procedures are likely to occur, resulting in a risk of lower quality.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0106] In this invention, the server includes: a means for a user to input a final output in text format in an application including an information set to be operated; a means for the terminal to transmit the user's input data to the processing device; a means for the processing device to generate specific operating procedures based on the user's request using a generative artificial intelligence model; a means for the processing device to convert the generated specific operating procedures into a visual display format; a means for the processing device to transmit the specific operating procedures in the visual display format to the terminal; a means for the terminal to display the received specific operating procedures on a visual user interface; and a means for the user to operate the application according to the operating procedures displayed on the visual user interface. This allows the user to easily understand and accurately execute the specific operating procedures. This significantly improves the efficiency and accuracy of data analysis and increases user productivity.

[0107] "User" refers to a user who utilizes the system to work with an application that includes an information set to be manipulated.

[0108] "Information set" refers to the entire collection of data that is the subject of analysis and processing.

[0109] "Application" refers to software that users use to analyze and manipulate data, such as Excel or Google Sheets.

[0110] "Output" refers to the final result or outcome that the user wants to achieve.

[0111] "Terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[0112] "Processing device" refers to a server or cloud computing environment that analyzes input data sent by a user, activates a generative AI model, and generates specific operating procedures.

[0113] "Generative artificial intelligence models" refer to AI technologies such as machine learning algorithms and large-scale language models that generate optimal operating procedures based on user requests.

[0114] The "visual display format" refers to a format in which the generated specific operating procedures are visualized so that the user can easily understand them.

[0115] "Visual user interface" refers to a graphical interface that allows a user to view and execute operating procedures.

[0116] "Feedback" refers to opinions and information provided by users regarding operating procedures.

[0117] The present invention is a system in which a generative AI model automatically presents specific operational procedures when a user simply inputs the desired final output in data analysis work using applications such as Excel or Google Spreadsheets. The following describes in detail an embodiment of this system.

[0118] User Input

[0119] The user inputs in text form the final output they want to achieve from the application containing the information set to be manipulated, for example, a request such as "aggregate sales data and create a monthly sales graph."

[0120] Sending input data

[0121] The terminal sends the user's input data to the processing device. This transmission uses an HTTP request, and the data is converted into JSON format. Specifically, the terminal converts the user's input "aggregate sales data and create a monthly sales graph" into JSON data ({"request": "aggregate sales data and create a monthly sales graph"}) and sends it.

[0122] Using generative AI models

[0123] The server analyzes the received input data and activates a generative AI model, which generates operational procedures based on the user's request. For example, it generates the following procedures:

[0124] Select the data range with dates in column A and sales amounts in column B.

[0125] Click the Insert tab and select PivotTable

[0126] Drag Date to Rows and Sales Amount to Values

[0127] Use the aggregated data to create a monthly bar chart

[0128] Converting the steps to a GUI format

[0129] The server converts the generated specific operation instructions into a visual display format, providing the instructions in a form that is easy for the user to understand visually. For example, it generates GUI code including instructions such as "Please select a data range with dates in column A and sales amounts in column B."

[0130] Sending procedure data

[0131] The server then sends the procedure data converted into a visual display format to the terminal, again using an HTTP response, with the data being sent in JSON format.

[0132] Displaying procedures on the GUI

[0133] The terminal analyzes the received instruction data and displays specific operation instructions on the GUI. For example, the terminal visually displays instructions such as "Select a data range with dates in column A and sales amounts in column B," "Click the Insert tab," "Select a pivot table," "Drag dates to rows," "Drag sales amounts to values," and "Create a bar graph by month."

[0134] User operations

[0135] Users follow the instructions displayed on the device's GUI to perform operations in Excel or Google Spreadsheets, allowing them to easily perform highly accurate data analysis.

[0136] Specific examples

[0137] Example: Summarizing and graphing sales data

[0138] 1. User Input

[0139] The user enters "aggregate sales data and create a monthly sales graph."

[0140] 2. Sending input data

[0141] The terminal transmits this input data to the processing unit.

[0142] 3. Use of generative AI models

[0143] The server uses the generative AI model to generate the following specific steps:

[0144] Select the data range with dates in column A and sales amounts in column B.

[0145] Click the Insert tab and select PivotTable

[0146] Drag Date to Rows and Sales Amount to Values

[0147] Use the aggregated data to create a monthly bar chart

[0148] 4. Converting the procedure to GUI format

[0149] The server translates these instructions into a visual display format.

[0150] 5. Sending procedure data

[0151] The server transmits the procedure data converted into a visual display format to the terminal.

[0152] 6. Displaying procedures on the GUI

[0153] The terminal analyzes the received procedure data and displays the procedure on the GUI.

[0154] "Please select a data range with dates in column A and sales amounts in column B."

[0155] Click the Insert tab and select PivotTable.

[0156] "Drag the dates to the rows and the sales amounts to the values."

[0157] "Create a bar graph for each month."

[0158] 7. User Operations

[0159] The user follows the displayed steps to perform operations in Excel and complete a monthly sales graph.

[0160] In this way, the present invention is a system that improves the efficiency and accuracy of data analysis by automatically generating the procedures required for users to perform data analysis and allowing them to operate intuitively.

[0161] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0162] Step 1:

[0163] (user input)

[0164] How it works: The user enters a textual description of the final output they want to achieve from the application, including the set of information they want to operate on. For example, they might enter "aggregate sales data and create a graph of sales by month."

[0165] Input: The user enters the request in text format into an Excel or Google Spreadsheet.

[0166] Output: The user's request is saved as text in the input field.

[0167] Step 2:

[0168] (Send input data)

[0169] How it works: The device converts the user's input data into JSON format and sends it to the server using an HTTP request.

[0170] Input: The request entered by the user in text format.

[0171] Data processing: Convert text data into JSON format.

[0172] Output: The data converted to JSON format is sent to the server via an HTTP request.

[0173] Step 3:

[0174] (Using generative AI models)

[0175] How it works: The server parses the received JSON data and inputs it as a prompt to the generative AI model, which then generates the optimal operation procedure based on the user's request.

[0176] Input: User request data in JSON format (e.g., {"request": "Aggregate sales data and create a monthly sales chart"}).

[0177] Data calculation: Analyzes JSON data and inputs it as prompts to the generative AI model to generate specific operating procedures.

[0178] Output: Generated instructions (e.g. "Select the data range with dates in column A and sales amounts in column B").

[0179] Step 4:

[0180] (Converting the procedure into GUI format)

[0181] How it works: The server analyzes the generated operating instructions and converts them into a format that is easy for the user to understand visually.

[0182] Input: Specific operational instructions generated by a generative AI model.

[0183] Data processing: The generated operating procedures are converted into a visual display format (GUI) using HTML, CSS, and JavaScript (registered trademark).

[0184] Output: Procedure data in GUI format.

[0185] Step 5:

[0186] (Send procedure data)

[0187] Operation: The server sends the procedure data converted into GUI format to the terminal using an HTTP response.

[0188] Input: Specific operation procedure data in GUI format.

[0189] Data processing: Convert GUI format data into JSON format.

[0190] Output: The data converted to JSON format is sent to the terminal as an HTTP response.

[0191] Step 6:

[0192] (Procedure displayed on GUI)

[0193] Operation: The terminal analyzes the received procedure data and displays specific operating procedures on the GUI.

[0194] Input: GUI steps data in JSON format (e.g., {"steps": ["Select a data range with dates in column A and sales amounts in column B", "Click the Insert tab", "Select PivotTable", "Drag Dates to Rows", "Drag Sales Amounts to Values", "Create a bar chart by month"]}).

[0195] Data operations: Parse the JSON data and process it to display it on a visual user interface.

[0196] Output: Specific operation steps visualized on the GUI.

[0197] Step 7:

[0198] (User operation)

[0199] How it works: Users follow the on-screen instructions to perform operations in Excel or Google Sheets. Specifically, they are instructed to "select a data range with dates in column A and sales amounts in column B," "click the Insert tab," "select PivotTable," "drag dates to rows," "drag sales amounts to values," and "create a bar graph by month."

[0200] Input: Operation steps displayed on the GUI.

[0201] Data processing: Users manually process data in Excel or Google Spreadsheets according to instructions.

[0202] Output: The final output desired by the user (e.g., a graph of sales by month).

[0203] (Application example 1)

[0204] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0205] Traditional data analysis and manufacturing process optimization required a great deal of specialized knowledge and time, resulting in reduced work efficiency on the factory floor. Furthermore, if workers were not familiar with data analysis and process improvement, it was difficult to obtain accurate analysis results and effective improvement measures. Furthermore, there was also the issue of workers finding it difficult to understand and apply procedures in real time.

[0206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0207] In this invention, the server includes: a means for a user to input a final output in text format in a software application including a dataset to be operated; a means for a terminal to send the user's input data to the server; a means for the server to generate specific operating procedures based on the user's request using a generative AI model; a means for the server to convert the generated specific operating procedures into a GUI format; a means for the server to send the specific operating procedures in GUI format to the terminal; a means for the terminal to display the specific operating procedures received on the GUI; a means for the user to operate the software application in accordance with the operating procedures displayed on the GUI; a means for the user to input voice instructions for data analysis and manufacturing process improvement in a factory and convert them into text format; a means for the server to generate manufacturing data analysis procedures and manufacturing process improvement procedures using the generative AI model and display them in GUI format; and a means for the user to operate systems in the factory according to the instructions displayed on the terminal's GUI. This enables workers without specialized knowledge to efficiently and accurately analyze data and improve manufacturing processes.

[0208] "User" refers to an individual or group of individuals who operate software applications and factory systems.

[0209] "Dataset to be manipulated" refers to a collection of data that a user processes and analyzes within a software application.

[0210] "Software Applications" refers to programs for data processing and analysis, such as Excel and Google Spreadsheets.

[0211] "Final output" refers to the result or goal the user wants to achieve.

[0212] "Means of inputting in text format" refers to methods of inputting characters using a keyboard, voice input, etc.

[0213] "Device" refers to devices such as smartphones, tablets, and smart glasses.

[0214] "Server" refers to a remote computing device that processes generative AI models and manages data.

[0215] A "generative AI model" refers to an artificial intelligence program that automatically generates specific operating procedures based on user requests.

[0216] "Specific operating procedures" refer to the specific steps that the user must actually take, such as selecting a data range or creating a graph.

[0217] "GUI format" refers to a graphical user interface format, a display format that is visually easy to understand.

[0218] A "manufacturing process" refers to a series of tasks and procedures for producing a product in a factory.

[0219] "Manufacturing process improvement" refers to measures or attempts to improve the efficiency or quality of a manufacturing process.

[0220] "Means for inputting voice and converting it into text format" refers to a method of converting verbal instructions into text data using voice recognition technology.

[0221] "Systems within the factory" refers to the various machines and programs used in the factory.

[0222] "Feedback" refers to opinions and evaluation information provided by users after using the system.

[0223] "Improvement procedures" refer to specific steps taken to review existing manufacturing processes and data analysis procedures and make them more effective.

[0224] The present invention provides a system for efficiently performing data analysis and manufacturing process improvements in factories. This system allows users to input instructions in the form of voice or text, and uses a generative AI model to provide specific operating procedures. An embodiment of this system is described in detail below.

[0225] System Overview

[0226] This system consists of a user, a terminal, a server, and a generative AI model. The user uses a software application containing the dataset to be operated on and inputs the final output in text format. The terminal sends the user's input data to the server, which uses the generative AI model to generate specific operating procedures. The generated operating procedures are converted into GUI format and sent to the terminal. The user performs operations by following the specific procedures displayed on the terminal.

[0227] Hardware and software used

[0228] Hardware: This system includes smart glasses, tablet devices, and servers. Specific examples include smart glasses such as Google Glass® and Vuzix Blade, and cloud servers such as Amazon Web Services (AWS®) and Microsoft® Azure®.

[0229] Software: The system includes OpenAI® GPT-4® as a generative AI model, Python libraries (Pandas, Numpy) for data processing, Google Speech-to-Text API for voice recognition, and React Native and Vue.js for GUI display.

[0230] Specific examples of the system

[0231] 1. User input method:

[0232] Using smart glasses or a tablet, users can input data analysis and improvement instructions for the manufacturing process in voice or text format. For example, they could give instructions such as "Analyze the manufacturing defect rate by month and propose improvement measures."

[0233] 2. Send input data:

[0234] The device converts the voice input into text format and sends the input data in JSON format to the server.

[0235] 3. Use of generative AI models:

[0236] The server launches a generative AI model (e.g., OpenAI GPT-4) to generate specific operating procedures based on the user's request.

[0237] 4. Converting the operating procedure to GUI format:

[0238] The server converts the generated specific operating procedures into GUI format and sends them to the terminal.

[0239] 5. GUI procedure:

[0240] The terminal analyzes the received procedure data and displays specific operation procedures on the GUI, allowing the user to receive instructions in a visually easy-to-understand format.

[0241] 6. User Actions:

[0242] Users follow the procedures displayed on the terminal's GUI to operate the factory's manufacturing system and perform data analysis and process improvements.

[0243] Specific examples

[0244] Example of user input:

[0245] "Analyze the manufacturing defect rate by month and create a graph."

[0246] Example prompts to send to the generative AI model:

[0247] "Please provide the steps to use a generative AI model to analyze the defect rate of manufacturing data by month and generate a graph showing this."

[0248] This enables the system to efficiently and accurately perform data analysis and process improvements at the manufacturing site.

[0249] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0250] Step 1:

[0251] User Input

[0252] The user uses smart glasses or a tablet device to input the dataset to be manipulated and the final output in voice or text format. For example, the user might give instructions such as "Analyze the manufacturing defect rate by month and propose improvements." The input is then converted into text by voice recognition software (e.g., Google Speech-to-Text API) and imported into the device.

[0253] Input data: "Analyze manufacturing defect rates by month and propose improvement measures"

[0254] Output data: Text data converted from audio

[0255] Step 2:

[0256] Sending input data

[0257] The device parses the converted text data into JSON format and sends it to the server using an HTTP request, where the data is sent to the server in JSON format.

[0258] Input data: JSON format text data

[0259] Output: HTTP request sent to the server

[0260] Step 3:

[0261] Using generative AI models

[0262] The server analyzes the received JSON-formatted text data and launches a generative AI model (e.g., OpenAI GPT-4). The server constructs a prompt sentence and requests the generative AI model to generate specific steps for data analysis and manufacturing process improvement.

[0263] Input data: JSON format text data

[0264] Data processing: Generate prompts to input to the generative AI model

[0265] Output data: Specific operating procedures

[0266] Step 4:

[0267] Converting the steps to a GUI format

[0268] The server analyzes the specific operation procedures returned by the generative AI model and converts them into a GUI format that is easy for users to understand visually, using a front-end framework such as React Native or Vue.js.

[0269] Input data: Operation instructions received from the generative AI model

[0270] Data processing: Converting procedure data into GUI format

[0271] Output data: GUI-based specific operation procedures

[0272] Step 5:

[0273] Sending procedure data

[0274] The server parses the specific operation procedures converted into GUI format into JSON format and sends them to the terminal using an HTTP response.

[0275] Input data: GUI-based specific operation procedures

[0276] Output data: HTTP response sent to the device

[0277] Step 6:

[0278] Displaying procedures on the GUI

[0279] The device analyzes the received procedure data and displays it on the screen of the smart glasses or tablet device, allowing the user to visually confirm the specific operation procedures.

[0280] Input data: Operation procedure data received from the server (JSON format)

[0281] Data processing: Rendering data for display on a GUI

[0282] Output data: Operational steps that can be visually confirmed by the user

[0283] Step 7:

[0284] User operations

[0285] Users operate the factory's manufacturing system by following specific operating procedures displayed on the terminal, such as "select a data range," "count the number of defective products," and "generate graphs," to analyze data and improve the manufacturing process.

[0286] Input data: Operation procedure displayed on the GUI

[0287] Output data: Completed data analysis and manufacturing process improvement operations

[0288] This allows all steps to work together, enabling users to analyze data and improve manufacturing processes efficiently and accurately, even without specialized knowledge.

[0289] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0290] This invention relates to a system in which a generative AI model automatically suggests specific operating procedures when a user performs data analysis using a software application, simply by inputting the final output. This system also combines an emotion engine that recognizes the user's emotions. This system allows users to receive more personalized support and perform data analysis efficiently and accurately.

[0291] Overall flow of the system program

[0292] User Input

[0293] The user enters the desired output in text format in the input field in Excel or Google Spreadsheet. For example, they might enter "aggregate sales data and create a monthly sales graph."

[0294] Sending input data

[0295] The terminal receives the user's input data and sends it to the server in JSON format via an HTTP request.

[0296] Using generative AI models

[0297] The server analyzes the received user input data and activates a generative AI model, which generates optimal operating procedures based on the user's requests.

[0298] Converting the steps to a GUI format

[0299] The server converts the generated specific operation procedures into a GUI format, which provides the procedures in a format that is visually easy for the user to understand.

[0300] Sending procedure data

[0301] The server then sends the procedure data converted into GUI format to the terminal. The data is sent in JSON format via an HTTP response.

[0302] Displaying procedures on the GUI

[0303] The terminal analyzes the received procedure data and visually displays specific operation steps on the GUI, such as "Select a data range" or "Click the Insert tab and select a pivot table."

[0304] User operations

[0305] Users follow the instructions displayed on the device's GUI to perform operations in Excel or Google Spreadsheets, enabling them to easily perform highly accurate data analysis.

[0306] Integration with emotion engine

[0307] emotion recognition

[0308] The device uses an emotion engine to recognize the user's emotions based on the user's input data and behavior during operation (e.g., keyboard operations, mouse movements, facial recognition, etc.). For example, it identifies emotional states such as "happiness," "anxiety," and "excitement."

[0309] Sending emotional data

[0310] The device sends the analyzed emotional data to the server in JSON format via an HTTP request.

[0311] Emotion-based procedural adjustments

[0312] The server receives the user's emotional data and adjusts the generated operation procedure based on the user's emotional state. For example, if the user feels "anxious," the server may provide more detailed instructions.

[0313] Gathering feedback and improving the model

[0314] The device collects feedback from the user during or after the operation is completed and sends it to the server. The feedback information is sent in JSON format via an HTTP request, and the server uses this information to improve the generative AI model. Emotional data is also analyzed to improve the accuracy of the generative AI model.

[0315] Specific examples

[0316] Example: Summarizing and graphing sales data

[0317] 1. User Input

[0318] The user enters "aggregate sales data and create a monthly sales graph."

[0319] 2. Sending input data

[0320] The terminal transmits this input data to the server.

[0321] 3. Use of generative AI models

[0322] The server uses the generative AI model to generate the following specific steps:

[0323] Select the data range with dates in column A and sales amounts in column B.

[0324] Click the Insert tab and select PivotTable

[0325] Drag Date to Rows and Sales Amount to Values

[0326] Use the aggregated data to create a monthly bar chart

[0327] 4. Emotion recognition

[0328] As the user performs the operation, the device recognizes the user's emotions based on keyboard operations and mouse movements.

[0329] 5. Sending Emotional Data

[0330] The terminal transmits the user's emotion data to the server.

[0331] 6. Emotion-Based Procedural Adjustments

[0332] The server analyzes the user's emotional data and provides more detailed instructions as needed to help alleviate the anxiety the user feels.

[0333] 7. Displaying procedures on the GUI

[0334] The terminal displays the generated procedures on a GUI, allowing the user to operate them intuitively.

[0335] 8. User Operations

[0336] The user follows the displayed steps to complete the monthly sales graph.

[0337] 9. Gather feedback and improve the model

[0338] After the user completes the operation, the device collects feedback and sends it to the server, which uses it to improve the generative AI model and emotion engine.

[0339] In this way, by combining a generative AI model and an emotion engine, the present invention provides a data analysis support system that is intuitively operable for users and takes emotions into consideration.

[0340] The processing flow will be explained below.

[0341] Step 1:

[0342] The user enters the final output they want to achieve using Excel or Google Spreadsheets (e.g., "aggregate sales data and create a monthly sales graph") in text format into the tool's input field.

[0343] Step 2:

[0344] The device receives the user's input data and sends it to the server in JSON format using an HTTP request.

[0345] Step 3:

[0346] The server analyzes the received user input data and activates a generative AI model, which generates optimal operating procedures based on the user's requests.

[0347] Step 4:

[0348] The server converts the generated specific operation procedures into a GUI format, where the procedures output by the generative AI model are converted into a format that can be displayed visually.

[0349] Step 5:

[0350] The server sends the procedure data converted into GUI format to the terminal. The procedure data is sent in JSON format using an HTTP response.

[0351] Step 6:

[0352] The device analyzes the received instruction data and visually displays specific operation steps on the GUI, such as "Select a data range" or "Click the Insert tab and select a pivot table."

[0353] Step 7:

[0354] The user follows the instructions on the GUI to perform operations on Excel or Google Spreadsheets. The user follows the instructions displayed step by step to select a data range, create a pivot table, insert a chart, and so on.

[0355] Step 8:

[0356] The device uses an emotion engine to recognize the user's emotions based on the user's input data and behavior during operation (e.g., keyboard operations, mouse movements, facial recognition, etc.). For example, it can identify emotional states such as "happiness," "anxiety," and "excitement" based on the user's facial expressions and operation speed.

[0357] Step 9:

[0358] The device sends the analyzed emotion data to the server in JSON format via an HTTP request.

[0359] Step 10:

[0360] The server analyzes the received user emotion data and adjusts the generated operation instructions based on the user's emotional state. For example, if the user is feeling "anxious," the instructions are made more detailed to provide a sense of security.

[0361] Step 11:

[0362] The device will then redisplay the specific operational steps on the GUI, allowing the user to receive support adapted to their emotional state.

[0363] Step 12:

[0364] The user continues to follow the instructions on the GUI and completes the final output (e.g., a monthly sales graph).

[0365] Step 13:

[0366] The device collects feedback from the user during or after the operation is completed. The feedback information is collected in the form of a questionnaire about whether the procedure was useful or not, what needs to be improved, etc.

[0367] Step 14:

[0368] The device sends the collected feedback to the server in JSON format via an HTTP request.

[0369] Step 15:

[0370] The server analyzes the collected feedback and uses it to improve the generative AI model and emotion engine. Based on the feedback, the algorithms and rules of the generative AI model are improved, resulting in more accurate procedure generation and emotion recognition.

[0371] Example 2

[0372] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0373] Modern data analysis tasks often require specialized knowledge for users to perform tasks efficiently and accurately. Furthermore, manually understanding and executing a series of operational steps is laborious and time-consuming. Therefore, a system is needed in which a generative AI model automatically suggests specific operational steps based on the user's input. Furthermore, there is a need to provide more personalized support and improve the user experience by adjusting the steps based on the user's emotions. Current technology does not offer a system that meets all of these requirements, which is the problem this invention aims to solve.

[0374] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0375] In this invention, the server includes means for analyzing user input data and generating specific operating procedures using a generative AI model, means for converting the generated specific operating procedures into a GUI format, and means for adjusting the generated operating procedures based on emotion data transmitted from the terminal in accordance with the user's emotional state. This allows the user to receive visually easy-to-understand specific operating procedures simply by inputting the final output, and furthermore, by taking the user's emotions into consideration, stress and anxiety can be reduced, and data analysis work can proceed more smoothly.

[0376] "User" refers to an individual or organization that uses the system to perform data analysis work.

[0377] A "terminal" is a computer device operated by a user, and refers to a device for inputting, transmitting, receiving, displaying, etc. data.

[0378] "Server" refers to a computer system that uses a generative AI model to generate specific operating procedures and communicates with terminals.

[0379] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates optimal operating procedures based on user requests.

[0380] A "prompt" is a document or instruction that allows a user to input the final output in text form.

[0381] An "operational procedure" refers to a set of specific instructions for a user to correctly perform a data analysis task.

[0382] "GUI format" refers to a graphical user interface format designed to make it easy for users to operate visually.

[0383] "Emotion engine" refers to an artificial intelligence algorithm that analyzes a user's emotional state and improves the user's operating experience.

[0384] "Feedback" refers to information in which a user provides evaluations and opinions on operating procedures.

[0385] The present invention is a system for solving problems users face in data analysis work, and combines a generative AI model and an emotion engine to provide specific operating procedures to users and provide personalized support based on emotions. Detailed embodiments of this system are described below.

[0386] Overall system overview

[0387] This system provides specific operating procedures through the following steps when users perform data analysis using software applications (e.g., Excel or Google Spreadsheets): Multiple hardware and software components, including a server, terminals, generative AI models, and an emotion engine, work together.

[0388] The specific hardware and software used

[0389] 1. Hardware

[0390] Terminal: The device operated by the user (PC, tablet, smartphone, etc.)

[0391] Server: A remote server that hosts the generative AI model and emotion engine.

[0392] 2. Software

[0393] Generative AI model: An AI algorithm that generates specific operating procedures based on user requests.

[0394] Emotion Engine: An AI algorithm for analyzing a user's emotional state

[0395] Software applications: Data analysis tools such as Excel and Google Sheets

[0396] Processing flow

[0397] 1. User Input

[0398] The user enters the final output they want to achieve in the software application in text format. An example prompt might be "Summarize sales data and create a monthly sales graph."

[0399] 2. Sending input data

[0400] The terminal receives the user's input data and sends it to the server in JSON format via an HTTP request.

[0401] 3. Use of generative AI models

[0402] The server analyzes the received user input data and activates a generative AI model, which generates optimal operation procedures based on the user's prompt.

[0403] 4. Converting the procedure to GUI format

[0404] The server converts the generated specific operation procedures into a GUI format.

[0405] 5. Sending procedure data

[0406] The server transmits the procedure data converted into GUI format to the terminal.

[0407] 6. Displaying procedures on the GUI

[0408] The device analyzes the received instruction data and visually displays specific operation steps on the GUI. Specific examples include instructions such as "Select a data range" and "Click the Insert tab and select a pivot table."

[0409] 7. User Operations

[0410] The user follows the instructions displayed on the device's GUI and performs operations on Excel or Google Spreadsheets.

[0411] Integration with emotion engine

[0412] 1. Emotion recognition

[0413] The device uses an emotion engine to recognize the user's emotions based on the user's input data and behavior during operation (keyboard operations, mouse movements, facial recognition, etc.). Recognized emotions include "happiness," "anxiety," and "excitement."

[0414] 2. Sending Emotional Data

[0415] The device sends the analyzed emotion data to the server in JSON format via an HTTP request.

[0416] 3. Emotion-Based Procedural Adjustments

[0417] The server receives the user's emotional data and adjusts the generated operation procedure based on the user's emotional state, for example, if the user is feeling "anxious," the procedure will be explained in more detail.

[0418] 4. Gather feedback and improve the model

[0419] After the user completes the operation, the device collects feedback and sends it to the server, which uses this feedback to improve the accuracy of the generative AI model and emotion engine.

[0420] Specific examples

[0421] If a user types "aggregate sales data and create a monthly sales graph," the following steps will be generated:

[0422] Select the data range with dates in column A and sales amounts in column B.

[0423] Click the Insert tab and select PivotTable

[0424] Drag Date to Rows and Sales Amount to Values

[0425] Use the aggregated data to create a monthly bar chart

[0426] By following these steps, users can easily complete data analysis tasks, and the emotion engine can recognize the user's emotional state and provide appropriate support to further improve the user experience.

[0427] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0428] Step 1: User Input

[0429] The user enters the final output in text format into the software application to be operated. For example, they might enter "aggregate sales data and create a monthly sales graph." This input becomes the prompt statement, which forms the basis for the next step of data processing.

[0430] Input: Text input for final output

[0431] Output: Prompt in text format

[0432] Step 2: Submitting input data

[0433] The terminal receives the user's input data (prompt text) and sends it to the server in JSON format using an HTTP request. Specifically, the prompt text is sent to the server in the format {"task": "Aggregate sales data and create a monthly sales graph"}.

[0434] Input: prompt statement

[0435] Output: JSON format data

[0436] Step 3: Use the generative AI model

[0437] The server analyzes the received prompt and activates a generative AI model. A generative AI model is an algorithm that generates specific operational steps based on the user's request. In response to the prompt "aggregate sales data and create a monthly sales graph," the AI ​​generates the following operational steps:

[0438] Select a data range

[0439] Click the Insert tab

[0440] Select the pivot table

[0441] Drag a date to a row

[0442] Drag Sales Amount to Values

[0443] Create a bar chart

[0444] Input: JSON format prompt statement

[0445] Output: List of specific operating procedures

[0446] Step 4: Converting the Instructions to GUI Format

[0447] The server converts the generated specific operation steps into a GUI format. To make it visually easy for users to understand, each step is formatted as a button or interactive interface. For example, the message "Please select a data range" is displayed as an appropriate button.

[0448] Input: List of operating procedures

[0449] Output: GUI format operation procedure

[0450] Step 5: Sending Instruction Data

[0451] The server sends the GUI-formatted step data to the terminal in JSON format using an HTTP response. For example, the following data is sent: {"steps": ["Select a data range", "Click the Insert tab and select a pivot table"]}.

[0452] Input: GUI-style operation procedure

[0453] Output: Procedure data in JSON format

[0454] Step 6: Displaying the procedure on the GUI

[0455] The terminal analyzes the received procedure data and visually displays specific operation procedures on the GUI. To make it easier for the user to understand, the procedures are displayed on the screen as pop-up windows or information bars. This allows the user to follow the instructions to proceed with the operation.

[0456] Input: Procedure data in JSON format

[0457] Output: Visual operation procedure on GUI

[0458] Step 7: User interaction

[0459] The user follows the instructions displayed on the device's GUI to perform operations on Excel or Google Spreadsheets. Specifically, the user follows the instructions to perform actions such as "select a data range," "click the Insert tab," and "select a pivot table."

[0460] Input: GUI operation procedure

[0461] Output: User completes an action within the software

[0462] Step 8: Emotion Recognition

[0463] The device uses an emotion engine to recognize the user's emotions based on the user's input data and actions (keyboard operations, mouse movements, facial recognition, etc.). For example, it uses a camera and facial recognition software to identify emotional states such as "happiness," "anxiety," and "excitement."

[0464] Input: User behavior data

[0465] Output: User emotion data

[0466] Step 9: Sending Emotion Data

[0467] The device sends the analyzed emotion data to the server in JSON format via an HTTP request. For example, data such as "{"emotion": "anxiety"}" is sent.

[0468] Input: User emotion data

[0469] Output: Emotion data in JSON format

[0470] Step 10: Adjusting procedures based on emotions

[0471] The server receives the user's emotional data and adjusts the generated operation instructions based on the user's emotional state. For example, if the user is feeling "anxious," the server may add more detailed instructions to the operation instructions.

[0472] Input: Emotion data in JSON format

[0473] Output: Adjusted operating procedure

[0474] Step 11: Gather feedback and improve the model

[0475] After the user completes the operation, the device collects feedback and sends it to the server. The user feedback is used to improve the accuracy of the generative AI model and emotion engine. For example, the model can be adjusted based on feedback such as "The instructions were difficult to understand."

[0476] Input: User feedback

[0477] Output: Improved generative AI models and emotion engines

[0478] (Application example 2)

[0479] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0480] In conventional data analysis support systems, when users perform data analysis tasks on software applications, they must decide the specific operating procedures themselves, making the process complicated and difficult to perform efficiently. Furthermore, when users feel emotionally confused or anxious, the system often fails to provide appropriate support, slowing down the progress of the task. In particular, in everyday tasks such as inventory management and sales forecasting in brick-and-mortar stores, where fast and accurate data analysis is required, there is a lack of flexible operational support that responds to the user's emotions.

[0481] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0482] In this invention, the server includes means for generating specific operating procedures based on a user's request using a generative AI model, means for converting the generated specific operating procedures into a GUI format, and means for recognizing the user's emotions using an emotion engine and adjusting the generated operating procedures based on the emotions. This allows the user to receive emotion-sensitive support while quickly and accurately analyzing data, enabling efficient inventory management and sales forecasting in physical stores.

[0483] A "target information set" is a data set that a user can manipulate and analyze within a software application.

[0484] "Final output" refers to the result that a user wants to achieve using a software application.

[0485] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[0486] The "server" is a central processing system that processes data received from users and provides operational instructions and support using generative AI models and emotion engines.

[0487] A "generative AI model" refers to an artificial intelligence model that generates specific operating procedures based on user requests.

[0488] "GUI format" refers to a format for visually displaying operating procedures in the form of a graphical user interface.

[0489] An "emotion engine" is a system that recognizes a user's emotions based on input data and actions during user operation and analyzes that data.

[0490] "Emotion data" is data that indicates the user's emotional state analyzed by the emotion engine.

[0491] "Feedback" refers to the opinions, evaluations, and data regarding operations that users provide to the system.

[0492] "Data analysis" is the act of processing a set of information to extract patterns and useful information to produce a final output.

[0493] "Inventory management" refers to the task of understanding the inventory status of products, merchandise, and other materials in physical stores and maintaining an adequate supply.

[0494] "Sales forecasting" is the activity of predicting future sales based on past sales data and current sales trends.

[0495] This invention is a system in which, when a user performs data analysis using a software application that manipulates a set of information, the generative AI model automatically presents specific operating procedures simply by inputting the final output, and further combines this with an emotion engine that recognizes the user's emotions.

[0496] First, the user uses the terminal to input the final output in text format. For example, they might input "I want to forecast this month's inventory" or "I want to identify best-selling products." The terminal then sends this input data in JSON format to the server. Communication is via HTTP requests.

[0497] The server analyzes the received user input data and activates a generative AI model. The generative AI model generates optimal operating procedures based on the user's requests. For example, for inventory forecasting, it presents analytical procedures based on historical sales data and current inventory data. The generated specific operating procedures are converted into GUI format by the server. This conversion provides the procedures in a format that is visually easy for users to understand.

[0498] Furthermore, the device uses an emotion engine to recognize the user's emotions based on input data and actions (e.g., keyboard input and touch operations) during operation. The recognized emotion data is sent to the server in JSON format. The server analyzes the emotion data and adjusts the generated operation instructions to reduce the user's anxiety or confusion. For example, if the user feels anxious, it may provide more detailed explanations of the instructions.

[0499] The terminal receives specific operating instructions from the server and displays them on a GUI. The user operates the software application according to the displayed instructions. This allows the user to perform data analysis efficiently and accurately by following the guide.

[0500] After providing the operation instructions, the server collects feedback from the user and uses it to improve the accuracy of the generative AI model and emotion engine. The feedback information is also sent to the server in JSON format.

[0501] Specifically, a smartphone is used as the terminal, and a generative AI model such as OpenAI's GPT-3 (registered trademark) is used on the server side. As an emotion engine, a machine learning framework such as Keras is used to analyze the user's facial expressions and input data.

[0502] (Example)

[0503] Here is an example prompt:

[0504] Example prompt sentence:

[0505] User request: I want to make inventory forecast for this month. Please provide specific steps.

[0506] Based on this prompt, a generative AI model generates instructions, and an emotion engine recognizes the user's emotions and adjusts the instructions accordingly, providing personalized assistance to the user.

[0507] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0508] Step 1:

[0509] A user inputs the final output in text format into a software application containing the set of information to be operated on. Specifically, using a smartphone as a terminal, the user inputs requests such as "I want to forecast this month's inventory" or "I want to identify best-selling products." The input data is saved in text format on the terminal.

[0510] Step 2:

[0511] The terminal sends the user's input data to the server. Communication is via HTTP request, and the data is sent in JSON format. The input data is text information that indicates the user's request, and is sent to the server.

[0512] Step 3:

[0513] The server analyzes the received user input data and activates a generative AI model. Specifically, it uses a generative AI model such as OpenAI's GPT-3 to generate optimal operating procedures based on the user's request. The generative AI model generates appropriate procedures for the request entered by the user and outputs the results in text format to the server.

[0514] Step 4:

[0515] The server converts the generated specific operation procedures into a GUI format. Specifically, it converts the generated textual procedures into a visually understandable format and builds them as GUI components. The converted procedures are saved as output on the server.

[0516] Step 5:

[0517] The server sends specific operation procedures in GUI format to the terminal. The data is sent in JSON format again using an HTTP response. The output data is the operation procedures converted into GUI format, and this is sent transparently to the terminal.

[0518] Step 6:

[0519] The specific operation procedures received by the terminal are displayed on the GUI, allowing the user to check the operation procedures visually rather than in text format. The received data is analyzed and rendered as GUI components.

[0520] Step 7:

[0521] A user operates a software application by following instructions displayed on a GUI. The user follows the instructions to perform specific operations, such as entering inventory data or performing analytical procedures.

[0522] Step 8:

[0523] The device uses an emotion engine to recognize the user's emotions based on the input data and actions of the user during operation. Specifically, the emotion recognition algorithm analyzes the keyboard input and touch operations performed by the user on the smartphone. This emotion data is saved as input data on the device.

[0524] Step 9:

[0525] The device sends the analyzed emotional data to the server. The emotional data is also sent in JSON format via an HTTP request. The input data indicates the user's emotional state and is sent to the server.

[0526] Step 10:

[0527] The server adjusts the generated operation procedures based on the emotional data. Specifically, to reduce the anxiety and confusion felt by the user, the server adjusts the procedures generated by the generative AI model and takes measures such as providing more detailed explanations. These adjusted procedures are saved as output data on the server.

[0528] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0529] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0530] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0531] [Second embodiment]

[0532] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0533] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0534] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0535] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0536] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0537] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0538] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0539] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0540] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0541] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0542] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0543] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0544] The present invention is a system in which a generative AI model automatically presents specific operational procedures when a user simply inputs the desired final output in data analysis work using software applications such as Excel or Google Spreadsheets. The following describes in detail an embodiment of this system.

[0545] Overall flow of the system program

[0546] User Input

[0547] The user enters the desired output in text format into an input field in Excel or Google Spreadsheet. For example, they can enter a specific request such as "aggregate sales data and create a monthly sales graph."

[0548] Sending input data

[0549] The terminal sends the user's input data to the server using an HTTP request. The input data is sent in JSON format.

[0550] Using generative AI models

[0551] The server analyzes the received user input data and activates a generative AI model. This generative AI model generates optimal operation procedures based on the user's request. For example, it generates specific instructions such as how to select a data range, how to create a pivot table, and how to insert a graph.

[0552] Converting the steps to a GUI format

[0553] The server converts the generated specific operation procedures into a GUI format, which provides the procedures in a format that is visually easy for the user to understand.

[0554] Sending procedure data

[0555] The server then sends the procedure data converted into GUI format to the terminal, again using an HTTP response, with the procedure data being sent in JSON format.

[0556] Displaying procedures on the GUI

[0557] The terminal analyzes the received instruction data and displays specific operation steps on the GUI, such as "Select a range" or "Click the Insert tab and select a pivot table."

[0558] User operations

[0559] Users follow the instructions displayed on the device's GUI to perform operations in Excel or Google Spreadsheets, allowing them to easily perform highly accurate data analysis.

[0560] Specific examples

[0561] Example: Summarizing and graphing sales data

[0562] 1. User Input

[0563] The user enters "aggregate sales data and create a monthly sales graph."

[0564] 2. Sending input data

[0565] The terminal transmits this input data to the server.

[0566] 3. Use of generative AI models

[0567] The server uses the generative AI model to generate the following specific steps:

[0568] Select the data range with dates in column A and sales amounts in column B.

[0569] Click the Insert tab and select PivotTable

[0570] Drag Date to Rows and Sales Amount to Values

[0571] Use the aggregated data to create a monthly bar chart

[0572] 4. Converting the procedure to GUI format

[0573] The server translates these steps into a GUI format.

[0574] 5. Sending procedure data

[0575] The server transmits the procedure data converted into GUI format to the terminal.

[0576] 6. Displaying procedures on the GUI

[0577] The terminal analyzes the received procedure data and displays the procedure on the GUI.

[0578] "Please select a data range with dates in column A and sales amounts in column B."

[0579] Click the Insert tab and select PivotTable.

[0580] "Drag the dates to the rows and the sales amounts to the values."

[0581] "Create a bar graph for each month."

[0582] 7. User Operations

[0583] The user follows the displayed steps to perform operations in Excel and complete a monthly sales graph.

[0584] In this way, the present invention is a system that improves the efficiency and accuracy of data analysis by automatically generating the procedures required for a user to perform data analysis and allowing intuitive operation.

[0585] The processing flow will be explained below.

[0586] Step 1:

[0587] The user enters the final output they want to achieve using Excel or Google Spreadsheets (e.g., "aggregate sales data and create a monthly sales graph") in text format into the tool's input field.

[0588] Step 2:

[0589] The device receives the user's input data and sends it to the server in JSON format via an HTTP request.

[0590] Step 3:

[0591] The server analyzes the received user input data and activates a generative AI model, which generates optimal operating procedures based on the user's requests.

[0592] Step 4:

[0593] The server converts the generated specific operation procedures into a GUI format, where the procedures output by the generative AI model are converted into a visually displayable format.

[0594] Step 5:

[0595] The server sends the procedure data converted into GUI format to the terminal. The data is sent in JSON format via an HTTP response.

[0596] Step 6:

[0597] The device analyzes the received procedure data and visually displays specific operation steps on the GUI, such as "Select a data range" or "Click the Insert tab and select a pivot table."

[0598] Step 7:

[0599] The user follows the instructions on the GUI to perform operations on Excel or Google Spreadsheets. The user follows the instructions displayed step by step to select a data range, create a pivot table, insert a chart, and so on.

[0600] Step 8:

[0601] The device collects feedback from the user during or after the operation is completed. The feedback is collected in the form of a questionnaire about whether the procedure was useful or not, what improvements can be made, etc.

[0602] Step 9:

[0603] The device sends the collected feedback to the server, which sends the feedback information in JSON format via an HTTP request.

[0604] Step 10:

[0605] The server analyzes the collected feedback and uses it to improve the generative AI model. Based on the feedback, the algorithms and rules of the generative AI model are improved, resulting in more accurate procedure generation.

[0606] Example 1

[0607] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0608] In traditional data analysis work, users often have to understand complex operating procedures in applications such as Excel or Google Spreadsheets and perform the operations manually. This tends to result in inefficient data analysis and low accuracy. Furthermore, users spend a lot of time understanding the necessary operating procedures, which reduces productivity. Furthermore, errors in operating procedures are likely to occur, resulting in a risk of lower quality.

[0609] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0610] In this invention, the server includes: a means for a user to input a final output in text format in an application including an information set to be operated; a means for the terminal to transmit the user's input data to the processing device; a means for the processing device to generate specific operating procedures based on the user's request using a generative artificial intelligence model; a means for the processing device to convert the generated specific operating procedures into a visual display format; a means for the processing device to transmit the specific operating procedures in the visual display format to the terminal; a means for the terminal to display the received specific operating procedures on a visual user interface; and a means for the user to operate the application according to the operating procedures displayed on the visual user interface. This allows the user to easily understand and accurately execute the specific operating procedures. This significantly improves the efficiency and accuracy of data analysis and increases user productivity.

[0611] "User" refers to a user who utilizes the system to work with an application that includes an information set to be manipulated.

[0612] "Information set" refers to the entire collection of data that is the subject of analysis and processing.

[0613] "Application" refers to software that users use to analyze and manipulate data, such as Excel or Google Sheets.

[0614] "Output" refers to the final result or outcome that the user wants to achieve.

[0615] "Terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[0616] "Processing device" refers to a server or cloud computing environment that analyzes input data sent by a user, activates a generative AI model, and generates specific operating procedures.

[0617] "Generative artificial intelligence models" refer to AI technologies such as machine learning algorithms and large-scale language models that generate optimal operating procedures based on user requests.

[0618] The "visual display format" refers to a format in which the generated specific operating procedures are visualized so that the user can easily understand them.

[0619] "Visual user interface" refers to a graphical interface that allows a user to view and execute operating procedures.

[0620] "Feedback" refers to opinions and information provided by users regarding operating procedures.

[0621] The present invention is a system in which a generative AI model automatically presents specific operational procedures when a user simply inputs the desired final output in data analysis work using applications such as Excel or Google Spreadsheets. The following describes in detail an embodiment of this system.

[0622] User Input

[0623] The user inputs in text form the final output they want to achieve from the application containing the information set to be manipulated, for example, a request such as "aggregate sales data and create a monthly sales graph."

[0624] Sending input data

[0625] The terminal sends the user's input data to the processing device. This transmission uses an HTTP request, and the data is converted into JSON format. Specifically, the terminal converts the user's input "aggregate sales data and create a monthly sales graph" into JSON data ({"request": "aggregate sales data and create a monthly sales graph"}) and sends it.

[0626] Using generative AI models

[0627] The server analyzes the received input data and activates a generative AI model, which generates operational procedures based on the user's request. For example, it generates the following procedures:

[0628] Select the data range with dates in column A and sales amounts in column B.

[0629] Click the Insert tab and select PivotTable

[0630] Drag Date to Rows and Sales Amount to Values

[0631] Use the aggregated data to create a monthly bar chart

[0632] Converting the steps to a GUI format

[0633] The server converts the generated specific operation instructions into a visual display format, providing the instructions in a form that is easy for the user to understand visually. For example, it generates GUI code including instructions such as "Please select a data range with dates in column A and sales amounts in column B."

[0634] Sending procedure data

[0635] The server then sends the procedure data converted into a visual display format to the terminal, again using an HTTP response, with the data being sent in JSON format.

[0636] Displaying procedures on the GUI

[0637] The terminal analyzes the received instruction data and displays specific operation instructions on the GUI. For example, the terminal visually displays instructions such as "Select a data range with dates in column A and sales amounts in column B," "Click the Insert tab," "Select a pivot table," "Drag dates to rows," "Drag sales amounts to values," and "Create a bar graph by month."

[0638] User operations

[0639] Users follow the instructions displayed on the device's GUI to perform operations in Excel or Google Spreadsheets, allowing them to easily perform highly accurate data analysis.

[0640] Specific examples

[0641] Example: Summarizing and graphing sales data

[0642] 1. User Input

[0643] The user enters "aggregate sales data and create a monthly sales graph."

[0644] 2. Sending input data

[0645] The terminal transmits this input data to the processing unit.

[0646] 3. Use of generative AI models

[0647] The server uses the generative AI model to generate the following specific steps:

[0648] Select the data range with dates in column A and sales amounts in column B.

[0649] Click the Insert tab and select PivotTable

[0650] Drag Date to Rows and Sales Amount to Values

[0651] Use the aggregated data to create a monthly bar chart

[0652] 4. Converting the procedure to GUI format

[0653] The server translates these instructions into a visual display format.

[0654] 5. Sending procedure data

[0655] The server transmits the procedure data converted into a visual display format to the terminal.

[0656] 6. Displaying procedures on the GUI

[0657] The terminal analyzes the received procedure data and displays the procedure on the GUI.

[0658] "Please select a data range with dates in column A and sales amounts in column B."

[0659] Click the Insert tab and select PivotTable.

[0660] "Drag the dates to the rows and the sales amounts to the values."

[0661] "Create a bar graph for each month."

[0662] 7. User Operations

[0663] The user follows the displayed steps to perform operations in Excel and complete a monthly sales graph.

[0664] In this way, the present invention is a system that improves the efficiency and accuracy of data analysis by automatically generating the procedures required for users to perform data analysis and allowing them to operate intuitively.

[0665] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0666] Step 1:

[0667] (user input)

[0668] How it works: The user enters a textual description of the final output they want to achieve from the application, including the set of information they want to operate on. For example, they might enter "aggregate sales data and create a graph of sales by month."

[0669] Input: The user enters the request in text format into an Excel or Google Spreadsheet.

[0670] Output: The user's request is saved as text in the input field.

[0671] Step 2:

[0672] (Send input data)

[0673] How it works: The device converts the user's input data into JSON format and sends it to the server using an HTTP request.

[0674] Input: The request entered by the user in text format.

[0675] Data processing: Convert text data into JSON format.

[0676] Output: The data converted to JSON format is sent to the server via an HTTP request.

[0677] Step 3:

[0678] (Using generative AI models)

[0679] How it works: The server parses the received JSON data and inputs it as a prompt to the generative AI model, which then generates the optimal operation procedure based on the user's request.

[0680] Input: User request data in JSON format (e.g., {"request": "Aggregate sales data and create a monthly sales chart"}).

[0681] Data calculation: Analyzes JSON data and inputs it as prompts to the generative AI model to generate specific operating procedures.

[0682] Output: Generated instructions (e.g. "Select the data range with dates in column A and sales amounts in column B").

[0683] Step 4:

[0684] (Converting the procedure into GUI format)

[0685] How it works: The server analyzes the generated operating instructions and converts them into a format that is easy for the user to understand visually.

[0686] Input: Specific operational instructions generated by a generative AI model.

[0687] Data processing: The generated operating procedures are converted into a visual display format (GUI) using HTML, CSS, and JavaScript.

[0688] Output: Procedure data in GUI format.

[0689] Step 5:

[0690] (Send procedure data)

[0691] Operation: The server sends the procedure data converted into GUI format to the terminal using an HTTP response.

[0692] Input: Specific operation procedure data in GUI format.

[0693] Data processing: Convert GUI format data into JSON format.

[0694] Output: The data converted to JSON format is sent to the terminal as an HTTP response.

[0695] Step 6:

[0696] (Procedure displayed on GUI)

[0697] Operation: The terminal analyzes the received procedure data and displays specific operating procedures on the GUI.

[0698] Input: GUI steps data in JSON format (e.g., {"steps": ["Select a data range with dates in column A and sales amounts in column B", "Click the Insert tab", "Select PivotTable", "Drag Dates to Rows", "Drag Sales Amounts to Values", "Create a bar chart by month"]}).

[0699] Data operations: Parse the JSON data and process it to display it on a visual user interface.

[0700] Output: Specific operation steps visualized on the GUI.

[0701] Step 7:

[0702] (User operation)

[0703] How it works: Users follow the on-screen instructions to perform operations in Excel or Google Sheets. Specifically, they are instructed to "select a data range with dates in column A and sales amounts in column B," "click the Insert tab," "select PivotTable," "drag dates to rows," "drag sales amounts to values," and "create a bar graph by month."

[0704] Input: Operation steps displayed on the GUI.

[0705] Data processing: Users manually process data in Excel or Google Spreadsheets according to instructions.

[0706] Output: The final output desired by the user (e.g., a graph of sales by month).

[0707] (Application example 1)

[0708] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0709] Traditional data analysis and manufacturing process optimization required a great deal of specialized knowledge and time, resulting in reduced work efficiency on the factory floor. Furthermore, if workers were not familiar with data analysis and process improvement, it was difficult to obtain accurate analysis results and effective improvement measures. Furthermore, there was also the issue of workers finding it difficult to understand and apply procedures in real time.

[0710] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0711] In this invention, the server includes: a means for a user to input a final output in text format in a software application including a dataset to be operated; a means for a terminal to send the user's input data to the server; a means for the server to generate specific operating procedures based on the user's request using a generative AI model; a means for the server to convert the generated specific operating procedures into a GUI format; a means for the server to send the specific operating procedures in GUI format to the terminal; a means for the terminal to display the specific operating procedures received on the GUI; a means for the user to operate the software application in accordance with the operating procedures displayed on the GUI; a means for the user to input voice instructions for data analysis and manufacturing process improvement in a factory and convert them into text format; a means for the server to generate manufacturing data analysis procedures and manufacturing process improvement procedures using the generative AI model and display them in GUI format; and a means for the user to operate systems in the factory according to the instructions displayed on the terminal's GUI. This enables workers without specialized knowledge to efficiently and accurately analyze data and improve manufacturing processes.

[0712] "User" refers to an individual or group of individuals who operate software applications and factory systems.

[0713] "Dataset to be manipulated" refers to a collection of data that a user processes and analyzes within a software application.

[0714] "Software Applications" refers to programs for data processing and analysis, such as Excel and Google Spreadsheets.

[0715] "Final output" refers to the result or goal the user wants to achieve.

[0716] "Means of inputting in text format" refers to methods of inputting characters using a keyboard, voice input, etc.

[0717] "Device" refers to devices such as smartphones, tablets, and smart glasses.

[0718] "Server" refers to a remote computing device that processes generative AI models and manages data.

[0719] A "generative AI model" refers to an artificial intelligence program that automatically generates specific operating procedures based on user requests.

[0720] "Specific operating procedures" refer to the specific steps that the user must actually take, such as selecting a data range or creating a graph.

[0721] "GUI format" refers to a graphical user interface format, a display format that is visually easy to understand.

[0722] A "manufacturing process" refers to a series of tasks and procedures for producing a product in a factory.

[0723] "Manufacturing process improvement" refers to measures or attempts to improve the efficiency or quality of a manufacturing process.

[0724] "Means for inputting voice and converting it into text format" refers to a method of converting verbal instructions into text data using voice recognition technology.

[0725] "Systems within the factory" refers to the various machines and programs used in the factory.

[0726] "Feedback" refers to opinions and evaluation information provided by users after using the system.

[0727] "Improvement procedures" refer to specific steps taken to review existing manufacturing processes and data analysis procedures and make them more effective.

[0728] The present invention provides a system for efficiently performing data analysis and manufacturing process improvements in factories. This system allows users to input instructions in the form of voice or text, and uses a generative AI model to provide specific operating procedures. An embodiment of this system is described in detail below.

[0729] System Overview

[0730] This system consists of a user, a terminal, a server, and a generative AI model. The user uses a software application containing the dataset to be operated on and inputs the final output in text format. The terminal sends the user's input data to the server, which uses the generative AI model to generate specific operating procedures. The generated operating procedures are converted into GUI format and sent to the terminal. The user performs operations by following the specific procedures displayed on the terminal.

[0731] Hardware and software used

[0732] Hardware: This system includes smart glasses, tablets, and servers. Specific examples include smart glasses such as Google Glass and Vuzix Blade, and cloud servers such as Amazon Web Services (AWS) and Microsoft Azure.

[0733] Software: The system includes OpenAI GPT-4 as a generative AI model, Python libraries (Pandas, Numpy) for data processing, Google Speech-to-Text API for voice recognition, and React Native and Vue.js for GUI display.

[0734] Specific examples of the system

[0735] 1. User input method:

[0736] Using smart glasses or a tablet, users can input data analysis and improvement instructions for the manufacturing process in voice or text format. For example, they could give instructions such as "Analyze the manufacturing defect rate by month and propose improvement measures."

[0737] 2. Send input data:

[0738] The device converts the voice input into text format and sends the input data in JSON format to the server.

[0739] 3. Use of generative AI models:

[0740] The server launches a generative AI model (e.g., OpenAI GPT-4) to generate specific operating procedures based on the user's request.

[0741] 4. Converting the operating procedure to GUI format:

[0742] The server converts the generated specific operating procedures into GUI format and sends them to the terminal.

[0743] 5. GUI procedure:

[0744] The terminal analyzes the received procedure data and displays specific operation procedures on the GUI, allowing the user to receive instructions in a visually easy-to-understand format.

[0745] 6. User Actions:

[0746] Users follow the procedures displayed on the terminal's GUI to operate the factory's manufacturing system and perform data analysis and process improvements.

[0747] Specific examples

[0748] Example of user input:

[0749] "Analyze the manufacturing defect rate by month and create a graph."

[0750] Example prompts to send to the generative AI model:

[0751] "Please provide the steps to use a generative AI model to analyze the defect rate of manufacturing data by month and generate a graph showing this."

[0752] This enables the system to efficiently and accurately perform data analysis and process improvements at the manufacturing site.

[0753] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0754] Step 1:

[0755] User Input

[0756] The user uses smart glasses or a tablet device to input the dataset to be manipulated and the final output in voice or text format. For example, the user might give instructions such as "Analyze the manufacturing defect rate by month and propose improvements." The input is then converted into text by voice recognition software (e.g., Google Speech-to-Text API) and imported into the device.

[0757] Input data: "Analyze manufacturing defect rates by month and propose improvement measures"

[0758] Output data: Text data converted from audio

[0759] Step 2:

[0760] Sending input data

[0761] The device parses the converted text data into JSON format and sends it to the server using an HTTP request, where the data is sent to the server in JSON format.

[0762] Input data: JSON format text data

[0763] Output: HTTP request sent to the server

[0764] Step 3:

[0765] Using generative AI models

[0766] The server analyzes the received JSON-formatted text data and launches a generative AI model (e.g., OpenAI GPT-4). The server constructs a prompt sentence and requests the generative AI model to generate specific steps for data analysis and manufacturing process improvement.

[0767] Input data: JSON format text data

[0768] Data processing: Generate prompts to input to the generative AI model

[0769] Output data: Specific operating procedures

[0770] Step 4:

[0771] Converting the steps to a GUI format

[0772] The server analyzes the specific operation procedures returned by the generative AI model and converts them into a GUI format that is easy for users to understand visually, using a front-end framework such as React Native or Vue.js.

[0773] Input data: Operation instructions received from the generative AI model

[0774] Data processing: Converting procedure data into GUI format

[0775] Output data: GUI-based specific operation procedures

[0776] Step 5:

[0777] Sending procedure data

[0778] The server parses the specific operation procedures converted into GUI format into JSON format and sends them to the terminal using an HTTP response.

[0779] Input data: GUI-based specific operation procedures

[0780] Output data: HTTP response sent to the device

[0781] Step 6:

[0782] Displaying procedures on the GUI

[0783] The device analyzes the received procedure data and displays it on the screen of the smart glasses or tablet device, allowing the user to visually confirm the specific operation procedures.

[0784] Input data: Operation procedure data received from the server (JSON format)

[0785] Data processing: Rendering data for display on a GUI

[0786] Output data: Operational steps that can be visually confirmed by the user

[0787] Step 7:

[0788] User operations

[0789] Users operate the factory's manufacturing system by following specific operating procedures displayed on the terminal, such as "select a data range," "count the number of defective products," and "generate graphs," to analyze data and improve the manufacturing process.

[0790] Input data: Operation procedure displayed on the GUI

[0791] Output data: Completed data analysis and manufacturing process improvement operations

[0792] This allows all steps to work together, enabling users to analyze data and improve manufacturing processes efficiently and accurately, even without specialized knowledge.

[0793] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0794] This invention relates to a system in which a generative AI model automatically suggests specific operating procedures when a user performs data analysis using a software application, simply by inputting the final output. This system also combines an emotion engine that recognizes the user's emotions. This system allows users to receive more personalized support and perform data analysis efficiently and accurately.

[0795] Overall flow of the system program

[0796] User Input

[0797] The user enters the desired output in text format in the input field in Excel or Google Spreadsheet. For example, they might enter "aggregate sales data and create a monthly sales graph."

[0798] Sending input data

[0799] The terminal receives the user's input data and sends it to the server in JSON format via an HTTP request.

[0800] Using generative AI models

[0801] The server analyzes the received user input data and activates a generative AI model, which generates optimal operating procedures based on the user's requests.

[0802] Converting the steps to a GUI format

[0803] The server converts the generated specific operation procedures into a GUI format, which provides the procedures in a format that is visually easy for the user to understand.

[0804] Sending procedure data

[0805] The server then sends the procedure data converted into GUI format to the terminal. The data is sent in JSON format via an HTTP response.

[0806] Displaying procedures on the GUI

[0807] The terminal analyzes the received procedure data and visually displays specific operation steps on the GUI, such as "Select a data range" or "Click the Insert tab and select a pivot table."

[0808] User operations

[0809] Users follow the instructions displayed on the device's GUI to perform operations in Excel or Google Spreadsheets, enabling them to easily perform highly accurate data analysis.

[0810] Integration with emotion engine

[0811] emotion recognition

[0812] The device uses an emotion engine to recognize the user's emotions based on the user's input data and behavior during operation (e.g., keyboard operations, mouse movements, facial recognition, etc.). For example, it identifies emotional states such as "happiness," "anxiety," and "excitement."

[0813] Sending emotional data

[0814] The device sends the analyzed emotional data to the server in JSON format via an HTTP request.

[0815] Emotion-based procedural adjustments

[0816] The server receives the user's emotional data and adjusts the generated operation procedure based on the user's emotional state. For example, if the user feels "anxious," the server may provide more detailed instructions.

[0817] Gathering feedback and improving the model

[0818] The device collects feedback from the user during or after the operation is completed and sends it to the server. The feedback information is sent in JSON format via an HTTP request, and the server uses this information to improve the generative AI model. Emotional data is also analyzed to improve the accuracy of the generative AI model.

[0819] Specific examples

[0820] Example: Summarizing and graphing sales data

[0821] 1. User Input

[0822] The user enters "aggregate sales data and create a monthly sales graph."

[0823] 2. Sending input data

[0824] The terminal transmits this input data to the server.

[0825] 3. Use of generative AI models

[0826] The server uses the generative AI model to generate the following specific steps:

[0827] Select the data range with dates in column A and sales amounts in column B.

[0828] Click the Insert tab and select PivotTable

[0829] Drag Date to Rows and Sales Amount to Values

[0830] Use the aggregated data to create a monthly bar chart

[0831] 4. Emotion recognition

[0832] As the user performs the operation, the device recognizes the user's emotions based on keyboard operations and mouse movements.

[0833] 5. Sending Emotional Data

[0834] The terminal transmits the user's emotion data to the server.

[0835] 6. Emotion-Based Procedural Adjustments

[0836] The server analyzes the user's emotional data and provides more detailed instructions as needed to help alleviate the anxiety the user feels.

[0837] 7. Displaying procedures on the GUI

[0838] The terminal displays the generated procedures on a GUI, allowing the user to operate them intuitively.

[0839] 8. User Operations

[0840] The user follows the displayed steps to complete the monthly sales graph.

[0841] 9. Gather feedback and improve the model

[0842] After the user completes the operation, the device collects feedback and sends it to the server, which uses it to improve the generative AI model and emotion engine.

[0843] In this way, by combining a generative AI model and an emotion engine, the present invention provides a data analysis support system that is intuitively operable for users and takes emotions into consideration.

[0844] The processing flow will be explained below.

[0845] Step 1:

[0846] The user enters the final output they want to achieve using Excel or Google Spreadsheets (e.g., "aggregate sales data and create a monthly sales graph") in text format into the tool's input field.

[0847] Step 2:

[0848] The device receives the user's input data and sends it to the server in JSON format using an HTTP request.

[0849] Step 3:

[0850] The server analyzes the received user input data and activates a generative AI model, which generates optimal operating procedures based on the user's requests.

[0851] Step 4:

[0852] The server converts the generated specific operation procedures into a GUI format, where the procedures output by the generative AI model are converted into a format that can be displayed visually.

[0853] Step 5:

[0854] The server sends the procedure data converted into GUI format to the terminal. The procedure data is sent in JSON format using an HTTP response.

[0855] Step 6:

[0856] The device analyzes the received instruction data and visually displays specific operation steps on the GUI, such as "Select a data range" or "Click the Insert tab and select a pivot table."

[0857] Step 7:

[0858] The user follows the instructions on the GUI to perform operations on Excel or Google Spreadsheets. The user follows the instructions displayed step by step to select a data range, create a pivot table, insert a chart, and so on.

[0859] Step 8:

[0860] The device uses an emotion engine to recognize the user's emotions based on the user's input data and behavior during operation (e.g., keyboard operations, mouse movements, facial recognition, etc.). For example, it can identify emotional states such as "happiness," "anxiety," and "excitement" based on the user's facial expressions and operation speed.

[0861] Step 9:

[0862] The device sends the analyzed emotion data to the server in JSON format via an HTTP request.

[0863] Step 10:

[0864] The server analyzes the received user emotion data and adjusts the generated operation instructions based on the user's emotional state. For example, if the user is feeling "anxious," the instructions are made more detailed to provide a sense of security.

[0865] Step 11:

[0866] The device will then redisplay the specific operational steps on the GUI, allowing the user to receive support adapted to their emotional state.

[0867] Step 12:

[0868] The user continues to follow the instructions on the GUI and completes the final output (e.g., a monthly sales graph).

[0869] Step 13:

[0870] The device collects feedback from the user during or after the operation is completed. The feedback information is collected in the form of a questionnaire about whether the procedure was useful or not, what needs to be improved, etc.

[0871] Step 14:

[0872] The device sends the collected feedback to the server in JSON format via an HTTP request.

[0873] Step 15:

[0874] The server analyzes the collected feedback and uses it to improve the generative AI model and emotion engine. Based on the feedback, the algorithms and rules of the generative AI model are improved, resulting in more accurate procedure generation and emotion recognition.

[0875] Example 2

[0876] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0877] Modern data analysis tasks often require specialized knowledge for users to perform tasks efficiently and accurately. Furthermore, manually understanding and executing a series of operational steps is laborious and time-consuming. Therefore, a system is needed in which a generative AI model automatically suggests specific operational steps based on the user's input. Furthermore, there is a need to provide more personalized support and improve the user experience by adjusting the steps based on the user's emotions. Current technology does not offer a system that meets all of these requirements, which is the problem this invention aims to solve.

[0878] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0879] In this invention, the server includes means for analyzing user input data and generating specific operating procedures using a generative AI model, means for converting the generated specific operating procedures into a GUI format, and means for adjusting the generated operating procedures based on emotion data transmitted from the terminal in accordance with the user's emotional state. This allows the user to receive visually easy-to-understand specific operating procedures simply by inputting the final output, and furthermore, by taking the user's emotions into consideration, stress and anxiety can be reduced, and data analysis work can proceed more smoothly.

[0880] "User" refers to an individual or organization that uses the system to perform data analysis work.

[0881] A "terminal" is a computer device operated by a user, and refers to a device for inputting, transmitting, receiving, displaying, etc. data.

[0882] "Server" refers to a computer system that uses a generative AI model to generate specific operating procedures and communicates with terminals.

[0883] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates optimal operating procedures based on user requests.

[0884] A "prompt" is a document or instruction that allows a user to input the final output in text form.

[0885] An "operational procedure" refers to a set of specific instructions for a user to correctly perform a data analysis task.

[0886] "GUI format" refers to a graphical user interface format designed to make it easy for users to operate visually.

[0887] "Emotion engine" refers to an artificial intelligence algorithm that analyzes a user's emotional state and improves the user's operating experience.

[0888] "Feedback" refers to information in which a user provides evaluations and opinions on operating procedures.

[0889] The present invention is a system for solving problems users face in data analysis work, and combines a generative AI model and an emotion engine to provide specific operating procedures to users and provide personalized support based on emotions. Detailed embodiments of this system are described below.

[0890] Overall system overview

[0891] This system provides specific operating procedures through the following steps when users perform data analysis using software applications (e.g., Excel or Google Spreadsheets): Multiple hardware and software components, including a server, terminals, generative AI models, and an emotion engine, work together.

[0892] The specific hardware and software used

[0893] 1. Hardware

[0894] Terminal: The device operated by the user (PC, tablet, smartphone, etc.)

[0895] Server: A remote server that hosts the generative AI model and emotion engine.

[0896] 2. Software

[0897] Generative AI model: An AI algorithm that generates specific operating procedures based on user requests.

[0898] Emotion Engine: An AI algorithm for analyzing a user's emotional state

[0899] Software applications: Data analysis tools such as Excel and Google Sheets

[0900] Processing flow

[0901] 1. User Input

[0902] The user enters the final output they want to achieve in the software application in text format. An example prompt might be "Summarize sales data and create a monthly sales graph."

[0903] 2. Sending input data

[0904] The terminal receives the user's input data and sends it to the server in JSON format via an HTTP request.

[0905] 3. Use of generative AI models

[0906] The server analyzes the received user input data and activates a generative AI model, which generates optimal operation procedures based on the user's prompt.

[0907] 4. Converting the procedure to GUI format

[0908] The server converts the generated specific operation procedures into a GUI format.

[0909] 5. Sending procedure data

[0910] The server transmits the procedure data converted into GUI format to the terminal.

[0911] 6. Displaying procedures on the GUI

[0912] The device analyzes the received instruction data and visually displays specific operation steps on the GUI. Specific examples include instructions such as "Select a data range" and "Click the Insert tab and select a pivot table."

[0913] 7. User Operations

[0914] The user follows the instructions displayed on the device's GUI and performs operations on Excel or Google Spreadsheets.

[0915] Integration with emotion engine

[0916] 1. Emotion recognition

[0917] The device uses an emotion engine to recognize the user's emotions based on the user's input data and behavior during operation (keyboard operations, mouse movements, facial recognition, etc.). Recognized emotions include "happiness," "anxiety," and "excitement."

[0918] 2. Sending Emotional Data

[0919] The device sends the analyzed emotion data to the server in JSON format via an HTTP request.

[0920] 3. Emotion-Based Procedural Adjustments

[0921] The server receives the user's emotional data and adjusts the generated operation procedure based on the user's emotional state, for example, if the user is feeling "anxious," the procedure will be explained in more detail.

[0922] 4. Gather feedback and improve the model

[0923] After the user completes the operation, the device collects feedback and sends it to the server, which uses this feedback to improve the accuracy of the generative AI model and emotion engine.

[0924] Specific examples

[0925] If a user types "aggregate sales data and create a monthly sales graph," the following steps will be generated:

[0926] Select the data range with dates in column A and sales amounts in column B.

[0927] Click the Insert tab and select PivotTable

[0928] Drag Date to Rows and Sales Amount to Values

[0929] Use the aggregated data to create a monthly bar chart

[0930] By following these steps, users can easily complete data analysis tasks, and the emotion engine can recognize the user's emotional state and provide appropriate support to further improve the user experience.

[0931] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0932] Step 1: User Input

[0933] The user enters the final output in text format into the software application to be operated. For example, they might enter "aggregate sales data and create a monthly sales graph." This input becomes the prompt statement, which forms the basis for the next step of data processing.

[0934] Input: Text input for final output

[0935] Output: Prompt in text format

[0936] Step 2: Submitting input data

[0937] The terminal receives the user's input data (prompt text) and sends it to the server in JSON format using an HTTP request. Specifically, the prompt text is sent to the server in the format {"task": "Aggregate sales data and create a monthly sales graph"}.

[0938] Input: prompt statement

[0939] Output: JSON format data

[0940] Step 3: Use the generative AI model

[0941] The server analyzes the received prompt and activates a generative AI model. A generative AI model is an algorithm that generates specific operational steps based on the user's request. In response to the prompt "aggregate sales data and create a monthly sales graph," the AI ​​generates the following operational steps:

[0942] Select a data range

[0943] Click the Insert tab

[0944] Select the pivot table

[0945] Drag a date to a row

[0946] Drag Sales Amount to Values

[0947] Create a bar chart

[0948] Input: JSON format prompt statement

[0949] Output: List of specific operating procedures

[0950] Step 4: Converting the Instructions to GUI Format

[0951] The server converts the generated specific operation steps into a GUI format. To make it visually easy for users to understand, each step is formatted as a button or interactive interface. For example, the message "Please select a data range" is displayed as an appropriate button.

[0952] Input: List of operating procedures

[0953] Output: GUI format operation procedure

[0954] Step 5: Sending Instruction Data

[0955] The server sends the GUI-formatted step data to the terminal in JSON format using an HTTP response. For example, the following data is sent: {"steps": ["Select a data range", "Click the Insert tab and select a pivot table"]}.

[0956] Input: GUI-style operation procedure

[0957] Output: Procedure data in JSON format

[0958] Step 6: Displaying the procedure on the GUI

[0959] The terminal analyzes the received procedure data and visually displays specific operation procedures on the GUI. To make it easier for the user to understand, the procedures are displayed on the screen as pop-up windows or information bars. This allows the user to follow the instructions to proceed with the operation.

[0960] Input: Procedure data in JSON format

[0961] Output: Visual operation procedure on GUI

[0962] Step 7: User interaction

[0963] The user follows the instructions displayed on the device's GUI to perform operations on Excel or Google Spreadsheets. Specifically, the user follows the instructions to perform actions such as "select a data range," "click the Insert tab," and "select a pivot table."

[0964] Input: GUI operation procedure

[0965] Output: User completes an action within the software

[0966] Step 8: Emotion Recognition

[0967] The device uses an emotion engine to recognize the user's emotions based on the user's input data and actions (keyboard operations, mouse movements, facial recognition, etc.). For example, it uses a camera and facial recognition software to identify emotional states such as "happiness," "anxiety," and "excitement."

[0968] Input: User behavior data

[0969] Output: User emotion data

[0970] Step 9: Sending Emotion Data

[0971] The device sends the analyzed emotion data to the server in JSON format via an HTTP request. For example, data such as "{"emotion": "anxiety"}" is sent.

[0972] Input: User emotion data

[0973] Output: Emotion data in JSON format

[0974] Step 10: Adjusting procedures based on emotions

[0975] The server receives the user's emotional data and adjusts the generated operation instructions based on the user's emotional state. For example, if the user is feeling "anxious," the server may add more detailed instructions to the operation instructions.

[0976] Input: Emotion data in JSON format

[0977] Output: Adjusted operating procedure

[0978] Step 11: Gather feedback and improve the model

[0979] After the user completes the operation, the device collects feedback and sends it to the server. The user feedback is used to improve the accuracy of the generative AI model and emotion engine. For example, the model can be adjusted based on feedback such as "The instructions were difficult to understand."

[0980] Input: User feedback

[0981] Output: Improved generative AI models and emotion engines

[0982] (Application example 2)

[0983] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0984] In conventional data analysis support systems, when users perform data analysis tasks on software applications, they must decide the specific operating procedures themselves, making the process complicated and difficult to perform efficiently. Furthermore, when users feel emotionally confused or anxious, the system often fails to provide appropriate support, slowing down the progress of the task. In particular, in everyday tasks such as inventory management and sales forecasting in brick-and-mortar stores, where fast and accurate data analysis is required, there is a lack of flexible operational support that responds to the user's emotions.

[0985] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0986] In this invention, the server includes means for generating specific operating procedures based on a user's request using a generative AI model, means for converting the generated specific operating procedures into a GUI format, and means for recognizing the user's emotions using an emotion engine and adjusting the generated operating procedures based on the emotions. This allows the user to receive emotion-sensitive support while quickly and accurately analyzing data, enabling efficient inventory management and sales forecasting in physical stores.

[0987] A "target information set" is a data set that a user can manipulate and analyze within a software application.

[0988] "Final output" refers to the result that a user wants to achieve using a software application.

[0989] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[0990] The "server" is a central processing system that processes data received from users and provides operational instructions and support using generative AI models and emotion engines.

[0991] A "generative AI model" refers to an artificial intelligence model that generates specific operating procedures based on user requests.

[0992] "GUI format" refers to a format for visually displaying operating procedures in the form of a graphical user interface.

[0993] An "emotion engine" is a system that recognizes a user's emotions based on input data and actions during user operation and analyzes that data.

[0994] "Emotion data" is data that indicates the user's emotional state analyzed by the emotion engine.

[0995] "Feedback" refers to the opinions, evaluations, and data regarding operations that users provide to the system.

[0996] "Data analysis" is the act of processing a set of information to extract patterns and useful information to produce a final output.

[0997] "Inventory management" refers to the task of understanding the inventory status of products, merchandise, and other materials in physical stores and maintaining an adequate supply.

[0998] "Sales forecasting" is the activity of predicting future sales based on past sales data and current sales trends.

[0999] This invention is a system in which, when a user performs data analysis using a software application that manipulates a set of information, the generative AI model automatically presents specific operating procedures simply by inputting the final output, and further combines this with an emotion engine that recognizes the user's emotions.

[1000] First, the user uses the terminal to input the final output in text format. For example, they might input "I want to forecast this month's inventory" or "I want to identify best-selling products." The terminal then sends this input data in JSON format to the server. Communication is via HTTP requests.

[1001] The server analyzes the received user input data and activates a generative AI model. The generative AI model generates optimal operating procedures based on the user's requests. For example, for inventory forecasting, it presents analytical procedures based on historical sales data and current inventory data. The generated specific operating procedures are converted into GUI format by the server. This conversion provides the procedures in a format that is visually easy for users to understand.

[1002] Furthermore, the device uses an emotion engine to recognize the user's emotions based on input data and actions (e.g., keyboard input and touch operations) during operation. The recognized emotion data is sent to the server in JSON format. The server analyzes the emotion data and adjusts the generated operation instructions to reduce the user's anxiety or confusion. For example, if the user feels anxious, it may provide more detailed explanations of the instructions.

[1003] The terminal receives specific operating instructions from the server and displays them on a GUI. The user operates the software application according to the displayed instructions. This allows the user to perform data analysis efficiently and accurately by following the guide.

[1004] After providing the operation instructions, the server collects feedback from the user and uses it to improve the accuracy of the generative AI model and emotion engine. The feedback information is also sent to the server in JSON format.

[1005] Specifically, a smartphone is used as the terminal, and a generative AI model such as OpenAI's GPT-3 is used on the server side. As an emotion engine, a machine learning framework such as Keras is used to analyze the user's facial expressions and input data.

[1006] (Example)

[1007] Here is an example prompt:

[1008] Example prompt sentence:

[1009] User request: I want to make inventory forecast for this month. Please provide specific steps.

[1010] Based on this prompt, a generative AI model generates instructions, and an emotion engine recognizes the user's emotions and adjusts the instructions accordingly, providing personalized assistance to the user.

[1011] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1012] Step 1:

[1013] A user inputs the final output in text format into a software application containing the set of information to be operated on. Specifically, using a smartphone as a terminal, the user inputs requests such as "I want to forecast this month's inventory" or "I want to identify best-selling products." The input data is saved in text format on the terminal.

[1014] Step 2:

[1015] The terminal sends the user's input data to the server. Communication is via HTTP request, and the data is sent in JSON format. The input data is text information that indicates the user's request, and is sent to the server.

[1016] Step 3:

[1017] The server analyzes the received user input data and activates a generative AI model. Specifically, it uses a generative AI model such as OpenAI's GPT-3 to generate optimal operating procedures based on the user's request. The generative AI model generates appropriate procedures for the request entered by the user and outputs the results in text format to the server.

[1018] Step 4:

[1019] The server converts the generated specific operation procedures into a GUI format. Specifically, it converts the generated textual procedures into a visually understandable format and builds them as GUI components. The converted procedures are saved as output on the server.

[1020] Step 5:

[1021] The server sends specific operation procedures in GUI format to the terminal. The data is sent in JSON format again using an HTTP response. The output data is the operation procedures converted into GUI format, and this is sent transparently to the terminal.

[1022] Step 6:

[1023] The specific operation procedures received by the terminal are displayed on the GUI, allowing the user to check the operation procedures visually rather than in text format. The received data is analyzed and rendered as GUI components.

[1024] Step 7:

[1025] A user operates a software application by following instructions displayed on a GUI. The user follows the instructions to perform specific operations, such as entering inventory data or performing analytical procedures.

[1026] Step 8:

[1027] The device uses an emotion engine to recognize the user's emotions based on the input data and actions of the user during operation. Specifically, the emotion recognition algorithm analyzes the keyboard input and touch operations performed by the user on the smartphone. This emotion data is saved as input data on the device.

[1028] Step 9:

[1029] The device sends the analyzed emotional data to the server. The emotional data is also sent in JSON format via an HTTP request. The input data indicates the user's emotional state and is sent to the server.

[1030] Step 10:

[1031] The server adjusts the generated operation procedures based on the emotional data. Specifically, to reduce the anxiety and confusion felt by the user, the server adjusts the procedures generated by the generative AI model and takes measures such as providing more detailed explanations. These adjusted procedures are saved as output data on the server.

[1032] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1033] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1034] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1035] [Third embodiment]

[1036] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1037] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1038] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1039] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1040] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1041] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1042] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1043] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1044] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1045] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1046] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1047] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1048] The present invention is a system in which a generative AI model automatically presents specific operational procedures when a user simply inputs the desired final output in data analysis work using software applications such as Excel or Google Spreadsheets. The following describes in detail an embodiment of this system.

[1049] Overall flow of the system program

[1050] User Input

[1051] The user enters the desired output in text format into an input field in Excel or Google Spreadsheet. For example, they can enter a specific request such as "aggregate sales data and create a monthly sales graph."

[1052] Sending input data

[1053] The terminal sends the user's input data to the server using an HTTP request. The input data is sent in JSON format.

[1054] Using generative AI models

[1055] The server analyzes the received user input data and activates a generative AI model. This generative AI model generates optimal operation procedures based on the user's request. For example, it generates specific instructions such as how to select a data range, how to create a pivot table, and how to insert a graph.

[1056] Converting the steps to a GUI format

[1057] The server converts the generated specific operation procedures into a GUI format, which provides the procedures in a format that is visually easy for the user to understand.

[1058] Sending procedure data

[1059] The server then sends the procedure data converted into GUI format to the terminal, again using an HTTP response, with the procedure data being sent in JSON format.

[1060] Displaying procedures on the GUI

[1061] The terminal analyzes the received instruction data and displays specific operation steps on the GUI, such as "Select a range" or "Click the Insert tab and select a pivot table."

[1062] User operations

[1063] Users follow the instructions displayed on the device's GUI to perform operations in Excel or Google Spreadsheets, allowing them to easily perform highly accurate data analysis.

[1064] Specific examples

[1065] Example: Summarizing and graphing sales data

[1066] 1. User Input

[1067] The user enters "aggregate sales data and create a monthly sales graph."

[1068] 2. Sending input data

[1069] The terminal transmits this input data to the server.

[1070] 3. Use of generative AI models

[1071] The server uses the generative AI model to generate the following specific steps:

[1072] Select the data range with dates in column A and sales amounts in column B.

[1073] Click the Insert tab and select PivotTable

[1074] Drag Date to Rows and Sales Amount to Values

[1075] Use the aggregated data to create a monthly bar chart

[1076] 4. Converting the procedure to GUI format

[1077] The server translates these steps into a GUI format.

[1078] 5. Sending procedure data

[1079] The server transmits the procedure data converted into GUI format to the terminal.

[1080] 6. Displaying procedures on the GUI

[1081] The terminal analyzes the received procedure data and displays the procedure on the GUI.

[1082] "Please select a data range with dates in column A and sales amounts in column B."

[1083] Click the Insert tab and select PivotTable.

[1084] "Drag the dates to the rows and the sales amounts to the values."

[1085] "Create a bar graph for each month."

[1086] 7. User Operations

[1087] The user follows the displayed steps to perform operations in Excel and complete a monthly sales graph.

[1088] In this way, the present invention is a system that improves the efficiency and accuracy of data analysis by automatically generating the procedures required for a user to perform data analysis and allowing intuitive operation.

[1089] The processing flow will be explained below.

[1090] Step 1:

[1091] The user enters the final output they want to achieve using Excel or Google Spreadsheets (e.g., "aggregate sales data and create a monthly sales graph") in text format into the tool's input field.

[1092] Step 2:

[1093] The device receives the user's input data and sends it to the server in JSON format via an HTTP request.

[1094] Step 3:

[1095] The server analyzes the received user input data and activates a generative AI model, which generates optimal operating procedures based on the user's requests.

[1096] Step 4:

[1097] The server converts the generated specific operation procedures into a GUI format, where the procedures output by the generative AI model are converted into a visually displayable format.

[1098] Step 5:

[1099] The server sends the procedure data converted into GUI format to the terminal. The data is sent in JSON format via an HTTP response.

[1100] Step 6:

[1101] The device analyzes the received procedure data and visually displays specific operation steps on the GUI, such as "Select a data range" or "Click the Insert tab and select a pivot table."

[1102] Step 7:

[1103] The user follows the instructions on the GUI to perform operations on Excel or Google Spreadsheets. The user follows the instructions displayed step by step to select a data range, create a pivot table, insert a chart, and so on.

[1104] Step 8:

[1105] The device collects feedback from the user during or after the operation is completed. The feedback is collected in the form of a questionnaire about whether the procedure was useful or not, what improvements can be made, etc.

[1106] Step 9:

[1107] The device sends the collected feedback to the server, which sends the feedback information in JSON format via an HTTP request.

[1108] Step 10:

[1109] The server analyzes the collected feedback and uses it to improve the generative AI model. Based on the feedback, the algorithms and rules of the generative AI model are improved, resulting in more accurate procedure generation.

[1110] Example 1

[1111] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1112] In traditional data analysis work, users often have to understand complex operating procedures in applications such as Excel or Google Spreadsheets and perform the operations manually. This tends to result in inefficient data analysis and low accuracy. Furthermore, users spend a lot of time understanding the necessary operating procedures, which reduces productivity. Furthermore, errors in operating procedures are likely to occur, resulting in a risk of lower quality.

[1113] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1114] In this invention, the server includes: a means for a user to input a final output in text format in an application including an information set to be operated; a means for the terminal to transmit the user's input data to the processing device; a means for the processing device to generate specific operating procedures based on the user's request using a generative artificial intelligence model; a means for the processing device to convert the generated specific operating procedures into a visual display format; a means for the processing device to transmit the specific operating procedures in the visual display format to the terminal; a means for the terminal to display the received specific operating procedures on a visual user interface; and a means for the user to operate the application according to the operating procedures displayed on the visual user interface. This allows the user to easily understand and accurately execute the specific operating procedures. This significantly improves the efficiency and accuracy of data analysis and increases user productivity.

[1115] "User" refers to a user who utilizes the system to work with an application that includes an information set to be manipulated.

[1116] "Information set" refers to the entire collection of data that is the subject of analysis and processing.

[1117] "Application" refers to software that users use to analyze and manipulate data, such as Excel or Google Sheets.

[1118] "Output" refers to the final result or outcome that the user wants to achieve.

[1119] "Terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[1120] "Processing device" refers to a server or cloud computing environment that analyzes input data sent by a user, activates a generative AI model, and generates specific operating procedures.

[1121] "Generative artificial intelligence models" refer to AI technologies such as machine learning algorithms and large-scale language models that generate optimal operating procedures based on user requests.

[1122] The "visual display format" refers to a format in which the generated specific operating procedures are visualized so that the user can easily understand them.

[1123] "Visual user interface" refers to a graphical interface that allows a user to view and execute operating procedures.

[1124] "Feedback" refers to opinions and information provided by users regarding operating procedures.

[1125] The present invention is a system in which a generative AI model automatically presents specific operational procedures when a user simply inputs the desired final output in data analysis work using applications such as Excel or Google Spreadsheets. The following describes in detail an embodiment of this system.

[1126] User Input

[1127] The user inputs in text form the final output they want to achieve from the application containing the information set to be manipulated, for example, a request such as "aggregate sales data and create a monthly sales graph."

[1128] Sending input data

[1129] The terminal sends the user's input data to the processing device. This transmission uses an HTTP request, and the data is converted into JSON format. Specifically, the terminal converts the user's input "aggregate sales data and create a monthly sales graph" into JSON data ({"request": "aggregate sales data and create a monthly sales graph"}) and sends it.

[1130] Using generative AI models

[1131] The server analyzes the received input data and activates a generative AI model, which generates operational procedures based on the user's request. For example, it generates the following procedures:

[1132] Select the data range with dates in column A and sales amounts in column B.

[1133] Click the Insert tab and select PivotTable

[1134] Drag Date to Rows and Sales Amount to Values

[1135] Use the aggregated data to create a monthly bar chart

[1136] Converting the steps to a GUI format

[1137] The server converts the generated specific operation instructions into a visual display format, providing the instructions in a form that is easy for the user to understand visually. For example, it generates GUI code including instructions such as "Please select a data range with dates in column A and sales amounts in column B."

[1138] Sending procedure data

[1139] The server then sends the procedure data converted into a visual display format to the terminal, again using an HTTP response, with the data being sent in JSON format.

[1140] Displaying procedures on the GUI

[1141] The terminal analyzes the received instruction data and displays specific operation instructions on the GUI. For example, the terminal visually displays instructions such as "Select a data range with dates in column A and sales amounts in column B," "Click the Insert tab," "Select a pivot table," "Drag dates to rows," "Drag sales amounts to values," and "Create a bar graph by month."

[1142] User operations

[1143] Users follow the instructions displayed on the device's GUI to perform operations in Excel or Google Spreadsheets, allowing them to easily perform highly accurate data analysis.

[1144] Specific examples

[1145] Example: Summarizing and graphing sales data

[1146] 1. User Input

[1147] The user enters "aggregate sales data and create a monthly sales graph."

[1148] 2. Sending input data

[1149] The terminal transmits this input data to the processing unit.

[1150] 3. Use of generative AI models

[1151] The server uses the generative AI model to generate the following specific steps:

[1152] Select the data range with dates in column A and sales amounts in column B.

[1153] Click the Insert tab and select PivotTable

[1154] Drag Date to Rows and Sales Amount to Values

[1155] Use the aggregated data to create a monthly bar chart

[1156] 4. Converting the procedure to GUI format

[1157] The server translates these instructions into a visual display format.

[1158] 5. Sending procedure data

[1159] The server transmits the procedure data converted into a visual display format to the terminal.

[1160] 6. Displaying procedures on the GUI

[1161] The terminal analyzes the received procedure data and displays the procedure on the GUI.

[1162] "Please select a data range with dates in column A and sales amounts in column B."

[1163] Click the Insert tab and select PivotTable.

[1164] "Drag the dates to the rows and the sales amounts to the values."

[1165] "Create a bar graph for each month."

[1166] 7. User Operations

[1167] The user follows the displayed steps to perform operations in Excel and complete a monthly sales graph.

[1168] In this way, the present invention is a system that improves the efficiency and accuracy of data analysis by automatically generating the procedures required for users to perform data analysis and allowing them to operate intuitively.

[1169] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1170] Step 1:

[1171] (user input)

[1172] How it works: The user enters a textual description of the final output they want to achieve from the application, including the set of information they want to operate on. For example, they might enter "aggregate sales data and create a graph of sales by month."

[1173] Input: The user enters the request in text format into an Excel or Google Spreadsheet.

[1174] Output: The user's request is saved as text in the input field.

[1175] Step 2:

[1176] (Send input data)

[1177] How it works: The device converts the user's input data into JSON format and sends it to the server using an HTTP request.

[1178] Input: The request entered by the user in text format.

[1179] Data processing: Convert text data into JSON format.

[1180] Output: The data converted to JSON format is sent to the server via an HTTP request.

[1181] Step 3:

[1182] (Using generative AI models)

[1183] How it works: The server parses the received JSON data and inputs it as a prompt to the generative AI model, which then generates the optimal operation procedure based on the user's request.

[1184] Input: User request data in JSON format (e.g., {"request": "Aggregate sales data and create a monthly sales chart"}).

[1185] Data calculation: Analyzes JSON data and inputs it as prompts to the generative AI model to generate specific operating procedures.

[1186] Output: Generated instructions (e.g. "Select the data range with dates in column A and sales amounts in column B").

[1187] Step 4:

[1188] (Converting the procedure into GUI format)

[1189] How it works: The server analyzes the generated operating instructions and converts them into a format that is easy for the user to understand visually.

[1190] Input: Specific operational instructions generated by a generative AI model.

[1191] Data processing: The generated operating procedures are converted into a visual display format (GUI) using HTML, CSS, and JavaScript.

[1192] Output: Procedure data in GUI format.

[1193] Step 5:

[1194] (Send procedure data)

[1195] Operation: The server sends the procedure data converted into GUI format to the terminal using an HTTP response.

[1196] Input: Specific operation procedure data in GUI format.

[1197] Data processing: Convert GUI format data into JSON format.

[1198] Output: The data converted to JSON format is sent to the terminal as an HTTP response.

[1199] Step 6:

[1200] (Procedure displayed on GUI)

[1201] Operation: The terminal analyzes the received procedure data and displays specific operating procedures on the GUI.

[1202] Input: GUI steps data in JSON format (e.g., {"steps": ["Select a data range with dates in column A and sales amounts in column B", "Click the Insert tab", "Select PivotTable", "Drag Dates to Rows", "Drag Sales Amounts to Values", "Create a bar chart by month"]}).

[1203] Data operations: Parse the JSON data and process it to display it on a visual user interface.

[1204] Output: Specific operation steps visualized on the GUI.

[1205] Step 7:

[1206] (User operation)

[1207] How it works: Users follow the on-screen instructions to perform operations in Excel or Google Sheets. Specifically, they are instructed to "select a data range with dates in column A and sales amounts in column B," "click the Insert tab," "select PivotTable," "drag dates to rows," "drag sales amounts to values," and "create a bar graph by month."

[1208] Input: Operation steps displayed on the GUI.

[1209] Data processing: Users manually process data in Excel or Google Spreadsheets according to instructions.

[1210] Output: The final output desired by the user (e.g., a graph of sales by month).

[1211] (Application example 1)

[1212] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1213] Traditional data analysis and manufacturing process optimization required a great deal of specialized knowledge and time, resulting in reduced work efficiency on the factory floor. Furthermore, if workers were not familiar with data analysis and process improvement, it was difficult to obtain accurate analysis results and effective improvement measures. Furthermore, there was also the issue of workers finding it difficult to understand and apply procedures in real time.

[1214] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1215] In this invention, the server includes: a means for a user to input a final output in text format in a software application including a dataset to be operated; a means for a terminal to send the user's input data to the server; a means for the server to generate specific operating procedures based on the user's request using a generative AI model; a means for the server to convert the generated specific operating procedures into a GUI format; a means for the server to send the specific operating procedures in GUI format to the terminal; a means for the terminal to display the specific operating procedures received on the GUI; a means for the user to operate the software application in accordance with the operating procedures displayed on the GUI; a means for the user to input voice instructions for data analysis and manufacturing process improvement in a factory and convert them into text format; a means for the server to generate manufacturing data analysis procedures and manufacturing process improvement procedures using the generative AI model and display them in GUI format; and a means for the user to operate systems in the factory according to the instructions displayed on the terminal's GUI. This enables workers without specialized knowledge to efficiently and accurately analyze data and improve manufacturing processes.

[1216] "User" refers to an individual or group of individuals who operate software applications and factory systems.

[1217] "Dataset to be manipulated" refers to a collection of data that a user processes and analyzes within a software application.

[1218] "Software Applications" refers to programs for data processing and analysis, such as Excel and Google Spreadsheets.

[1219] "Final output" refers to the result or goal the user wants to achieve.

[1220] "Means of inputting in text format" refers to methods of inputting characters using a keyboard, voice input, etc.

[1221] "Device" refers to devices such as smartphones, tablets, and smart glasses.

[1222] "Server" refers to a remote computing device that processes generative AI models and manages data.

[1223] A "generative AI model" refers to an artificial intelligence program that automatically generates specific operating procedures based on user requests.

[1224] "Specific operating procedures" refer to the specific steps that the user must actually take, such as selecting a data range or creating a graph.

[1225] "GUI format" refers to a graphical user interface format, a display format that is visually easy to understand.

[1226] A "manufacturing process" refers to a series of tasks and procedures for producing a product in a factory.

[1227] "Manufacturing process improvement" refers to measures or attempts to improve the efficiency or quality of a manufacturing process.

[1228] "Means for inputting voice and converting it into text format" refers to a method of converting verbal instructions into text data using voice recognition technology.

[1229] "Systems within the factory" refers to the various machines and programs used in the factory.

[1230] "Feedback" refers to opinions and evaluation information provided by users after using the system.

[1231] "Improvement procedures" refer to specific steps taken to review existing manufacturing processes and data analysis procedures and make them more effective.

[1232] The present invention provides a system for efficiently performing data analysis and manufacturing process improvements in factories. This system allows users to input instructions in the form of voice or text, and uses a generative AI model to provide specific operating procedures. An embodiment of this system is described in detail below.

[1233] System Overview

[1234] This system consists of a user, a terminal, a server, and a generative AI model. The user uses a software application containing the dataset to be operated on and inputs the final output in text format. The terminal sends the user's input data to the server, which uses the generative AI model to generate specific operating procedures. The generated operating procedures are converted into GUI format and sent to the terminal. The user performs operations by following the specific procedures displayed on the terminal.

[1235] Hardware and software used

[1236] Hardware: This system includes smart glasses, tablets, and servers. Specific examples include smart glasses such as Google Glass and Vuzix Blade, and cloud servers such as Amazon Web Services (AWS) and Microsoft Azure.

[1237] Software: The system includes OpenAI GPT-4 as a generative AI model, Python libraries (Pandas, Numpy) for data processing, Google Speech-to-Text API for voice recognition, and React Native and Vue.js for GUI display.

[1238] Specific examples of the system

[1239] 1. User input method:

[1240] Using smart glasses or a tablet, users can input data analysis and improvement instructions for the manufacturing process in voice or text format. For example, they could give instructions such as "Analyze the manufacturing defect rate by month and propose improvement measures."

[1241] 2. Send input data:

[1242] The device converts the voice input into text format and sends the input data in JSON format to the server.

[1243] 3. Use of generative AI models:

[1244] The server launches a generative AI model (e.g., OpenAI GPT-4) to generate specific operating procedures based on the user's request.

[1245] 4. Converting the operating procedure to GUI format:

[1246] The server converts the generated specific operating procedures into GUI format and sends them to the terminal.

[1247] 5. GUI procedure:

[1248] The terminal analyzes the received procedure data and displays specific operation procedures on the GUI, allowing the user to receive instructions in a visually easy-to-understand format.

[1249] 6. User Actions:

[1250] Users follow the procedures displayed on the terminal's GUI to operate the factory's manufacturing system and perform data analysis and process improvements.

[1251] Specific examples

[1252] Example of user input:

[1253] "Analyze the manufacturing defect rate by month and create a graph."

[1254] Example prompts to send to the generative AI model:

[1255] "Please provide the steps to use a generative AI model to analyze the defect rate of manufacturing data by month and generate a graph showing this."

[1256] This enables the system to efficiently and accurately perform data analysis and process improvements at the manufacturing site.

[1257] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1258] Step 1:

[1259] User Input

[1260] The user uses smart glasses or a tablet device to input the dataset to be manipulated and the final output in voice or text format. For example, the user might give instructions such as "Analyze the manufacturing defect rate by month and propose improvements." The input is then converted into text by voice recognition software (e.g., Google Speech-to-Text API) and imported into the device.

[1261] Input data: "Analyze manufacturing defect rates by month and propose improvement measures"

[1262] Output data: Text data converted from audio

[1263] Step 2:

[1264] Sending input data

[1265] The device parses the converted text data into JSON format and sends it to the server using an HTTP request, where the data is sent to the server in JSON format.

[1266] Input data: JSON format text data

[1267] Output: HTTP request sent to the server

[1268] Step 3:

[1269] Using generative AI models

[1270] The server analyzes the received JSON-formatted text data and launches a generative AI model (e.g., OpenAI GPT-4). The server constructs a prompt sentence and requests the generative AI model to generate specific steps for data analysis and manufacturing process improvement.

[1271] Input data: JSON format text data

[1272] Data processing: Generate prompts to input to the generative AI model

[1273] Output data: Specific operating procedures

[1274] Step 4:

[1275] Converting the steps to a GUI format

[1276] The server analyzes the specific operation procedures returned by the generative AI model and converts them into a GUI format that is easy for users to understand visually, using a front-end framework such as React Native or Vue.js.

[1277] Input data: Operation instructions received from the generative AI model

[1278] Data processing: Converting procedure data into GUI format

[1279] Output data: GUI-based specific operation procedures

[1280] Step 5:

[1281] Sending procedure data

[1282] The server parses the specific operation procedures converted into GUI format into JSON format and sends them to the terminal using an HTTP response.

[1283] Input data: GUI-based specific operation procedures

[1284] Output data: HTTP response sent to the device

[1285] Step 6:

[1286] Displaying procedures on the GUI

[1287] The device analyzes the received procedure data and displays it on the screen of the smart glasses or tablet device, allowing the user to visually confirm the specific operation procedures.

[1288] Input data: Operation procedure data received from the server (JSON format)

[1289] Data processing: Rendering data for display on a GUI

[1290] Output data: Operational steps that can be visually confirmed by the user

[1291] Step 7:

[1292] User operations

[1293] Users operate the factory's manufacturing system by following specific operating procedures displayed on the terminal, such as "select a data range," "count the number of defective products," and "generate graphs," to analyze data and improve the manufacturing process.

[1294] Input data: Operation procedure displayed on the GUI

[1295] Output data: Completed data analysis and manufacturing process improvement operations

[1296] This allows all steps to work together, enabling users to analyze data and improve manufacturing processes efficiently and accurately, even without specialized knowledge.

[1297] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1298] This invention relates to a system in which a generative AI model automatically suggests specific operating procedures when a user performs data analysis using a software application, simply by inputting the final output. This system also combines an emotion engine that recognizes the user's emotions. This system allows users to receive more personalized support and perform data analysis efficiently and accurately.

[1299] Overall flow of the system program

[1300] User Input

[1301] The user enters the desired output in text format in the input field in Excel or Google Spreadsheet. For example, they might enter "aggregate sales data and create a monthly sales graph."

[1302] Sending input data

[1303] The terminal receives the user's input data and sends it to the server in JSON format via an HTTP request.

[1304] Using generative AI models

[1305] The server analyzes the received user input data and activates a generative AI model, which generates optimal operating procedures based on the user's requests.

[1306] Converting the steps to a GUI format

[1307] The server converts the generated specific operation procedures into a GUI format, which provides the procedures in a format that is visually easy for the user to understand.

[1308] Sending procedure data

[1309] The server then sends the procedure data converted into GUI format to the terminal. The data is sent in JSON format via an HTTP response.

[1310] Displaying procedures on the GUI

[1311] The terminal analyzes the received procedure data and visually displays specific operation steps on the GUI, such as "Select a data range" or "Click the Insert tab and select a pivot table."

[1312] User operations

[1313] Users follow the instructions displayed on the device's GUI to perform operations in Excel or Google Spreadsheets, enabling them to easily perform highly accurate data analysis.

[1314] Integration with emotion engine

[1315] emotion recognition

[1316] The device uses an emotion engine to recognize the user's emotions based on the user's input data and behavior during operation (e.g., keyboard operations, mouse movements, facial recognition, etc.). For example, it identifies emotional states such as "happiness," "anxiety," and "excitement."

[1317] Sending emotional data

[1318] The device sends the analyzed emotional data to the server in JSON format via an HTTP request.

[1319] Emotion-based procedural adjustments

[1320] The server receives the user's emotional data and adjusts the generated operation procedure based on the user's emotional state. For example, if the user feels "anxious," the server may provide more detailed instructions.

[1321] Gathering feedback and improving the model

[1322] The device collects feedback from the user during or after the operation is completed and sends it to the server. The feedback information is sent in JSON format via an HTTP request, and the server uses this information to improve the generative AI model. Emotional data is also analyzed to improve the accuracy of the generative AI model.

[1323] Specific examples

[1324] Example: Summarizing and graphing sales data

[1325] 1. User Input

[1326] The user enters "aggregate sales data and create a monthly sales graph."

[1327] 2. Sending input data

[1328] The terminal transmits this input data to the server.

[1329] 3. Use of generative AI models

[1330] The server uses the generative AI model to generate the following specific steps:

[1331] Select the data range with dates in column A and sales amounts in column B.

[1332] Click the Insert tab and select PivotTable

[1333] Drag Date to Rows and Sales Amount to Values

[1334] Use the aggregated data to create a monthly bar chart

[1335] 4. Emotion recognition

[1336] As the user performs the operation, the device recognizes the user's emotions based on keyboard operations and mouse movements.

[1337] 5. Sending Emotional Data

[1338] The terminal transmits the user's emotion data to the server.

[1339] 6. Emotion-Based Procedural Adjustments

[1340] The server analyzes the user's emotional data and provides more detailed instructions as needed to help alleviate the anxiety the user feels.

[1341] 7. Displaying procedures on the GUI

[1342] The terminal displays the generated procedures on a GUI, allowing the user to operate them intuitively.

[1343] 8. User Operations

[1344] The user follows the displayed steps to complete the monthly sales graph.

[1345] 9. Gather feedback and improve the model

[1346] After the user completes the operation, the device collects feedback and sends it to the server, which uses it to improve the generative AI model and emotion engine.

[1347] In this way, by combining a generative AI model and an emotion engine, the present invention provides a data analysis support system that is intuitively operable for users and takes emotions into consideration.

[1348] The processing flow will be explained below.

[1349] Step 1:

[1350] The user enters the final output they want to achieve using Excel or Google Spreadsheets (e.g., "aggregate sales data and create a monthly sales graph") in text format into the tool's input field.

[1351] Step 2:

[1352] The device receives the user's input data and sends it to the server in JSON format using an HTTP request.

[1353] Step 3:

[1354] The server analyzes the received user input data and activates a generative AI model, which generates optimal operating procedures based on the user's requests.

[1355] Step 4:

[1356] The server converts the generated specific operation procedures into a GUI format, where the procedures output by the generative AI model are converted into a format that can be displayed visually.

[1357] Step 5:

[1358] The server sends the procedure data converted into GUI format to the terminal. The procedure data is sent in JSON format using an HTTP response.

[1359] Step 6:

[1360] The device analyzes the received instruction data and visually displays specific operation steps on the GUI, such as "Select a data range" or "Click the Insert tab and select a pivot table."

[1361] Step 7:

[1362] The user follows the instructions on the GUI to perform operations on Excel or Google Spreadsheets. The user follows the instructions displayed step by step to select a data range, create a pivot table, insert a chart, and so on.

[1363] Step 8:

[1364] The device uses an emotion engine to recognize the user's emotions based on the user's input data and behavior during operation (e.g., keyboard operations, mouse movements, facial recognition, etc.). For example, it can identify emotional states such as "happiness," "anxiety," and "excitement" based on the user's facial expressions and operation speed.

[1365] Step 9:

[1366] The device sends the analyzed emotion data to the server in JSON format via an HTTP request.

[1367] Step 10:

[1368] The server analyzes the received user emotion data and adjusts the generated operation instructions based on the user's emotional state. For example, if the user is feeling "anxious," the instructions are made more detailed to provide a sense of security.

[1369] Step 11:

[1370] The device will then redisplay the specific operational steps on the GUI, allowing the user to receive support adapted to their emotional state.

[1371] Step 12:

[1372] The user continues to follow the instructions on the GUI and completes the final output (e.g., a monthly sales graph).

[1373] Step 13:

[1374] The device collects feedback from the user during or after the operation is completed. The feedback information is collected in the form of a questionnaire about whether the procedure was useful or not, what needs to be improved, etc.

[1375] Step 14:

[1376] The device sends the collected feedback to the server in JSON format via an HTTP request.

[1377] Step 15:

[1378] The server analyzes the collected feedback and uses it to improve the generative AI model and emotion engine. Based on the feedback, the algorithms and rules of the generative AI model are improved, resulting in more accurate procedure generation and emotion recognition.

[1379] Example 2

[1380] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1381] Modern data analysis tasks often require specialized knowledge for users to perform tasks efficiently and accurately. Furthermore, manually understanding and executing a series of operational steps is laborious and time-consuming. Therefore, a system is needed in which a generative AI model automatically suggests specific operational steps based on the user's input. Furthermore, there is a need to provide more personalized support and improve the user experience by adjusting the steps based on the user's emotions. Current technology does not offer a system that meets all of these requirements, which is the problem this invention aims to solve.

[1382] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1383] In this invention, the server includes means for analyzing user input data and generating specific operating procedures using a generative AI model, means for converting the generated specific operating procedures into a GUI format, and means for adjusting the generated operating procedures based on emotion data transmitted from the terminal in accordance with the user's emotional state. This allows the user to receive visually easy-to-understand specific operating procedures simply by inputting the final output, and furthermore, by taking the user's emotions into consideration, stress and anxiety can be reduced, and data analysis work can proceed more smoothly.

[1384] "User" refers to an individual or organization that uses the system to perform data analysis work.

[1385] A "terminal" is a computer device operated by a user, and refers to a device for inputting, transmitting, receiving, displaying, etc. data.

[1386] "Server" refers to a computer system that uses a generative AI model to generate specific operating procedures and communicates with terminals.

[1387] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates optimal operating procedures based on user requests.

[1388] A "prompt" is a document or instruction that allows a user to input the final output in text form.

[1389] An "operational procedure" refers to a set of specific instructions for a user to correctly perform a data analysis task.

[1390] "GUI format" refers to a graphical user interface format designed to make it easy for users to operate visually.

[1391] "Emotion engine" refers to an artificial intelligence algorithm that analyzes a user's emotional state and improves the user's operating experience.

[1392] "Feedback" refers to information in which a user provides evaluations and opinions on operating procedures.

[1393] The present invention is a system for solving problems users face in data analysis work, and combines a generative AI model and an emotion engine to provide specific operating procedures to users and provide personalized support based on emotions. Detailed embodiments of this system are described below.

[1394] Overall system overview

[1395] This system provides specific operating procedures through the following steps when users perform data analysis using software applications (e.g., Excel or Google Spreadsheets): Multiple hardware and software components, including a server, terminals, generative AI models, and an emotion engine, work together.

[1396] The specific hardware and software used

[1397] 1. Hardware

[1398] Terminal: The device operated by the user (PC, tablet, smartphone, etc.)

[1399] Server: A remote server that hosts the generative AI model and emotion engine.

[1400] 2. Software

[1401] Generative AI model: An AI algorithm that generates specific operating procedures based on user requests.

[1402] Emotion Engine: An AI algorithm for analyzing a user's emotional state

[1403] Software applications: Data analysis tools such as Excel and Google Sheets

[1404] Processing flow

[1405] 1. User Input

[1406] The user enters the final output they want to achieve in the software application in text format. An example prompt might be "Summarize sales data and create a monthly sales graph."

[1407] 2. Sending input data

[1408] The terminal receives the user's input data and sends it to the server in JSON format via an HTTP request.

[1409] 3. Use of generative AI models

[1410] The server analyzes the received user input data and activates a generative AI model, which generates optimal operation procedures based on the user's prompt.

[1411] 4. Converting the procedure to GUI format

[1412] The server converts the generated specific operation procedures into a GUI format.

[1413] 5. Sending procedure data

[1414] The server transmits the procedure data converted into GUI format to the terminal.

[1415] 6. Displaying procedures on the GUI

[1416] The device analyzes the received instruction data and visually displays specific operation steps on the GUI. Specific examples include instructions such as "Select a data range" and "Click the Insert tab and select a pivot table."

[1417] 7. User Operations

[1418] The user follows the instructions displayed on the device's GUI and performs operations on Excel or Google Spreadsheets.

[1419] Integration with emotion engine

[1420] 1. Emotion recognition

[1421] The device uses an emotion engine to recognize the user's emotions based on the user's input data and behavior during operation (keyboard operations, mouse movements, facial recognition, etc.). Recognized emotions include "happiness," "anxiety," and "excitement."

[1422] 2. Sending Emotional Data

[1423] The device sends the analyzed emotion data to the server in JSON format via an HTTP request.

[1424] 3. Emotion-Based Procedural Adjustments

[1425] The server receives the user's emotional data and adjusts the generated operation procedure based on the user's emotional state, for example, if the user is feeling "anxious," the procedure will be explained in more detail.

[1426] 4. Gather feedback and improve the model

[1427] After the user completes the operation, the device collects feedback and sends it to the server, which uses this feedback to improve the accuracy of the generative AI model and emotion engine.

[1428] Specific examples

[1429] If a user types "aggregate sales data and create a monthly sales graph," the following steps will be generated:

[1430] Select the data range with dates in column A and sales amounts in column B.

[1431] Click the Insert tab and select PivotTable

[1432] Drag Date to Rows and Sales Amount to Values

[1433] Use the aggregated data to create a monthly bar chart

[1434] By following these steps, users can easily complete data analysis tasks, and the emotion engine can recognize the user's emotional state and provide appropriate support to further improve the user experience.

[1435] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1436] Step 1: User Input

[1437] The user enters the final output in text format into the software application to be operated. For example, they might enter "aggregate sales data and create a monthly sales graph." This input becomes the prompt statement, which forms the basis for the next step of data processing.

[1438] Input: Text input for final output

[1439] Output: Prompt in text format

[1440] Step 2: Submitting input data

[1441] The terminal receives the user's input data (prompt text) and sends it to the server in JSON format using an HTTP request. Specifically, the prompt text is sent to the server in the format {"task": "Aggregate sales data and create a monthly sales graph"}.

[1442] Input: prompt statement

[1443] Output: JSON format data

[1444] Step 3: Use the generative AI model

[1445] The server analyzes the received prompt and activates a generative AI model. A generative AI model is an algorithm that generates specific operational steps based on the user's request. In response to the prompt "aggregate sales data and create a monthly sales graph," the AI ​​generates the following operational steps:

[1446] Select a data range

[1447] Click the Insert tab

[1448] Select the pivot table

[1449] Drag a date to a row

[1450] Drag Sales Amount to Values

[1451] Create a bar chart

[1452] Input: JSON format prompt statement

[1453] Output: List of specific operating procedures

[1454] Step 4: Converting the Instructions to GUI Format

[1455] The server converts the generated specific operation steps into a GUI format. To make it visually easy for users to understand, each step is formatted as a button or interactive interface. For example, the message "Please select a data range" is displayed as an appropriate button.

[1456] Input: List of operating procedures

[1457] Output: GUI format operation procedure

[1458] Step 5: Sending Instruction Data

[1459] The server sends the GUI-formatted step data to the terminal in JSON format using an HTTP response. For example, the following data is sent: {"steps": ["Select a data range", "Click the Insert tab and select a pivot table"]}.

[1460] Input: GUI-style operation procedure

[1461] Output: Procedure data in JSON format

[1462] Step 6: Displaying the procedure on the GUI

[1463] The terminal analyzes the received procedure data and visually displays specific operation procedures on the GUI. To make it easier for the user to understand, the procedures are displayed on the screen as pop-up windows or information bars. This allows the user to follow the instructions to proceed with the operation.

[1464] Input: Procedure data in JSON format

[1465] Output: Visual operation procedure on GUI

[1466] Step 7: User interaction

[1467] The user follows the instructions displayed on the device's GUI to perform operations on Excel or Google Spreadsheets. Specifically, the user follows the instructions to perform actions such as "select a data range," "click the Insert tab," and "select a pivot table."

[1468] Input: GUI operation procedure

[1469] Output: User completes an action within the software

[1470] Step 8: Emotion Recognition

[1471] The device uses an emotion engine to recognize the user's emotions based on the user's input data and actions (keyboard operations, mouse movements, facial recognition, etc.). For example, it uses a camera and facial recognition software to identify emotional states such as "happiness," "anxiety," and "excitement."

[1472] Input: User behavior data

[1473] Output: User emotion data

[1474] Step 9: Sending Emotion Data

[1475] The device sends the analyzed emotion data to the server in JSON format via an HTTP request. For example, data such as "{"emotion": "anxiety"}" is sent.

[1476] Input: User emotion data

[1477] Output: Emotion data in JSON format

[1478] Step 10: Adjusting procedures based on emotions

[1479] The server receives the user's emotional data and adjusts the generated operation instructions based on the user's emotional state. For example, if the user is feeling "anxious," the server may add more detailed instructions to the operation instructions.

[1480] Input: Emotion data in JSON format

[1481] Output: Adjusted operating procedure

[1482] Step 11: Gather feedback and improve the model

[1483] After the user completes the operation, the device collects feedback and sends it to the server. The user feedback is used to improve the accuracy of the generative AI model and emotion engine. For example, the model can be adjusted based on feedback such as "The instructions were difficult to understand."

[1484] Input: User feedback

[1485] Output: Improved generative AI models and emotion engines

[1486] (Application example 2)

[1487] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1488] In conventional data analysis support systems, when users perform data analysis tasks on software applications, they must decide the specific operating procedures themselves, making the process complicated and difficult to perform efficiently. Furthermore, when users feel emotionally confused or anxious, the system often fails to provide appropriate support, slowing down the progress of the task. In particular, in everyday tasks such as inventory management and sales forecasting in brick-and-mortar stores, where fast and accurate data analysis is required, there is a lack of flexible operational support that responds to the user's emotions.

[1489] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1490] In this invention, the server includes means for generating specific operating procedures based on a user's request using a generative AI model, means for converting the generated specific operating procedures into a GUI format, and means for recognizing the user's emotions using an emotion engine and adjusting the generated operating procedures based on the emotions. This allows the user to receive emotion-sensitive support while quickly and accurately analyzing data, enabling efficient inventory management and sales forecasting in physical stores.

[1491] A "target information set" is a data set that a user can manipulate and analyze within a software application.

[1492] "Final output" refers to the result that a user wants to achieve using a software application.

[1493] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[1494] The "server" is a central processing system that processes data received from users and provides operational instructions and support using generative AI models and emotion engines.

[1495] A "generative AI model" refers to an artificial intelligence model that generates specific operating procedures based on user requests.

[1496] "GUI format" refers to a format for visually displaying operating procedures in the form of a graphical user interface.

[1497] An "emotion engine" is a system that recognizes a user's emotions based on input data and actions during user operation and analyzes that data.

[1498] "Emotion data" is data that indicates the user's emotional state analyzed by the emotion engine.

[1499] "Feedback" refers to the opinions, evaluations, and data regarding operations that users provide to the system.

[1500] "Data analysis" is the act of processing a set of information to extract patterns and useful information to produce a final output.

[1501] "Inventory management" refers to the task of understanding the inventory status of products, merchandise, and other materials in physical stores and maintaining an adequate supply.

[1502] "Sales forecasting" is the activity of predicting future sales based on past sales data and current sales trends.

[1503] This invention is a system in which, when a user performs data analysis using a software application that manipulates a set of information, the generative AI model automatically presents specific operating procedures simply by inputting the final output, and further combines this with an emotion engine that recognizes the user's emotions.

[1504] First, the user uses the terminal to input the final output in text format. For example, they might input "I want to forecast this month's inventory" or "I want to identify best-selling products." The terminal then sends this input data in JSON format to the server. Communication is via HTTP requests.

[1505] The server analyzes the received user input data and activates a generative AI model. The generative AI model generates optimal operating procedures based on the user's requests. For example, for inventory forecasting, it presents analytical procedures based on historical sales data and current inventory data. The generated specific operating procedures are converted into GUI format by the server. This conversion provides the procedures in a format that is visually easy for users to understand.

[1506] Furthermore, the device uses an emotion engine to recognize the user's emotions based on input data and actions (e.g., keyboard input and touch operations) during operation. The recognized emotion data is sent to the server in JSON format. The server analyzes the emotion data and adjusts the generated operation instructions to reduce the user's anxiety or confusion. For example, if the user feels anxious, it may provide more detailed explanations of the instructions.

[1507] The terminal receives specific operating instructions from the server and displays them on a GUI. The user operates the software application according to the displayed instructions. This allows the user to perform data analysis efficiently and accurately by following the guide.

[1508] After providing the operation instructions, the server collects feedback from the user and uses it to improve the accuracy of the generative AI model and emotion engine. The feedback information is also sent to the server in JSON format.

[1509] Specifically, a smartphone is used as the terminal, and a generative AI model such as OpenAI's GPT-3 is used on the server side. As an emotion engine, a machine learning framework such as Keras is used to analyze the user's facial expressions and input data.

[1510] (Example)

[1511] Here is an example prompt:

[1512] Example prompt sentence:

[1513] User request: I want to make inventory forecast for this month. Please provide specific steps.

[1514] Based on this prompt, a generative AI model generates instructions, and an emotion engine recognizes the user's emotions and adjusts the instructions accordingly, providing personalized assistance to the user.

[1515] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1516] Step 1:

[1517] A user inputs the final output in text format into a software application containing the set of information to be operated on. Specifically, using a smartphone as a terminal, the user inputs requests such as "I want to forecast this month's inventory" or "I want to identify best-selling products." The input data is saved in text format on the terminal.

[1518] Step 2:

[1519] The terminal sends the user's input data to the server. Communication is via HTTP request, and the data is sent in JSON format. The input data is text information that indicates the user's request, and is sent to the server.

[1520] Step 3:

[1521] The server analyzes the received user input data and activates a generative AI model. Specifically, it uses a generative AI model such as OpenAI's GPT-3 to generate optimal operating procedures based on the user's request. The generative AI model generates appropriate procedures for the request entered by the user and outputs the results in text format to the server.

[1522] Step 4:

[1523] The server converts the generated specific operation procedures into a GUI format. Specifically, it converts the generated textual procedures into a visually understandable format and builds them as GUI components. The converted procedures are saved as output on the server.

[1524] Step 5:

[1525] The server sends specific operation procedures in GUI format to the terminal. The data is sent in JSON format again using an HTTP response. The output data is the operation procedures converted into GUI format, and this is sent transparently to the terminal.

[1526] Step 6:

[1527] The specific operation procedures received by the terminal are displayed on the GUI, allowing the user to check the operation procedures visually rather than in text format. The received data is analyzed and rendered as GUI components.

[1528] Step 7:

[1529] A user operates a software application by following instructions displayed on a GUI. The user follows the instructions to perform specific operations, such as entering inventory data or performing analytical procedures.

[1530] Step 8:

[1531] The device uses an emotion engine to recognize the user's emotions based on the input data and actions of the user during operation. Specifically, the emotion recognition algorithm analyzes the keyboard input and touch operations performed by the user on the smartphone. This emotion data is saved as input data on the device.

[1532] Step 9:

[1533] The device sends the analyzed emotional data to the server. The emotional data is also sent in JSON format via an HTTP request. The input data indicates the user's emotional state and is sent to the server.

[1534] Step 10:

[1535] The server adjusts the generated operation procedures based on the emotional data. Specifically, to reduce the anxiety and confusion felt by the user, the server adjusts the procedures generated by the generative AI model and takes measures such as providing more detailed explanations. These adjusted procedures are saved as output data on the server.

[1536] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1537] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1538] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1539] [Fourth embodiment]

[1540] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1541] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1542] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1543] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1544] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1545] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1546] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1547] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1548] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1549] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1550] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1551] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1552] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1553] The present invention is a system in which a generative AI model automatically presents specific operational procedures when a user simply inputs the desired final output in data analysis work using software applications such as Excel or Google Spreadsheets. The following describes in detail an embodiment of this system.

[1554] Overall flow of the system program

[1555] User Input

[1556] The user enters the desired output in text format into an input field in Excel or Google Spreadsheet. For example, they can enter a specific request such as "aggregate sales data and create a monthly sales graph."

[1557] Sending input data

[1558] The terminal sends the user's input data to the server using an HTTP request. The input data is sent in JSON format.

[1559] Using generative AI models

[1560] The server analyzes the received user input data and activates a generative AI model. This generative AI model generates optimal operation procedures based on the user's request. For example, it generates specific instructions such as how to select a data range, how to create a pivot table, and how to insert a graph.

[1561] Converting the steps to a GUI format

[1562] The server converts the generated specific operation procedures into a GUI format, which provides the procedures in a format that is visually easy for the user to understand.

[1563] Sending procedure data

[1564] The server then sends the procedure data converted into GUI format to the terminal, again using an HTTP response, with the procedure data being sent in JSON format.

[1565] Displaying procedures on the GUI

[1566] The terminal analyzes the received instruction data and displays specific operation steps on the GUI, such as "Select a range" or "Click the Insert tab and select a pivot table."

[1567] User operations

[1568] Users follow the instructions displayed on the device's GUI to perform operations in Excel or Google Spreadsheets, allowing them to easily perform highly accurate data analysis.

[1569] Specific examples

[1570] Example: Summarizing and graphing sales data

[1571] 1. User Input

[1572] The user enters "aggregate sales data and create a monthly sales graph."

[1573] 2. Sending input data

[1574] The terminal transmits this input data to the server.

[1575] 3. Use of generative AI models

[1576] The server uses the generative AI model to generate the following specific steps:

[1577] Select the data range with dates in column A and sales amounts in column B.

[1578] Click the Insert tab and select PivotTable

[1579] Drag Date to Rows and Sales Amount to Values

[1580] Use the aggregated data to create a monthly bar chart

[1581] 4. Converting the procedure to GUI format

[1582] The server translates these steps into a GUI format.

[1583] 5. Sending procedure data

[1584] The server transmits the procedure data converted into GUI format to the terminal.

[1585] 6. Displaying procedures on the GUI

[1586] The terminal analyzes the received procedure data and displays the procedure on the GUI.

[1587] "Please select a data range with dates in column A and sales amounts in column B."

[1588] Click the Insert tab and select PivotTable.

[1589] "Drag the dates to the rows and the sales amounts to the values."

[1590] "Create a bar graph for each month."

[1591] 7. User Operations

[1592] The user follows the displayed steps to perform operations in Excel and complete a monthly sales graph.

[1593] In this way, the present invention is a system that improves the efficiency and accuracy of data analysis by automatically generating the procedures required for a user to perform data analysis and allowing intuitive operation.

[1594] The processing flow will be explained below.

[1595] Step 1:

[1596] The user enters the final output they want to achieve using Excel or Google Spreadsheets (e.g., "aggregate sales data and create a monthly sales graph") in text format into the tool's input field.

[1597] Step 2:

[1598] The device receives the user's input data and sends it to the server in JSON format via an HTTP request.

[1599] Step 3:

[1600] The server analyzes the received user input data and activates a generative AI model, which generates optimal operating procedures based on the user's requests.

[1601] Step 4:

[1602] The server converts the generated specific operation procedures into a GUI format, where the procedures output by the generative AI model are converted into a visually displayable format.

[1603] Step 5:

[1604] The server sends the procedure data converted into GUI format to the terminal. The data is sent in JSON format via an HTTP response.

[1605] Step 6:

[1606] The device analyzes the received procedure data and visually displays specific operation steps on the GUI, such as "Select a data range" or "Click the Insert tab and select a pivot table."

[1607] Step 7:

[1608] The user follows the instructions on the GUI to perform operations on Excel or Google Spreadsheets. The user follows the instructions displayed step by step to select a data range, create a pivot table, insert a chart, and so on.

[1609] Step 8:

[1610] The device collects feedback from the user during or after the operation is completed. The feedback is collected in the form of a questionnaire about whether the procedure was useful or not, what improvements can be made, etc.

[1611] Step 9:

[1612] The device sends the collected feedback to the server, which sends the feedback information in JSON format via an HTTP request.

[1613] Step 10:

[1614] The server analyzes the collected feedback and uses it to improve the generative AI model. Based on the feedback, the algorithms and rules of the generative AI model are improved, resulting in more accurate procedure generation.

[1615] Example 1

[1616] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1617] In traditional data analysis work, users often have to understand complex operating procedures in applications such as Excel or Google Spreadsheets and perform the operations manually. This tends to result in inefficient data analysis and low accuracy. Furthermore, users spend a lot of time understanding the necessary operating procedures, which reduces productivity. Furthermore, errors in operating procedures are likely to occur, resulting in a risk of lower quality.

[1618] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1619] In this invention, the server includes: a means for a user to input a final output in text format in an application including an information set to be operated; a means for the terminal to transmit the user's input data to the processing device; a means for the processing device to generate specific operating procedures based on the user's request using a generative artificial intelligence model; a means for the processing device to convert the generated specific operating procedures into a visual display format; a means for the processing device to transmit the specific operating procedures in the visual display format to the terminal; a means for the terminal to display the received specific operating procedures on a visual user interface; and a means for the user to operate the application according to the operating procedures displayed on the visual user interface. This allows the user to easily understand and accurately execute the specific operating procedures. This significantly improves the efficiency and accuracy of data analysis and increases user productivity.

[1620] "User" refers to a user who utilizes the system to work with an application that includes an information set to be manipulated.

[1621] "Information set" refers to the entire collection of data that is the subject of analysis and processing.

[1622] "Application" refers to software that users use to analyze and manipulate data, such as Excel or Google Sheets.

[1623] "Output" refers to the final result or outcome that the user wants to achieve.

[1624] "Terminal" refers to a device used by a user, such as a computer, smartphone, or tablet.

[1625] "Processing device" refers to a server or cloud computing environment that analyzes input data sent by a user, activates a generative AI model, and generates specific operating procedures.

[1626] "Generative artificial intelligence models" refer to AI technologies such as machine learning algorithms and large-scale language models that generate optimal operating procedures based on user requests.

[1627] The "visual display format" refers to a format in which the generated specific operating procedures are visualized so that the user can easily understand them.

[1628] "Visual user interface" refers to a graphical interface that allows a user to view and execute operating procedures.

[1629] "Feedback" refers to opinions and information provided by users regarding operating procedures.

[1630] The present invention is a system in which a generative AI model automatically presents specific operational procedures when a user simply inputs the desired final output in data analysis work using applications such as Excel or Google Spreadsheets. The following describes in detail an embodiment of this system.

[1631] User Input

[1632] The user inputs in text form the final output they want to achieve from the application containing the information set to be manipulated, for example, a request such as "aggregate sales data and create a monthly sales graph."

[1633] Sending input data

[1634] The terminal sends the user's input data to the processing device. This transmission uses an HTTP request, and the data is converted into JSON format. Specifically, the terminal converts the user's input "aggregate sales data and create a monthly sales graph" into JSON data ({"request": "aggregate sales data and create a monthly sales graph"}) and sends it.

[1635] Using generative AI models

[1636] The server analyzes the received input data and activates a generative AI model, which generates operational procedures based on the user's request. For example, it generates the following procedures:

[1637] Select the data range with dates in column A and sales amounts in column B.

[1638] Click the Insert tab and select PivotTable

[1639] Drag Date to Rows and Sales Amount to Values

[1640] Use the aggregated data to create a monthly bar chart

[1641] Converting the steps to a GUI format

[1642] The server converts the generated specific operation instructions into a visual display format, providing the instructions in a form that is easy for the user to understand visually. For example, it generates GUI code including instructions such as "Please select a data range with dates in column A and sales amounts in column B."

[1643] Sending procedure data

[1644] The server then sends the procedure data converted into a visual display format to the terminal, again using an HTTP response, with the data being sent in JSON format.

[1645] Displaying procedures on the GUI

[1646] The terminal analyzes the received instruction data and displays specific operation instructions on the GUI. For example, the terminal visually displays instructions such as "Select a data range with dates in column A and sales amounts in column B," "Click the Insert tab," "Select a pivot table," "Drag dates to rows," "Drag sales amounts to values," and "Create a bar graph by month."

[1647] User operations

[1648] Users follow the instructions displayed on the device's GUI to perform operations in Excel or Google Spreadsheets, allowing them to easily perform highly accurate data analysis.

[1649] Specific examples

[1650] Example: Summarizing and graphing sales data

[1651] 1. User Input

[1652] The user enters "aggregate sales data and create a monthly sales graph."

[1653] 2. Sending input data

[1654] The terminal transmits this input data to the processing unit.

[1655] 3. Use of generative AI models

[1656] The server uses the generative AI model to generate the following specific steps:

[1657] Select the data range with dates in column A and sales amounts in column B.

[1658] Click the Insert tab and select PivotTable

[1659] Drag Date to Rows and Sales Amount to Values

[1660] Use the aggregated data to create a monthly bar chart

[1661] 4. Converting the procedure to GUI format

[1662] The server translates these instructions into a visual display format.

[1663] 5. Sending procedure data

[1664] The server transmits the procedure data converted into a visual display format to the terminal.

[1665] 6. Displaying procedures on the GUI

[1666] The terminal analyzes the received procedure data and displays the procedure on the GUI.

[1667] "Please select a data range with dates in column A and sales amounts in column B."

[1668] Click the Insert tab and select PivotTable.

[1669] "Drag the dates to the rows and the sales amounts to the values."

[1670] "Create a bar graph for each month."

[1671] 7. User Operations

[1672] The user follows the displayed steps to perform operations in Excel and complete a monthly sales graph.

[1673] In this way, the present invention is a system that improves the efficiency and accuracy of data analysis by automatically generating the procedures required for users to perform data analysis and allowing them to operate intuitively.

[1674] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1675] Step 1:

[1676] (user input)

[1677] How it works: The user enters a textual description of the final output they want to achieve from the application, including the set of information they want to operate on. For example, they might enter "aggregate sales data and create a graph of sales by month."

[1678] Input: The user enters the request in text format into an Excel or Google Spreadsheet.

[1679] Output: The user's request is saved as text in the input field.

[1680] Step 2:

[1681] (Send input data)

[1682] How it works: The device converts the user's input data into JSON format and sends it to the server using an HTTP request.

[1683] Input: The request entered by the user in text format.

[1684] Data processing: Convert text data into JSON format.

[1685] Output: The data converted to JSON format is sent to the server via an HTTP request.

[1686] Step 3:

[1687] (Using generative AI models)

[1688] How it works: The server parses the received JSON data and inputs it as a prompt to the generative AI model, which then generates the optimal operation procedure based on the user's request.

[1689] Input: User request data in JSON format (e.g., {"request": "Aggregate sales data and create a monthly sales chart"}).

[1690] Data calculation: Analyzes JSON data and inputs it as prompts to the generative AI model to generate specific operating procedures.

[1691] Output: Generated instructions (e.g. "Select the data range with dates in column A and sales amounts in column B").

[1692] Step 4:

[1693] (Converting the procedure into GUI format)

[1694] How it works: The server analyzes the generated operating instructions and converts them into a format that is easy for the user to understand visually.

[1695] Input: Specific operational instructions generated by a generative AI model.

[1696] Data processing: The generated operating procedures are converted into a visual display format (GUI) using HTML, CSS, and JavaScript.

[1697] Output: Procedure data in GUI format.

[1698] Step 5:

[1699] (Send procedure data)

[1700] Operation: The server sends the procedure data converted into GUI format to the terminal using an HTTP response.

[1701] Input: Specific operation procedure data in GUI format.

[1702] Data processing: Convert GUI format data into JSON format.

[1703] Output: The data converted to JSON format is sent to the terminal as an HTTP response.

[1704] Step 6:

[1705] (Procedure displayed on GUI)

[1706] Operation: The terminal analyzes the received procedure data and displays specific operating procedures on the GUI.

[1707] Input: GUI steps data in JSON format (e.g., {"steps": ["Select a data range with dates in column A and sales amounts in column B", "Click the Insert tab", "Select PivotTable", "Drag Dates to Rows", "Drag Sales Amounts to Values", "Create a bar chart by month"]}).

[1708] Data operations: Parse the JSON data and process it to display it on a visual user interface.

[1709] Output: Specific operation steps visualized on the GUI.

[1710] Step 7:

[1711] (User operation)

[1712] How it works: Users follow the on-screen instructions to perform operations in Excel or Google Sheets. Specifically, they are instructed to "select a data range with dates in column A and sales amounts in column B," "click the Insert tab," "select PivotTable," "drag dates to rows," "drag sales amounts to values," and "create a bar graph by month."

[1713] Input: Operation steps displayed on the GUI.

[1714] Data processing: Users manually process data in Excel or Google Spreadsheets according to instructions.

[1715] Output: The final output desired by the user (e.g., a graph of sales by month).

[1716] (Application example 1)

[1717] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1718] Traditional data analysis and manufacturing process optimization required a great deal of specialized knowledge and time, resulting in reduced work efficiency on the factory floor. Furthermore, if workers were not familiar with data analysis and process improvement, it was difficult to obtain accurate analysis results and effective improvement measures. Furthermore, there was also the issue of workers finding it difficult to understand and apply procedures in real time.

[1719] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1720] In this invention, the server includes: a means for a user to input a final output in text format in a software application including a dataset to be operated; a means for a terminal to send the user's input data to the server; a means for the server to generate specific operating procedures based on the user's request using a generative AI model; a means for the server to convert the generated specific operating procedures into a GUI format; a means for the server to send the specific operating procedures in GUI format to the terminal; a means for the terminal to display the specific operating procedures received on the GUI; a means for the user to operate the software application in accordance with the operating procedures displayed on the GUI; a means for the user to input voice instructions for data analysis and manufacturing process improvement in a factory and convert them into text format; a means for the server to generate manufacturing data analysis procedures and manufacturing process improvement procedures using the generative AI model and display them in GUI format; and a means for the user to operate systems in the factory according to the instructions displayed on the terminal's GUI. This enables workers without specialized knowledge to efficiently and accurately analyze data and improve manufacturing processes.

[1721] "User" refers to an individual or group of individuals who operate software applications and factory systems.

[1722] "Dataset to be manipulated" refers to a collection of data that a user processes and analyzes within a software application.

[1723] "Software Applications" refers to programs for data processing and analysis, such as Excel and Google Spreadsheets.

[1724] "Final output" refers to the result or goal the user wants to achieve.

[1725] "Means of inputting in text format" refers to methods of inputting characters using a keyboard, voice input, etc.

[1726] "Device" refers to devices such as smartphones, tablets, and smart glasses.

[1727] "Server" refers to a remote computing device that processes generative AI models and manages data.

[1728] A "generative AI model" refers to an artificial intelligence program that automatically generates specific operating procedures based on user requests.

[1729] "Specific operating procedures" refer to the specific steps that the user must actually take, such as selecting a data range or creating a graph.

[1730] "GUI format" refers to a graphical user interface format, a display format that is visually easy to understand.

[1731] A "manufacturing process" refers to a series of tasks and procedures for producing a product in a factory.

[1732] "Manufacturing process improvement" refers to measures or attempts to improve the efficiency or quality of a manufacturing process.

[1733] "Means for inputting voice and converting it into text format" refers to a method of converting verbal instructions into text data using voice recognition technology.

[1734] "Systems within the factory" refers to the various machines and programs used in the factory.

[1735] "Feedback" refers to opinions and evaluation information provided by users after using the system.

[1736] "Improvement procedures" refer to specific steps taken to review existing manufacturing processes and data analysis procedures and make them more effective.

[1737] The present invention provides a system for efficiently performing data analysis and manufacturing process improvements in factories. This system allows users to input instructions in the form of voice or text, and uses a generative AI model to provide specific operating procedures. An embodiment of this system is described in detail below.

[1738] System Overview

[1739] This system consists of a user, a terminal, a server, and a generative AI model. The user uses a software application containing the dataset to be operated on and inputs the final output in text format. The terminal sends the user's input data to the server, which uses the generative AI model to generate specific operating procedures. The generated operating procedures are converted into GUI format and sent to the terminal. The user performs operations by following the specific procedures displayed on the terminal.

[1740] Hardware and software used

[1741] Hardware: This system includes smart glasses, tablets, and servers. Specific examples include smart glasses such as Google Glass and Vuzix Blade, and cloud servers such as Amazon Web Services (AWS) and Microsoft Azure.

[1742] Software: The system includes OpenAI GPT-4 as a generative AI model, Python libraries (Pandas, Numpy) for data processing, Google Speech-to-Text API for voice recognition, and React Native and Vue.js for GUI display.

[1743] Specific examples of the system

[1744] 1. User input method:

[1745] Using smart glasses or a tablet, users can input data analysis and improvement instructions for the manufacturing process in voice or text format. For example, they could give instructions such as "Analyze the manufacturing defect rate by month and propose improvement measures."

[1746] 2. Send input data:

[1747] The device converts the voice input into text format and sends the input data in JSON format to the server.

[1748] 3. Use of generative AI models:

[1749] The server launches a generative AI model (e.g., OpenAI GPT-4) to generate specific operating procedures based on the user's request.

[1750] 4. Converting the operating procedure to GUI format:

[1751] The server converts the generated specific operating procedures into GUI format and sends them to the terminal.

[1752] 5. GUI procedure:

[1753] The terminal analyzes the received procedure data and displays specific operation procedures on the GUI, allowing the user to receive instructions in a visually easy-to-understand format.

[1754] 6. User Actions:

[1755] Users follow the procedures displayed on the terminal's GUI to operate the factory's manufacturing system and perform data analysis and process improvements.

[1756] Specific examples

[1757] Example of user input:

[1758] "Analyze the manufacturing defect rate by month and create a graph."

[1759] Example prompts to send to the generative AI model:

[1760] "Please provide the steps to use a generative AI model to analyze the defect rate of manufacturing data by month and generate a graph showing this."

[1761] This enables the system to efficiently and accurately perform data analysis and process improvements at the manufacturing site.

[1762] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1763] Step 1:

[1764] User Input

[1765] The user uses smart glasses or a tablet device to input the dataset to be manipulated and the final output in voice or text format. For example, the user might give instructions such as "Analyze the manufacturing defect rate by month and propose improvements." The input is then converted into text by voice recognition software (e.g., Google Speech-to-Text API) and imported into the device.

[1766] Input data: "Analyze manufacturing defect rates by month and propose improvement measures"

[1767] Output data: Text data converted from audio

[1768] Step 2:

[1769] Sending input data

[1770] The device parses the converted text data into JSON format and sends it to the server using an HTTP request, where the data is sent to the server in JSON format.

[1771] Input data: JSON format text data

[1772] Output: HTTP request sent to the server

[1773] Step 3:

[1774] Using generative AI models

[1775] The server analyzes the received JSON-formatted text data and launches a generative AI model (e.g., OpenAI GPT-4). The server constructs a prompt sentence and requests the generative AI model to generate specific steps for data analysis and manufacturing process improvement.

[1776] Input data: JSON format text data

[1777] Data processing: Generate prompts to input to the generative AI model

[1778] Output data: Specific operating procedures

[1779] Step 4:

[1780] Converting the steps to a GUI format

[1781] The server analyzes the specific operation procedures returned by the generative AI model and converts them into a GUI format that is easy for users to understand visually, using a front-end framework such as React Native or Vue.js.

[1782] Input data: Operation instructions received from the generative AI model

[1783] Data processing: Converting procedure data into GUI format

[1784] Output data: GUI-based specific operation procedures

[1785] Step 5:

[1786] Sending procedure data

[1787] The server parses the specific operation procedures converted into GUI format into JSON format and sends them to the terminal using an HTTP response.

[1788] Input data: GUI-based specific operation procedures

[1789] Output data: HTTP response sent to the device

[1790] Step 6:

[1791] Displaying procedures on the GUI

[1792] The device analyzes the received procedure data and displays it on the screen of the smart glasses or tablet device, allowing the user to visually confirm the specific operation procedures.

[1793] Input data: Operation procedure data received from the server (JSON format)

[1794] Data processing: Rendering data for display on a GUI

[1795] Output data: Operational steps that can be visually confirmed by the user

[1796] Step 7:

[1797] User operations

[1798] Users operate the factory's manufacturing system by following specific operating procedures displayed on the terminal, such as "select a data range," "count the number of defective products," and "generate graphs," to analyze data and improve the manufacturing process.

[1799] Input data: Operation procedure displayed on the GUI

[1800] Output data: Completed data analysis and manufacturing process improvement operations

[1801] This allows all steps to work together, enabling users to analyze data and improve manufacturing processes efficiently and accurately, even without specialized knowledge.

[1802] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1803] This invention relates to a system in which a generative AI model automatically suggests specific operating procedures when a user performs data analysis using a software application, simply by inputting the final output. This system also combines an emotion engine that recognizes the user's emotions. This system allows users to receive more personalized support and perform data analysis efficiently and accurately.

[1804] Overall flow of the system program

[1805] User Input

[1806] The user enters the desired output in text format in the input field in Excel or Google Spreadsheet. For example, they might enter "aggregate sales data and create a monthly sales graph."

[1807] Sending input data

[1808] The terminal receives the user's input data and sends it to the server in JSON format via an HTTP request.

[1809] Using generative AI models

[1810] The server analyzes the received user input data and activates a generative AI model, which generates optimal operating procedures based on the user's requests.

[1811] Converting the steps to a GUI format

[1812] The server converts the generated specific operation procedures into a GUI format, which provides the procedures in a format that is visually easy for the user to understand.

[1813] Sending procedure data

[1814] The server then sends the procedure data converted into GUI format to the terminal. The data is sent in JSON format via an HTTP response.

[1815] Displaying procedures on the GUI

[1816] The terminal analyzes the received procedure data and visually displays specific operation steps on the GUI, such as "Select a data range" or "Click the Insert tab and select a pivot table."

[1817] User operations

[1818] Users follow the instructions displayed on the device's GUI to perform operations in Excel or Google Spreadsheets, enabling them to easily perform highly accurate data analysis.

[1819] Integration with emotion engine

[1820] emotion recognition

[1821] The device uses an emotion engine to recognize the user's emotions based on the user's input data and behavior during operation (e.g., keyboard operations, mouse movements, facial recognition, etc.). For example, it identifies emotional states such as "happiness," "anxiety," and "excitement."

[1822] Sending emotional data

[1823] The device sends the analyzed emotional data to the server in JSON format via an HTTP request.

[1824] Emotion-based procedural adjustments

[1825] The server receives the user's emotional data and adjusts the generated operation procedure based on the user's emotional state. For example, if the user feels "anxious," the server may provide more detailed instructions.

[1826] Gathering feedback and improving the model

[1827] The device collects feedback from the user during or after the operation is completed and sends it to the server. The feedback information is sent in JSON format via an HTTP request, and the server uses this information to improve the generative AI model. Emotional data is also analyzed to improve the accuracy of the generative AI model.

[1828] Specific examples

[1829] Example: Summarizing and graphing sales data

[1830] 1. User Input

[1831] The user enters "aggregate sales data and create a monthly sales graph."

[1832] 2. Sending input data

[1833] The terminal transmits this input data to the server.

[1834] 3. Use of generative AI models

[1835] The server uses the generative AI model to generate the following specific steps:

[1836] Select the data range with dates in column A and sales amounts in column B.

[1837] Click the Insert tab and select PivotTable

[1838] Drag Date to Rows and Sales Amount to Values

[1839] Use the aggregated data to create a monthly bar chart

[1840] 4. Emotion recognition

[1841] As the user performs the operation, the device recognizes the user's emotions based on keyboard operations and mouse movements.

[1842] 5. Sending Emotional Data

[1843] The terminal transmits the user's emotion data to the server.

[1844] 6. Emotion-Based Procedural Adjustments

[1845] The server analyzes the user's emotional data and provides more detailed instructions as needed to help alleviate the anxiety the user feels.

[1846] 7. Displaying procedures on the GUI

[1847] The terminal displays the generated procedures on a GUI, allowing the user to operate them intuitively.

[1848] 8. User Operations

[1849] The user follows the displayed steps to complete the monthly sales graph.

[1850] 9. Gather feedback and improve the model

[1851] After the user completes the operation, the device collects feedback and sends it to the server, which uses it to improve the generative AI model and emotion engine.

[1852] In this way, by combining a generative AI model and an emotion engine, the present invention provides a data analysis support system that is intuitively operable for users and takes emotions into consideration.

[1853] The processing flow will be explained below.

[1854] Step 1:

[1855] The user enters the final output they want to achieve using Excel or Google Spreadsheets (e.g., "aggregate sales data and create a monthly sales graph") in text format into the tool's input field.

[1856] Step 2:

[1857] The device receives the user's input data and sends it to the server in JSON format using an HTTP request.

[1858] Step 3:

[1859] The server analyzes the received user input data and activates a generative AI model, which generates optimal operating procedures based on the user's requests.

[1860] Step 4:

[1861] The server converts the generated specific operation procedures into a GUI format, where the procedures output by the generative AI model are converted into a format that can be displayed visually.

[1862] Step 5:

[1863] The server sends the procedure data converted into GUI format to the terminal. The procedure data is sent in JSON format using an HTTP response.

[1864] Step 6:

[1865] The device analyzes the received instruction data and visually displays specific operation steps on the GUI, such as "Select a data range" or "Click the Insert tab and select a pivot table."

[1866] Step 7:

[1867] The user follows the instructions on the GUI to perform operations on Excel or Google Spreadsheets. The user follows the instructions displayed step by step to select a data range, create a pivot table, insert a chart, and so on.

[1868] Step 8:

[1869] The device uses an emotion engine to recognize the user's emotions based on the user's input data and behavior during operation (e.g., keyboard operations, mouse movements, facial recognition, etc.). For example, it can identify emotional states such as "happiness," "anxiety," and "excitement" based on the user's facial expressions and operation speed.

[1870] Step 9:

[1871] The device sends the analyzed emotion data to the server in JSON format via an HTTP request.

[1872] Step 10:

[1873] The server analyzes the received user emotion data and adjusts the generated operation instructions based on the user's emotional state. For example, if the user is feeling "anxious," the instructions are made more detailed to provide a sense of security.

[1874] Step 11:

[1875] The device will then redisplay the specific operational steps on the GUI, allowing the user to receive support adapted to their emotional state.

[1876] Step 12:

[1877] The user continues to follow the instructions on the GUI and completes the final output (e.g., a monthly sales graph).

[1878] Step 13:

[1879] The device collects feedback from the user during or after the operation is completed. The feedback information is collected in the form of a questionnaire about whether the procedure was useful or not, what needs to be improved, etc.

[1880] Step 14:

[1881] The device sends the collected feedback to the server in JSON format via an HTTP request.

[1882] Step 15:

[1883] The server analyzes the collected feedback and uses it to improve the generative AI model and emotion engine. Based on the feedback, the algorithms and rules of the generative AI model are improved, resulting in more accurate procedure generation and emotion recognition.

[1884] Example 2

[1885] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1886] Modern data analysis tasks often require specialized knowledge for users to perform tasks efficiently and accurately. Furthermore, manually understanding and executing a series of operational steps is laborious and time-consuming. Therefore, a system is needed in which a generative AI model automatically suggests specific operational steps based on the user's input. Furthermore, there is a need to provide more personalized support and improve the user experience by adjusting the steps based on the user's emotions. Current technology does not offer a system that meets all of these requirements, which is the problem this invention aims to solve.

[1887] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1888] In this invention, the server includes means for analyzing user input data and generating specific operating procedures using a generative AI model, means for converting the generated specific operating procedures into a GUI format, and means for adjusting the generated operating procedures based on emotion data transmitted from the terminal in accordance with the user's emotional state. This allows the user to receive visually easy-to-understand specific operating procedures simply by inputting the final output, and furthermore, by taking the user's emotions into consideration, stress and anxiety can be reduced, and data analysis work can proceed more smoothly.

[1889] "User" refers to an individual or organization that uses the system to perform data analysis work.

[1890] A "terminal" is a computer device operated by a user, and refers to a device for inputting, transmitting, receiving, displaying, etc. data.

[1891] "Server" refers to a computer system that uses a generative AI model to generate specific operating procedures and communicates with terminals.

[1892] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates optimal operating procedures based on user requests.

[1893] A "prompt" is a document or instruction that allows a user to input the final output in text form.

[1894] An "operational procedure" refers to a set of specific instructions for a user to correctly perform a data analysis task.

[1895] "GUI format" refers to a graphical user interface format designed to make it easy for users to operate visually.

[1896] "Emotion engine" refers to an artificial intelligence algorithm that analyzes a user's emotional state and improves the user's operating experience.

[1897] "Feedback" refers to information in which a user provides evaluations and opinions on operating procedures.

[1898] The present invention is a system for solving problems users face in data analysis work, and combines a generative AI model and an emotion engine to provide specific operating procedures to users and provide personalized support based on emotions. Detailed embodiments of this system are described below.

[1899] Overall system overview

[1900] This system provides specific operating procedures through the following steps when users perform data analysis using software applications (e.g., Excel or Google Spreadsheets): Multiple hardware and software components, including a server, terminals, generative AI models, and an emotion engine, work together.

[1901] The specific hardware and software used

[1902] 1. Hardware

[1903] Terminal: The device operated by the user (PC, tablet, smartphone, etc.)

[1904] Server: A remote server that hosts the generative AI model and emotion engine.

[1905] 2. Software

[1906] Generative AI model: An AI algorithm that generates specific operating procedures based on user requests.

[1907] Emotion Engine: An AI algorithm for analyzing a user's emotional state

[1908] Software applications: Data analysis tools such as Excel and Google Sheets

[1909] Processing flow

[1910] 1. User Input

[1911] The user enters the final output they want to achieve in the software application in text format. An example prompt might be "Summarize sales data and create a monthly sales graph."

[1912] 2. Sending input data

[1913] The terminal receives the user's input data and sends it to the server in JSON format via an HTTP request.

[1914] 3. Use of generative AI models

[1915] The server analyzes the received user input data and activates a generative AI model, which generates optimal operation procedures based on the user's prompt.

[1916] 4. Converting the procedure to GUI format

[1917] The server converts the generated specific operation procedures into a GUI format.

[1918] 5. Sending procedure data

[1919] The server transmits the procedure data converted into GUI format to the terminal.

[1920] 6. Displaying procedures on the GUI

[1921] The device analyzes the received instruction data and visually displays specific operation steps on the GUI. Specific examples include instructions such as "Select a data range" and "Click the Insert tab and select a pivot table."

[1922] 7. User Operations

[1923] The user follows the instructions displayed on the device's GUI and performs operations on Excel or Google Spreadsheets.

[1924] Integration with emotion engine

[1925] 1. Emotion recognition

[1926] The device uses an emotion engine to recognize the user's emotions based on the user's input data and behavior during operation (keyboard operations, mouse movements, facial recognition, etc.). Recognized emotions include "happiness," "anxiety," and "excitement."

[1927] 2. Sending Emotional Data

[1928] The device sends the analyzed emotion data to the server in JSON format via an HTTP request.

[1929] 3. Emotion-Based Procedural Adjustments

[1930] The server receives the user's emotional data and adjusts the generated operation procedure based on the user's emotional state, for example, if the user is feeling "anxious," the procedure will be explained in more detail.

[1931] 4. Gather feedback and improve the model

[1932] After the user completes the operation, the device collects feedback and sends it to the server, which uses this feedback to improve the accuracy of the generative AI model and emotion engine.

[1933] Specific examples

[1934] If a user types "aggregate sales data and create a monthly sales graph," the following steps will be generated:

[1935] Select the data range with dates in column A and sales amounts in column B.

[1936] Click the Insert tab and select PivotTable

[1937] Drag Date to Rows and Sales Amount to Values

[1938] Use the aggregated data to create a monthly bar chart

[1939] By following these steps, users can easily complete data analysis tasks, and the emotion engine can recognize the user's emotional state and provide appropriate support to further improve the user experience.

[1940] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1941] Step 1: User Input

[1942] The user enters the final output in text format into the software application to be operated. For example, they might enter "aggregate sales data and create a monthly sales graph." This input becomes the prompt statement, which forms the basis for the next step of data processing.

[1943] Input: Text input for final output

[1944] Output: Prompt in text format

[1945] Step 2: Submitting input data

[1946] The terminal receives the user's input data (prompt text) and sends it to the server in JSON format using an HTTP request. Specifically, the prompt text is sent to the server in the format {"task": "Aggregate sales data and create a monthly sales graph"}.

[1947] Input: prompt statement

[1948] Output: JSON format data

[1949] Step 3: Use the generative AI model

[1950] The server analyzes the received prompt and activates a generative AI model. A generative AI model is an algorithm that generates specific operational steps based on the user's request. In response to the prompt "aggregate sales data and create a monthly sales graph," the AI ​​generates the following operational steps:

[1951] Select a data range

[1952] Click the Insert tab

[1953] Select the pivot table

[1954] Drag a date to a row

[1955] Drag Sales Amount to Values

[1956] Create a bar chart

[1957] Input: JSON format prompt statement

[1958] Output: List of specific operating procedures

[1959] Step 4: Converting the Instructions to GUI Format

[1960] The server converts the generated specific operation steps into a GUI format. To make it visually easy for users to understand, each step is formatted as a button or interactive interface. For example, the message "Please select a data range" is displayed as an appropriate button.

[1961] Input: List of operating procedures

[1962] Output: GUI format operation procedure

[1963] Step 5: Sending Instruction Data

[1964] The server sends the GUI-formatted step data to the terminal in JSON format using an HTTP response. For example, the following data is sent: {"steps": ["Select a data range", "Click the Insert tab and select a pivot table"]}.

[1965] Input: GUI-style operation procedure

[1966] Output: Procedure data in JSON format

[1967] Step 6: Displaying the procedure on the GUI

[1968] The terminal analyzes the received procedure data and visually displays specific operation procedures on the GUI. To make it easier for the user to understand, the procedures are displayed on the screen as pop-up windows or information bars. This allows the user to follow the instructions to proceed with the operation.

[1969] Input: Procedure data in JSON format

[1970] Output: Visual operation procedure on GUI

[1971] Step 7: User interaction

[1972] The user follows the instructions displayed on the device's GUI to perform operations on Excel or Google Spreadsheets. Specifically, the user follows the instructions to perform actions such as "select a data range," "click the Insert tab," and "select a pivot table."

[1973] Input: GUI operation procedure

[1974] Output: User completes an action within the software

[1975] Step 8: Emotion Recognition

[1976] The device uses an emotion engine to recognize the user's emotions based on the user's input data and actions (keyboard operations, mouse movements, facial recognition, etc.). For example, it uses a camera and facial recognition software to identify emotional states such as "happiness," "anxiety," and "excitement."

[1977] Input: User behavior data

[1978] Output: User emotion data

[1979] Step 9: Sending Emotion Data

[1980] The device sends the analyzed emotion data to the server in JSON format via an HTTP request. For example, data such as "{"emotion": "anxiety"}" is sent.

[1981] Input: User emotion data

[1982] Output: Emotion data in JSON format

[1983] Step 10: Adjusting procedures based on emotions

[1984] The server receives the user's emotional data and adjusts the generated operation instructions based on the user's emotional state. For example, if the user is feeling "anxious," the server may add more detailed instructions to the operation instructions.

[1985] Input: Emotion data in JSON format

[1986] Output: Adjusted operating procedure

[1987] Step 11: Gather feedback and improve the model

[1988] After the user completes the operation, the device collects feedback and sends it to the server. The user feedback is used to improve the accuracy of the generative AI model and emotion engine. For example, the model can be adjusted based on feedback such as "The instructions were difficult to understand."

[1989] Input: User feedback

[1990] Output: Improved generative AI models and emotion engines

[1991] (Application example 2)

[1992] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1993] In conventional data analysis support systems, when users perform data analysis tasks on software applications, they must decide the specific operating procedures themselves, making the process complicated and difficult to perform efficiently. Furthermore, when users feel emotionally confused or anxious, the system often fails to provide appropriate support, slowing down the progress of the task. In particular, in everyday tasks such as inventory management and sales forecasting in brick-and-mortar stores, where fast and accurate data analysis is required, there is a lack of flexible operational support that responds to the user's emotions.

[1994] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1995] In this invention, the server includes means for generating specific operating procedures based on a user's request using a generative AI model, means for converting the generated specific operating procedures into a GUI format, and means for recognizing the user's emotions using an emotion engine and adjusting the generated operating procedures based on the emotions. This allows the user to receive emotion-sensitive support while quickly and accurately analyzing data, enabling efficient inventory management and sales forecasting in physical stores.

[1996] A "target information set" is a data set that a user can manipulate and analyze within a software application.

[1997] "Final output" refers to the result that a user wants to achieve using a software application.

[1998] "Terminal" refers to a device used by a user, such as a smartphone, tablet, or PC.

[1999] The "server" is a central processing system that processes data received from users and provides operational instructions and support using generative AI models and emotion engines.

[2000] A "generative AI model" refers to an artificial intelligence model that generates specific operating procedures based on user requests.

[2001] "GUI format" refers to a format for visually displaying operating procedures in the form of a graphical user interface.

[2002] An "emotion engine" is a system that recognizes a user's emotions based on input data and actions during user operation and analyzes that data.

[2003] "Emotion data" is data that indicates the user's emotional state analyzed by the emotion engine.

[2004] "Feedback" refers to the opinions, evaluations, and data regarding operations that users provide to the system.

[2005] "Data analysis" is the act of processing a set of information to extract patterns and useful information to produce a final output.

[2006] "Inventory management" refers to the task of understanding the inventory status of products, merchandise, and other materials in physical stores and maintaining an adequate supply.

[2007] "Sales forecasting" is the activity of predicting future sales based on past sales data and current sales trends.

[2008] This invention is a system in which, when a user performs data analysis using a software application that manipulates a set of information, the generative AI model automatically presents specific operating procedures simply by inputting the final output, and further combines this with an emotion engine that recognizes the user's emotions.

[2009] First, the user uses the terminal to input the final output in text format. For example, they might input "I want to forecast this month's inventory" or "I want to identify best-selling products." The terminal then sends this input data in JSON format to the server. Communication is via HTTP requests.

[2010] The server analyzes the received user input data and activates a generative AI model. The generative AI model generates optimal operating procedures based on the user's requests. For example, for inventory forecasting, it presents analytical procedures based on historical sales data and current inventory data. The generated specific operating procedures are converted into GUI format by the server. This conversion provides the procedures in a format that is visually easy for users to understand.

[2011] Furthermore, the device uses an emotion engine to recognize the user's emotions based on input data and actions (e.g., keyboard input and touch operations) during operation. The recognized emotion data is sent to the server in JSON format. The server analyzes the emotion data and adjusts the generated operation instructions to reduce the user's anxiety or confusion. For example, if the user feels anxious, it may provide more detailed explanations of the instructions.

[2012] The terminal receives specific operating instructions from the server and displays them on a GUI. The user operates the software application according to the displayed instructions. This allows the user to perform data analysis efficiently and accurately by following the guide.

[2013] After providing the operation instructions, the server collects feedback from the user and uses it to improve the accuracy of the generative AI model and emotion engine. The feedback information is also sent to the server in JSON format.

[2014] Specifically, a smartphone is used as the terminal, and a generative AI model such as OpenAI's GPT-3 is used on the server side. As an emotion engine, a machine learning framework such as Keras is used to analyze the user's facial expressions and input data.

[2015] (Example)

[2016] Here is an example prompt:

[2017] Example prompt sentence:

[2018] User request: I want to make inventory forecast for this month. Please provide specific steps.

[2019] Based on this prompt, a generative AI model generates instructions, and an emotion engine recognizes the user's emotions and adjusts the instructions accordingly, providing personalized assistance to the user.

[2020] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2021] Step 1:

[2022] A user inputs the final output in text format into a software application containing the set of information to be operated on. Specifically, using a smartphone as a terminal, the user inputs requests such as "I want to forecast this month's inventory" or "I want to identify best-selling products." The input data is saved in text format on the terminal.

[2023] Step 2:

[2024] The terminal sends the user's input data to the server. Communication is via HTTP request, and the data is sent in JSON format. The input data is text information that indicates the user's request, and is sent to the server.

[2025] Step 3:

[2026] The server analyzes the received user input data and activates a generative AI model. Specifically, it uses a generative AI model such as OpenAI's GPT-3 to generate optimal operating procedures based on the user's request. The generative AI model generates appropriate procedures for the request entered by the user and outputs the results in text format to the server.

[2027] Step 4:

[2028] The server converts the generated specific operation procedures into a GUI format. Specifically, it converts the generated textual procedures into a visually understandable format and builds them as GUI components. The converted procedures are saved as output on the server.

[2029] Step 5:

[2030] The server sends specific operation procedures in GUI format to the terminal. The data is sent in JSON format again using an HTTP response. The output data is the operation procedures converted into GUI format, and this is sent transparently to the terminal.

[2031] Step 6:

[2032] The specific operation procedures received by the terminal are displayed on the GUI, allowing the user to check the operation procedures visually rather than in text format. The received data is analyzed and rendered as GUI components.

[2033] Step 7:

[2034] A user operates a software application by following instructions displayed on a GUI. The user follows the instructions to perform specific operations, such as entering inventory data or performing analytical procedures.

[2035] Step 8:

[2036] The device uses an emotion engine to recognize the user's emotions based on the input data and actions of the user during operation. Specifically, the emotion recognition algorithm analyzes the keyboard input and touch operations performed by the user on the smartphone. This emotion data is saved as input data on the device.

[2037] Step 9:

[2038] The device sends the analyzed emotional data to the server. The emotional data is also sent in JSON format via an HTTP request. The input data indicates the user's emotional state and is sent to the server.

[2039] Step 10:

[2040] The server adjusts the generated operation procedures based on the emotional data. Specifically, to reduce the anxiety and confusion felt by the user, the server adjusts the procedures generated by the generative AI model and takes measures such as providing more detailed explanations. These adjusted procedures are saved as output data on the server.

[2041] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2042] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2043] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2044] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2045] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2046] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2047] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2048] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2049] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2050] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2051] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2052] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2053] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2054] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2055] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2056] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2057] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2058] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2059] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2060] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2061] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2062] The following is further disclosed regarding the above embodiment.

[2063] (Claim 1)

[2064] a means for a user to input the final output in text form into a software application containing the dataset to be manipulated;

[2065] A means for the terminal to transmit user input data to the server;

[2066] A means for the server to use the generated AI model to generate specific operation procedures based on a user's request;

[2067] A means to convert the specific operation procedures generated by the server into a GUI format;

[2068] A means for the server to send specific operation procedures in GUI format to the terminal;

[2069] A means for displaying the specific operation procedure received by the terminal on a GUI;

[2070] means for a user to operate the software application according to operating procedures displayed on the GUI;

[2071] A system including:

[2072] (Claim 2)

[2073] 10. The system of claim 1, further comprising means for the terminal to collect and transmit feedback from the user to the server.

[2074] (Claim 3)

[2075] 10. The system of claim 1, wherein the server further comprises means for improving the generative AI model based on the collected feedback.

[2076] "Example 1"

[2077] (Claim 1)

[2078] a means for a user to input a final output in text form into an application containing an information set to be manipulated;

[2079] means for the terminal to transmit user input data to the processing device;

[2080] A means for generating specific operation procedures based on a user's request using the generated artificial intelligence model by the processing device;

[2081] means for the processing device to convert the generated specific operating procedures into a visual display format;

[2082] means for the processing device to transmit specific operation procedures in a visual display format to the terminal;

[2083] means for displaying the specific operation procedures received by the terminal on a visual user interface;

[2084] means for a user to operate the application according to operating procedures displayed on a visual user interface;

[2085] A system including:

[2086] (Claim 2)

[2087] 10. The system of claim 1, further comprising means for the terminal to collect and transmit to the processing unit feedback from the user.

[2088] (Claim 3)

[2089] 10. The system of claim 1, wherein the processing unit further comprises means for improving the generative artificial intelligence model based on the collected feedback.

[2090] "Application Example 1"

[2091] (Claim 1)

[2092] a means for a user to input the final output in text form into a software application containing the dataset to be manipulated;

[2093] A means for the terminal to transmit user input data to the server;

[2094] A means for the server to use the generated AI model to generate specific operation procedures based on a user's request;

[2095] A means to convert the specific operation procedures generated by the server into a GUI format;

[2096] A means for the server to send specific operation procedures in GUI format to the terminal;

[2097] A means for displaying the specific operation procedure received by the terminal on a GUI;

[2098] means for a user to operate the software application according to operating procedures displayed on the GUI;

[2099] a means for a user to input instructions for data analysis and improvement of the manufacturing process in the factory by voice and convert the input instructions into text format;

[2100] A means for the server to generate a manufacturing data analysis procedure and a manufacturing process improvement procedure using the generative AI model and display them in a GUI format;

[2101] A means for users to operate the system in the factory by following the instructions displayed on the GUI of the terminal;

[2102] A system including:

[2103] (Claim 2)

[2104] 10. The system of claim 1, further comprising means for the terminal to collect and transmit feedback from the user to the server.

[2105] (Claim 3)

[2106] 10. The system of claim 1, wherein the server further comprises means for improving the generative AI model based on the collected feedback.

[2107] "Example 2: Combining Emotion Engines"

[2108] (Claim 1)

[2109] means for a user to input the final output in text form into the application software containing the dataset to be manipulated;

[2110] A means for the terminal to transmit user input data to the server;

[2111] A means for the server to use the generated AI model to generate specific operation procedures based on a user's request;

[2112] A means to convert the specific operation procedures generated by the server into a GUI format;

[2113] A means for the server to send specific operation procedures in GUI format to the terminal;

[2114] A means for displaying the specific operation procedure received by the terminal on a GUI;

[2115] A means for a user to operate the application software according to the operating procedure displayed on the GUI;

[2116] A means for the terminal to recognize the user's emotions using an emotion engine based on the user's input data and behavior during operation;

[2117] A means for transmitting the emotion data analyzed by the terminal to a server;

[2118] a server receiving the user's emotional data and adjusting the generated operation procedure based on the user's emotional state;

[2119] A system including:

[2120] (Claim 2)

[2121] 10. The system of claim 1, further comprising means for the terminal to collect and transmit feedback from the user to the server.

[2122] (Claim 3)

[2123] 10. The system of claim 1, wherein the server further comprises means for improving the generative AI model and emotion engine based on the collected feedback.

[2124] "Application example 2 when combining emotion engines"

[2125] (Claim 1)

[2126] a means for a user to input a final output in text form into a software application containing the set of information to be manipulated;

[2127] A means for the terminal to transmit user input data to the server;

[2128] A means for the server to use the generated AI model to generate specific operation procedures based on a user's request;

[2129] A means to convert the specific operation procedures generated by the server into a GUI format;

[2130] A means for the server to send specific operation procedures in GUI format to the terminal;

[2131] A means for displaying the specific operation procedure received by the terminal on a GUI;

[2132] means for a user to operate the software application according to operating procedures displayed on the GUI;

[2133] A means for the terminal to recognize the user's emotions using an emotion engine based on input data and actions during user operation;

[2134] A means for transmitting the emotion data analyzed by the terminal to a server;

[2135] A means for adjusting the operation procedure generated by the server based on the emotion data;

[2136] A system including:

[2137] (Claim 2)

[2138] 10. The system of claim 1, further comprising means for the terminal to collect and transmit feedback from the user to the server.

[2139] (Claim 3)

[2140] 10. The system of claim 1, wherein the server further comprises means for improving the generative AI model based on the collected feedback. [Explanation of symbols]

[2141] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for a user to input the final output in text form into a software application containing the dataset to be manipulated; A means for the terminal to transmit user input data to the server; A means for the server to use the generated AI model to generate specific operation procedures based on a user's request; A means to convert the specific operation procedures generated by the server into a GUI format; A means for the server to send specific operation procedures in GUI format to the terminal; A means for displaying the specific operation procedure received by the terminal on a GUI; means for a user to operate the software application according to operating procedures displayed on the GUI; A system including:

2. The system of claim 1 further comprising means for the terminal to collect and transmit feedback from the user to the server.

3. The system of claim 1 , wherein the server further comprises means for improving the generative AI model based on the collected feedback.

Images