system

Abstract

We provide the system. [Solution] Means of receiving instructions from users, A means of interpreting and analyzing a specific task based on instructions, A means for collecting and pre-processing the necessary data, A means of using generative AI to execute a specified task and generate results, A means of presenting the generated results to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional systems, when a user executes a specific task, many manual operations and advanced expertise are often required. In particular, in complex data analysis tasks such as generating business reports and inventory forecasting, it has been a problem that high technical skills and a great deal of time are required. Also, the collection and preprocessing of necessary data are generally performed manually, which has been a factor in efficiency reduction and errors. In order to solve these problems, a system that allows users to easily execute advanced data analysis tasks and obtain results quickly and accurately has been demanded.

Means for Solving the Problems

[0005] This invention provides a means for receiving instructions from a user, interpreting and analyzing specific tasks based on those instructions, and further providing a system that includes means for automatically collecting and preprocessing necessary data. This allows for the execution of specified tasks using generative AI and the generation of results. The system then includes a function for presenting the generated results to the user. This system enables users to easily perform advanced data analysis tasks and obtain rapid and accurate results. Specifically, it can automatically generate sales reports based on user instructions and perform advanced predictions and analyses using generative AI. This dramatically improves operational efficiency and minimizes the occurrence of errors.

[0006] A "user" is the entity that operates the system and inputs instructions.

[0007] An "instruction" is a specific task or command that a user requests the system to perform.

[0008] A "specific task" refers to a concrete operation performed by the system based on user instructions, such as generating sales reports or forecasting inventory.

[0009] "Analysis" is the process of interpreting input instructions and identifying the necessary steps and data.

[0010] "Data collection" is the act of a system automatically gathering the necessary data from the necessary information sources.

[0011] "Preprocessing" refers to the procedure of normalizing collected data and preparing it for analysis and generation.

[0012] "Generative AI" is an artificial intelligence technology that analyzes data to perform tasks assigned by users and generates appropriate results.

[0013] "Results" refer to the output of information and data generated by a generative AI based on user instructions.

[0014] "Prompting" refers to the act of presenting the generated result to the user in an appropriate format.

[0015] "Report" refers to a report containing documents, graphs, statistical information, etc. generated based on the user's instructions.

[0016] With the above definitions, the intended content of the important words included in the claims of the patent is clarified.

Brief Description of Drawings

[0017] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12]It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Embodiments for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0021] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] The embodiments for carrying out the present invention are described below.

[0039] This system aims to enable users to easily perform advanced data analysis tasks and obtain results quickly and accurately. This allows for the generation of complex reports and predictive analytics without the manual operations and advanced expertise previously required.

[0040] System Overview

[0041] When a user enters a specific instruction into the terminal, the instruction is analyzed and sent to the server. The server collects the necessary data based on the instruction and performs preprocessing. A generative AI then uses the preprocessed data to execute the specified task and presents the results to the user through the terminal.

[0042] Program processing

[0043] 1. User input

[0044] The user enters the instruction "Generate this week's sales report" into the terminal. This input is done via voice recognition or keyboard input.

[0045] 2. Instruction Analysis

[0046] The device receives user instructions and analyzes their content. This analysis uses natural language processing technology to determine the meaning of the instructions.

[0047] 3. Transmission of commands and activation of generative AI

[0048] The terminal sends the analysis results to the server. This is done using network communication.

[0049] The server activates the generative AI based on the received instructions.

[0050] 4. Data Collection

[0051] The server automatically collects the necessary data. For example, it retrieves the required information from customer databases, sales databases, and marketing databases.

[0052] 5. Data preprocessing

[0053] The server preprocesses the collected data. Specifically, this involves data cleaning, normalization, and feature selection. This process prepares the data for analysis.

[0054] 6. Task execution and result generation

[0055] Generative AI uses pre-processed data to perform specified tasks. For example, it can analyze sales trends or perform statistical analysis of customer feedback.

[0056] Generative AI creates reports from generated data and uses text generation technology to produce explanatory text that includes insights.

[0057] 7. Submission and presentation of results

[0058] The server sends the generated report to the terminal. This communication is conducted through a highly secure protocol.

[0059] The terminal analyzes the received report and displays it to the user in an appropriate format. This uses an interactive GUI that includes graphs and text.

[0060] Specific example

[0061] Sales report generation

[0062] 1. The user enters "Generate this week's sales report" into the terminal.

[0063] 2. The terminal analyzes this instruction and sends it to the server.

[0064] 3. The server activates the generative AI and collects and preprocesses the necessary data.

[0065] 4. Generative AI analyzes sales data and generates reports. These reports include monthly sales trends and feedback from key customers.

[0066] 5. The server sends the generated report to the terminal, and the terminal displays it to the user.

[0067] Thus, this system is designed to allow users to easily perform advanced data analysis by automatically collecting, preprocessing, analyzing, generating, and presenting data based on user instructions.

[0068] The following describes the processing flow.

[0069] Step 1:

[0070] The user enters the instruction "Generate this week's sales report" into the terminal.

[0071] The device receives these instructions through voice recognition or text analysis.

[0072] Step 2:

[0073] The device analyzes the user's instructions and extracts the task details from the text.

[0074] The terminal sends the analysis results (task details) to the server.

[0075] Step 3:

[0076] The server analyzes the instructions received from the terminal and determines what type of report needs to be generated.

[0077] The server starts the generative AI and loads the necessary settings.

[0078] Step 4:

[0079] The server collects the necessary data based on instructions. For example, it retrieves data from customer databases, sales databases, marketing databases, etc.

[0080] The server communicates with each database via APIs and data connections to collect necessary information.

[0081] Step 5:

[0082] The server preprocesses the collected data. Specifically, this involves imputing missing values, removing outliers, and standardizing data formats.

[0083] The server prepares the pre-processed data as a dataset for analysis.

[0084] Step 6:

[0085] The generative AI begins its analysis using pre-processed data. Here, it calculates metrics such as sales revenue, conversion rate, and customer feedback.

[0086] The generative AI uses the results to prepare data for generating explanatory text and graphs.

[0087] Step 7:

[0088] The generative AI generates a report based on the analysis results. The report includes insights created using text generation technology, sales trend graphs, and statistical information.

[0089] The generative AI returns the final generated report to the server.

[0090] Step 8:

[0091] The server sends the generated report to the terminal. Communication is usually done using the HTTPS protocol to ensure data security.

[0092] The server records transmission logs for later use in troubleshooting and feedback.

[0093] Step 9:

[0094] The terminal analyzes the received reports and displays them in a user-friendly format. This includes GUI rendering using HTML and CSS.

[0095] The terminal will provide reports in an interactive format, allowing users to quickly obtain the information they need.

[0096] The above outlines the specific processing steps of the system that generates sales reports based on user instructions and presents the results.

[0097] (Example 1)

[0098] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] Traditionally, advanced data analysis and complex report generation required users to possess advanced expertise and manual operation. This resulted in significant time and effort, making it inefficient. This invention solves this problem by providing a system that allows users to perform data analysis and report generation easily and quickly without requiring specialized knowledge or manual operation.

[0100] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0101] In this invention, the server includes means for receiving instructions from a user, means for interpreting tasks based on the instructions and analyzing them using natural language processing technology, means for sending the analyzed instructions to the server and activating a generative AI, means for collecting necessary data from a database, cleaning and normalizing the data, and performing preprocessing to select features, means for executing a specified task based on the preprocessed data using the generative AI and generating results, and means for presenting the generated results to the user in an interactive format on a terminal. This enables users to perform advanced data analysis and report generation in a short time without specialized knowledge.

[0102] 1. "Means for receiving instructions from the user" refers to an interface that allows the user to input instructions to the system using methods such as voice recognition or keyboard input.

[0103] 2. "Natural language processing technology" refers to technologies that enable computers to understand and analyze human language, specifically technologies that perform text analysis and semantic analysis.

[0104] 3. "Means for interpreting and analyzing tasks" refers to the process of analyzing instructions received from the user, understanding their content, and determining the appropriate action to take.

[0105] 4. "Generative AI" refers to artificial intelligence models that have been trained to perform specified tasks, and are models that automatically perform data analysis and report generation.

[0106] 5. "Means of data collection" refers to mechanisms for obtaining necessary data from various databases and external data sources.

[0107] 6. "Data preprocessing means" refers to processes such as cleaning, normalization, and feature selection that are performed to prepare collected data into a format suitable for analysis.

[0108] 7. "Means for executing tasks and generating results" refers to the process by which a generative AI performs specified data analysis and report generation tasks based on pre-processed data and creates the results.

[0109] 8. "Means for presenting the generated results to the user on the terminal" refers to an interface for visually displaying the generated analysis results and reports to the user.

[0110] This invention provides a system that allows users to easily perform advanced data analysis and report generation. Specific embodiments for carrying out this invention are described below.

[0111] System Overview

[0112] This system consists of three main components: a user, a terminal, and a server. The user inputs instructions into the terminal via voice recognition or keyboard input. The terminal analyzes the instructions using natural language processing technology and sends them to the server. The server activates a generative AI to collect and preprocess the necessary data, then generates analysis results and reports, which are presented to the user.

[0113] Hardware and software to use

[0114] 1. Terminal

[0115] Hardware: PCs, tablets, smartphones, etc.

[0116] Software: Google® Speech-to-Text API (speech recognition), Tensorflow® (registered trademark), NLTK (natural language processing)

[0117] 2. Server

[0118] Hardware: Cloud servers, on-premises servers

[0119] Software: MySQL® / PostgreSQL (database), Pandas, Scikit-learn (data preprocessing), GPT-4® (generative AI model)

[0120] 3. Communications

[0121] Protocol: HTTPS (Secure communication)

[0122] Specific examples of actions

[0123] Sales report generation

[0124] 1. The user enters "Generate this week's sales report" into the terminal.

[0125] Example prompt: "Generate this week's sales report."

[0126] 2. The device receives instructions using speech recognition technology (Google Speech-to-Text API) or keyboard input.

[0127] In the case of voice input, the speech is converted to text.

[0128] 3. The device analyzes the instructions using TensorFlow or NLTK.

[0129] Identify tasks based on the keyword "sales report".

[0130] 4. The terminal sends the analyzed commands to the server using the HTTPS protocol.

[0131] 5. The server starts the generative AI model (GPT-4) based on the instructions.

[0132] 6. The server collects sales data and customer data from MySQL or PostgreSQL.

[0133] Execute the SQL query and extract the necessary data.

[0134] 7. The server performs data preprocessing using Pandas or Scikit-learn.

[0135] Cleaning (imputing missing values ​​and deleting invalid data)

[0136] Data normalization (converting values ​​to a scale of 0 to 1)

[0137] Feature selection (select important variables)

[0138] 8. The generative AI model performs the specified task based on the pre-processed data and generates the results.

[0139] We analyze sales trends and customer feedback statistically, and then use text generation technology to generate explanatory text that includes insights.

[0140] 9. The server sends the generated report to the terminal using the HTTPS protocol.

[0141] 10. The terminal presents the received report to the user in an interactive format.

[0142] This program displays graphs using D3.js or Chart.js, along with text styled using HTML and CSS.

[0143] Effects of implementation

[0144] This system allows users to easily perform data analysis and generate reports without requiring advanced expertise. This improves the quality of data-driven decision-making and significantly enhances operational efficiency.

[0145] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0146] Step 1:

[0147] The user enters "Generate this week's sales report" into the terminal. Input is done via voice recognition or keyboard.

[0148] Input: User instruction ("Generate this week's sales report")

[0149] Output: Instruction text displayed on the terminal

[0150] Step 2:

[0151] The device receives user instructions using speech recognition technology (Google Speech-to-Text API) or keyboard input. The received instructions are then analyzed using natural language processing technology (TensorFlow, NLTK, etc.).

[0152] Input: Speech recognition result or keyboard input text

[0153] Output: Analyzed instructions (report generation task)

[0154] Step 3:

[0155] The terminal transfers the analysis results to the server. This communication is securely performed using the HTTPS protocol.

[0156] Input: Analyzed instructions

[0157] Output: Parsing instructions to the server

[0158] Step 4:

[0159] The server launches a generative AI model (GPT-4) based on the analysis results. The generative AI then prepares to execute the specified task.

[0160] Input: Analyzed instructions

[0161] Output: Preparation for starting a generative AI

[0162] Step 5:

[0163] The server collects the necessary data. Specifically, it executes SQL queries to retrieve sales data and customer data from databases (such as MySQL and PostgreSQL).

[0164] Input: Data to be collected as instructed

[0165] Output: Acquired dataset (sales data, customer data, etc.)

[0166] Step 6:

[0167] The server preprocesses the collected data. It uses Pandas and Scikit-learn to perform tasks such as data cleaning, normalization, and feature selection.

[0168] Input: Acquired dataset

[0169] Output: Preprocessed data

[0170] Step 7:

[0171] Generative AI models perform specified tasks based on pre-processed data and generate results. For example, they can analyze sales trends or perform statistical analysis of customer feedback.

[0172] Input: Preprocessed data

[0173] Output: Analysis results and report content

[0174] Step 8:

[0175] The server sends the generated results to the terminal. This communication also uses the HTTPS protocol.

[0176] Input: Generated results (report content)

[0177] Output: Sending results to the terminal

[0178] Step 9:

[0179] The terminal analyzes the received report and displays it to the user in an interactive format. It visualizes the data using D3.js and Chart.js, and styles it with HTML and CSS.

[0180] Input: Report results from the server

[0181] Output: Interactive report display on the user's device.

[0182] (Application Example 1)

[0183] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0184] Current factory operations generate a large amount of data, but analyzing and reporting on it requires advanced expertise and manual operation, making efficient data utilization difficult. Furthermore, there is a need for real-time monitoring of operating conditions and rapid report generation, but there is a lack of effective means to achieve this.

[0185] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0186] In this invention, the server includes means for receiving instructions from a user, means for interpreting and analyzing specific tasks based on the instructions, means for collecting and pre-processing necessary data, means for executing specified tasks using generative AI and generating results, means for presenting the generated results to the user, means for creating an operational status report from the generated data, and means including an interactive interface for displaying the operational status report. This enables the user to easily perform advanced data analysis, generate status reports in real time, and respond quickly.

[0187] "Means of receiving instructions from the user" refers to an interface that receives instructions from the user via voice input or text input, and a system that interprets that input data.

[0188] "A means of interpreting and analyzing a specific task based on instructions" refers to a system that uses natural language processing technology to identify a task from user input and analyze its content.

[0189] "Means for collecting and pre-processing necessary data" refers to processes and systems that collect appropriate data from sensors, databases, etc., and then clean and normalize that data.

[0190] "A means of executing a specified task and generating results using generative AI" refers to a system that utilizes generative artificial intelligence to analyze pre-processed data and generate the necessary results and reports.

[0191] "Means of presenting generated results to the user" refers to a system that displays generated data and reports to the user through an interface.

[0192] "Means for creating operational status reports from generated data" refers to a system that automatically generates reports on operational status based on collected and pre-processed data.

[0193] "Means including an interactive interface for displaying operational status reports" refers to an interface that visually displays the generated reports and allows users to interact with them.

[0194] This invention relates to a system that enables the efficient collection and analysis of data within a factory, as well as the automation of report generation. The aim of this system is to allow users to easily perform advanced data analysis tasks and obtain results quickly and accurately.

[0195] System Overview

[0196] First, the user gives instructions to the robot via voice or text input. For example, an instruction such as "Generate this week's operational status report" is conceivable. The robot uses a voice recognition system to convert these instructions into text and analyzes its content. Specifically, it uses natural language processing technology to determine the meaning of the instructions.

[0197] Hardware and software

[0198] The hardware used in this process includes a voice input device for recognizing the user's voice, the robot body, and a central server. The software includes a voice recognition system (e.g., Dialogflow or other natural language processing (NLP) techniques), Python libraries for data collection and preprocessing (e.g., Pandas, NumPy), and a model as a generative AI (e.g., GPT-4).

[0199] Data collection and preprocessing

[0200] The server collects necessary data from various sensors and databases. PLCs (Programmable Logic Controllers) and IoT devices are used for data collection. The collected data is cleaned and normalized using Python's Pandas and NumPy libraries.

[0201] Report generation

[0202] Generative AI (e.g., GPT-4) uses pre-processed data to perform a specified task and generate results. An operational status report is automatically created from the generated data. This report includes uptime, downtime, and other key metrics.

[0203] Presentation of results

[0204] The generated report is sent from the server to the robot and displayed to the user through the robot's interface. An interactive GUI is used for display, and data visualization using tools such as Tableau or D3.js is envisioned.

[0205] Specific example

[0206] A concrete example is shown below. When a user gives a voice command to the robot, "Generate this week's operational status report," the robot analyzes this command and sends a request to a central server. The server collects the necessary data from various sensors and databases and preprocesses it using Python's Pandas and NumPy. The preprocessed data is then analyzed by GPT-4, and an operational status report is generated. The generated report is sent to the robot and displayed to the user in an interactive format.

[0207] Example of a prompt

[0208] User: "Generate this week's operational status report."

[0209] Robot: "Analyzing the command..."

[0210] Robot: "Collecting data..."

[0211] Robot: "We are pre-processing the data..."

[0212] Robot: "Generating report..."

[0213] Robot: "Here is this week's operational report. Operating hours and downtime are as follows..."

[0214] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0215] Step 1:

[0216] Receiving and analyzing user instructions

[0217] The user instructs the robot via voice or text input, "Generate this week's operational status report." The robot uses a speech recognition system to convert the speech into text and natural language processing technology to analyze the meaning of the instruction. Specifically, a voice input device captures the user's voice, and speech recognition software converts it into text data. Next, a natural language processing engine analyzes the instruction and recognizes the specific task (in this case, generating an operational status report).

[0218] Input: User's voice instructions

[0219] Output: Analyzed text instructions

[0220] Step 2:

[0221] Transmission of instructions to the server

[0222] The terminal sends the analysis results to the server. This communication is secure using the HTTPS protocol. Specifically, the robot sends the analyzed text data to the server via the network communication module. The server receives this instruction and proceeds to the next data collection process.

[0223] Input: Parsed text instructions

[0224] Output: Instruction data to the server

[0225] Step 3:

[0226] Data collection

[0227] The server collects the necessary data from sensors and databases within the factory. PLCs and IoT devices are used to collect data from sensors. The server sends data requests back to each device and receives operational status data in return.

[0228] Input: Instruction data to the server

[0229] Output: Collected raw data

[0230] Step 4:

[0231] Data preprocessing

[0232] The server preprocesses the collected data. Preprocessing includes data cleaning (imputing missing values ​​and removing outliers), normalization, and feature selection. Specifically, it uses Python's Pandas and NumPy to generate dataframes and perform cleanup and standardization operations.

[0233] Input: Collected raw data

[0234] Output: Preprocessed data

[0235] Step 5:

[0236] Report generation

[0237] The server uses pre-processed data to execute specified tasks using a generative AI (e.g., GPT-4) and generate reports. The generative AI analyzes the input data and automatically creates operational status reports. Specifically, the generative AI model takes in pre-processed data and performs statistics on uptime and downtime analysis.

[0238] Input: Preprocessed data

[0239] Output: Generated operational status report

[0240] Step 6:

[0241] Presentation of results

[0242] The server sends the generated report to the robot, which then presents it to the user. An interactive GUI is used for the display, allowing the user to visually check the operational status. Specifically, visualization tools such as Tableau and D3.js are used.

[0243] Input: Generated operational status report

[0244] Output: An interactive report displayed to the user.

[0245] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0246] The present invention will now describe embodiments for carrying out the present invention. The present invention is a system that combines user emotion recognition technology, enabling users to easily perform advanced data analysis tasks and obtain results quickly and accurately.

[0247] System Overview

[0248] When a user enters a specific instruction into the device, the instruction is analyzed and sent to the server. The server collects the necessary data based on the instruction and performs preprocessing. A generative AI uses the preprocessed data to execute the specified task and presents the results to the user through the device. In addition, an emotion engine recognizes the user's emotions and reflects them in the generated results and how they are presented.

[0249] Program processing

[0250] 1. User input

[0251] The user inputs a command into the terminal, such as "Generate this week's sales report." This input is done via voice recognition or keyboard input.

[0252] 2. Instruction Analysis

[0253] The device analyzes the user's instructions and extracts task details from the text. Natural language processing technology is used for the analysis.

[0254] The terminal sends the analysis results (task details) to the server.

[0255] 3. Transmission of commands and activation of generative AI

[0256] The terminal sends the analysis results to the server. Network protocols are used for communication.

[0257] The server analyzes the received instructions and determines what type of report needs to be generated.

[0258] The server starts the generative AI and loads the necessary settings.

[0259] 4. Activation of the Emotional Engine

[0260] The device collects the user's voice and facial expressions through its camera and microphone and transmits them to the emotion engine.

[0261] The emotion engine analyzes the user's emotions and sends the results to the server.

[0262] 5. Data Collection

[0263] The server automatically collects the necessary data, including data retrieval from customer databases, sales databases, and marketing databases.

[0264] The server communicates with each database via APIs and data connections to collect necessary information.

[0265] 6. Data preprocessing

[0266] The server preprocesses the collected data, performing tasks such as imputing missing values, removing outliers, and standardizing data formats.

[0267] The server prepares the pre-processed data as a dataset for analysis.

[0268] 7. Task execution and result generation

[0269] The generative AI begins analysis using pre-processed data. For example, it calculates metrics such as sales revenue, conversion rate, and customer feedback.

[0270] Generative AI creates reports from generated data, and text generation technology generates explanatory text that includes insights.

[0271] 8. Submission and presentation of results

[0272] The server sends the generated report to the terminal. This communication uses the HTTPS protocol to ensure data security.

[0273] The device analyzes the received reports and adjusts the presentation method based on the user's perceived emotions. The format and emphasis of the presentation will differ depending on whether the user is relaxed or stressed.

[0274] Specific example

[0275] Sales report generation and sentiment feedback

[0276] 1. The user enters "Generate this week's sales report" into the terminal.

[0277] 2. The terminal analyzes the instructions and sends the information to the server.

[0278] 3. The server activates the generative AI and collects and preprocesses the necessary data.

[0279] 4. The emotion engine analyzes the user's emotions and sends the results to the server. For example, it might determine that the user is feeling stressed.

[0280] 5. Generative AI analyzes sales data and generates reports. These reports include sales trends and feedback from key customers.

[0281] 6. The server transmits the generated port and emotion information to the terminal, and the terminal displays it to the user.

[0282] In this way, this system automatically performs data collection, preprocessing, analysis, result generation, and presentation based on the user's instructions. Also, by recognizing the user's emotions and adjusting the result presentation method based on them, the user experience can be improved.

[0283] The following describes the processing flow.

[0284] Step 1:

[0285] The user inputs an instruction to the terminal saying "Generate this week's sales report".

[0286] The terminal receives this instruction through voice recognition or text input.

[0287] Step 2:

[0288] The terminal analyzes the user's instruction. Using natural language processing technology, it extracts the task content (sales report generation) from the text.

[0289] The terminal transmits the analysis result (task content) to the server.

[0290] Step 3:

[0291] The terminal collects the user's voice and facial expression data through the camera and microphone.

[0292] The terminal transmits the collected data to the emotion engine.

[0293] Step 4:

[0294] The emotion engine analyzes the voice and facial expression data and determines the user's emotional state (e.g., relaxed, stressed, etc.).

[0295] The emotion engine sends the analysis results (emotion information) to the server.

[0296] Step 5:

[0297] The server analyzes the instruction content and emotion information received from the terminal.

[0298] The server starts the generative AI and loads the necessary settings.

[0299] Step 6:

[0300] The server automatically collects the necessary data based on the instructions. This includes pulling data from customer databases, sales databases, marketing databases, etc.

[0301] The server communicates with each database through APIs and data connections to collect the necessary information.

[0302] Step 7:

[0303] The server performs preprocessing on the collected data. Specifically, it performs data cleaning (completing missing values and removing outliers), normalization, and selection of necessary attributes, etc.

[0304] The server arranges the preprocessed data into a format that can be used by the generative AI for analysis.

[0305] Step 8:

[0306] The generative AI executes tasks using the preprocessed data. Here, it calculates metrics such as sales volume, conversion rate, and customer feedback, etc.

[0307] Based on the analysis results, the generative AI prepares data for generating explanations and graphs.

[0308] Step 9:

[0309] The generative AI uses text generation technology to create a report based on the analysis results. The report includes insights, sales trend graphs, and statistical information.

[0310] The generative AI returns the final generated report to the server.

[0311] Step 10:

[0312] The server sends the generated report and sentiment information to the terminal. The communication uses the HTTPS protocol to ensure data security.

[0313] The server records transmission logs for later use in troubleshooting and feedback.

[0314] Step 11:

[0315] Based on the reports received by the device, the presentation method is adjusted according to the user's perceived emotions. For example, if the user is feeling stressed, the results are displayed in a concise and easy-to-understand format.

[0316] The terminal displays the generated report on the screen, allowing the user to review it.

[0317] Following the steps described above, the present invention automatically performs data collection, preprocessing, analysis, result generation, and presentation based on user instructions. Furthermore, it improves the user experience by recognizing the user's emotions and adjusting the result presentation method accordingly.

[0318] (Example 2)

[0319] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0320] Conventional data analysis systems often required numerous manual operations from the user's input of specific instructions to obtaining results, resulting in lengthy result generation times. Furthermore, they struggled to respond flexibly to the user's emotional state, sometimes leading to a diminished user experience. To address these challenges, it is necessary to provide a system that allows users to easily, quickly, and accurately obtain data analysis results while also presenting appropriate information based on their emotional state.

[0321] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0322] In this invention, the server includes means for receiving instructions from a user, means for interpreting and analyzing specific tasks based on the instructions, means for collecting and pre-processing necessary data, means for executing a specified task using generative artificial intelligence and generating results, means for recognizing the user's emotions, means for reflecting the user's emotions in the generated results and how they are presented, and means for presenting the generated results to the user. This makes it possible to automate everything from receiving instructions from the user to data collection, analysis, result generation, and flexible information presentation based on emotions, thereby improving the user experience.

[0323] A "user" is a person or entity that operates a system and provides instructions or inputs.

[0324] "Instructions" refer to the specific operations or commands that a user performs on the system.

[0325] A "task" is a specific task or process that a system should perform based on instructions.

[0326] "Analysis" refers to data processing and analysis that interprets user instructions and translates them into specific tasks.

[0327] "Data" is a collection of information that a system needs to perform a task.

[0328] "Data collection" is the process of obtaining necessary data from various data sources.

[0329] "Preprocessing" refers to the process of preparing collected data for analysis or task execution.

[0330] "Generative artificial intelligence" refers to artificial intelligence technology used to perform specified tasks and generate results.

[0331] "Results" refer to the output or outcome obtained when a generative artificial intelligence performs a task.

[0332] "Emotions" are internal reactions that indicate a user's psychological state or feedback.

[0333] "Recognition" is the process of analyzing the user's voice and facial expressions to determine their emotions.

[0334] "Presentation" refers to the act of showing the generated results to the user visually or audibly.

[0335] The system of this invention performs advanced data analysis tasks based on user instructions and provides the results quickly and accurately. Furthermore, it can recognize the user's emotions and adjust the way the results are presented accordingly. This system is comprised of the following key hardware and software components.

[0336] User input

[0337] When users input instructions into the terminal, they use voice recognition technology (e.g., a voice recognition API) or keyboard input. The input is a prompt message such as "Generate this week's sales report."

[0338] Instruction parsing

[0339] The device uses speech recognition technology (speech recognition API) to convert voice input into text in order to analyze user instructions. Next, it uses natural language processing technology (natural language processing API) to analyze the input text and extract the task details. The analysis results are stored in a database and sent from the device to the server.

[0340] Transmission of commands and activation of generative AI

[0341] The HTTPS protocol is used when the terminal sends the analyzed instructions to the server. Based on the received analysis results, the server determines what type of report to generate. The server then activates the generative artificial intelligence (generative artificial intelligence API) and loads the necessary settings.

[0342] Emotional engine activation

[0343] The device collects the user's voice and facial expression data through its camera and microphone and sends it to an emotion engine (emotion recognition API). The emotion engine analyzes the user's emotions and sends the results to a server.

[0344] Data collection

[0345] When the server automatically collects the necessary data, it retrieves data from external data sources such as customer databases, sales databases, and marketing databases through APIs and data connections.

[0346] Data preprocessing

[0347] The server preprocesses the collected data. This includes imputing missing values, removing outliers, and standardizing data formats, preparing the dataset for analysis.

[0348] Task execution and result generation

[0349] Generative artificial intelligence begins analysis using pre-processed data. For example, it calculates metrics such as sales revenue, conversion rate, and customer feedback. A report is created from the generated data, and explanatory text containing insights is generated using text generation technology.

[0350] Sending and presenting results

[0351] The server sends the generated report to the terminal using the HTTPS protocol. The terminal analyzes the received report and adjusts the presentation method based on the user's emotion recognition results. For example, if the user is stressed, important information is highlighted and displayed clearly. Conversely, if the user is relaxed, detailed information is also presented.

[0352] Specific example

[0353] Sales report generation and sentiment feedback

[0354] The user enters a command into the terminal, such as "Generate this week's sales report." The terminal analyzes this command and sends it to the server. The server collects the necessary data and preprocesses it. The emotion engine analyzes the user's emotions and sends the results to the server. For example, if it determines that the user is feeling stressed, it appropriately highlights important information in the report. Generative artificial intelligence analyzes the sales data and generates the report. The server then sends the generated report to the terminal, which displays it to the user.

[0355] This allows users to obtain data analysis results quickly and accurately, and enables the presentation of information tailored to the user's emotional state.

[0356] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0357] Step 1:

[0358] The user inputs the command "Generate this week's sales report" into the device. Speech recognition technology (such as a speech recognition API) is used to convert the voice input into text. The input data is output in text format as "Generate this week's sales report".

[0359] Step 2:

[0360] The terminal analyzes the input text using natural language processing technology (such as a natural language processing API). Specifically, it extracts the keywords "generate," "sales report," and "this week," and recognizes them as tasks. The analysis results are output as JSON data containing the task details, such as "Generate Sales Report."

[0361] Step 3:

[0362] The terminal sends the analysis results to the server using the HTTPS protocol. The transmitted data is in JSON format and contains task details. The server checks the received instructions and determines which type of task to execute. Based on this determination, the instruction "Generate Sales Report" is issued.

[0363] Step 4:

[0364] The device collects user voice and facial expression data through its camera and microphone and sends it to an emotion engine (emotion recognition API). For example, it analyzes stress levels from the user's face. The emotion engine analyzes the emotional state and outputs the result to the server as data such as "the user is feeling stressed."

[0365] Step 5:

[0366] The server collects the necessary data based on the "Sales Report Generation" task. For example, it retrieves data from external data sources such as customer databases, sales databases, and marketing databases via APIs. The collected data includes customer information and sales data. This data is aggregated on the server in an integrated format.

[0367] Step 6:

[0368] The server preprocesses the collected data. Specifically, it imputes missing values, removes outliers, and standardizes the data format. The preprocessed dataset is output in a format suitable for analysis.

[0369] Step 7:

[0370] Generative artificial intelligence performs tasks using pre-processed data. Specifically, it calculates metrics such as sales trends, conversion rates, and customer feedback. Based on the analysis results, the generative AI creates a report and generates explanatory text containing insights using text generation technology. The generated report is output in text and graph formats.

[0371] Step 8:

[0372] The server sends the generated report to the terminal using the HTTPS protocol. The transmitted data includes not only the report information but also the user's emotional state obtained from the emotion engine.

[0373] Step 9:

[0374] The device analyzes the received report and adjusts the presentation method based on the user's emotional state. For example, if the user is feeling stressed, important parts of the report will be highlighted. The presentation method ensures that the information is presented in a format that is easiest for the user to understand.

[0375] Example prompt statements

[0376] "Generate this week's sales report."

[0377] "A compilation of the latest customer feedback"

[0378] "Tell me your sales forecast for next week."

[0379] The above outlines the detailed processing steps of the system. This process allows users to obtain data analysis results quickly and accurately, and further facilitates the presentation of information tailored to the user's emotional state.

[0380] (Application Example 2)

[0381] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0382] Traditional in-store customer service struggled to recognize customer emotions in real time and suggest appropriate products and services based on those emotions. Furthermore, the lack of systems to provide feedback and recommendations tailored to customer emotional states prevented improvements in the customer experience. This resulted in problems such as decreased customer satisfaction and reduced purchasing intent.

[0383] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving instructions from the user, means for interpreting and analyzing specific tasks based on the instructions, and means for recognizing the user's emotions and adjusting the method of presenting the results generated based on those emotions. This makes it possible to recognize customer emotions in real time in a physical store and propose appropriate products and services based on those emotions.

[0384] "Means for receiving instructions from the user" refers to an interface for receiving voice or text instructions entered by the user through a device.

[0385] "Means for interpreting and analyzing specific tasks based on instructions" refers to a process that uses natural language processing technology to analyze the user's instructions and determine which task to execute.

[0386] "Means for collecting and pre-processing necessary data" refers to data processing functions that collect necessary information from diverse data sources and perform pre-processing such as data cleansing and formatting standardization.

[0387] "A means of executing a specified task and generating results using generative AI" refers to an algorithm that uses a generative AI model to perform a specified task from pre-processed data and generate results.

[0388] "Means for recognizing user emotions and adjusting the presentation method of results generated based on those emotions" refers to technology that analyzes user emotions from facial expressions and voice, and presents the results in the most optimal format based on the analysis results.

[0389] "Means of presenting the generated results to the user" refers to UI / UX functions for displaying the generated results on the user's device.

[0390] A description of embodiments for carrying out the present invention will be provided.

[0391] System Configuration

[0392] This system can recognize user emotions and provide personalized product and service recommendations based on those emotions. The system consists of the following components:

[0393] 1. Devices: These include user devices such as smart glasses and smartphones. They are used to collect user input and emotional data.

[0394] 2. Server: The cloud server performs instruction analysis, data collection, preprocessing, activation of generative AI, result generation, and result presentation.

[0395] 3. Emotion Recognition Engine: This engine analyzes data collected by the device through its camera and microphone to recognize the user's emotions. For example, Microsoft® Azure®'s Emotion API is used for this engine.

[0396] 4. Generative AI: AI modules that perform tasks using pre-processed data and generate results. For example, OpenAI's GPT model is used.

[0397] Data collection and analysis

[0398] The device collects the user's voice and facial expression data in real time. This data is sent to an emotion recognition engine to recognize the user's emotions. The emotion-recognized data is sent to a server and analyzed along with the user's instructions.

[0399] The analysis uses natural language processing technology (e.g., Google Cloud Natural Language API) to convert the instructions into tasks. For example, if a user enters "I'm looking for new summer clothes," this instruction is parsed into the task "Recommend new summer items."

[0400] Data collection and preprocessing

[0401] The server collects the necessary data from the database based on the analyzed task. This data may include customer purchase history, inventory information, and promotional information. The collected data is preprocessed (data cleansing, formatting, etc.) and prepared in a format suitable for generative AI.

[0402] Task execution using generative AI

[0403] A generative AI analyzes pre-processed data to generate optimal product and service recommendations based on the user's emotions and preferences. The generated results are then sent back to the terminal via the server.

[0404] Presentation of results

[0405] The device displays results generated based on the user's emotions. It adjusts the display method and emphasis depending on whether the user is relaxed or stressed. For example, it highlights relaxing items for tense users and recommends stimulating items for excited users.

[0406] Specific example

[0407] For example, if a customer voice-inputs "I'm looking for new summer clothes" into smart glasses, the emotion recognition engine detects that the customer is relaxed. The system then generates a list of the latest summer items and displays them in a relaxed and easy-to-view format.

[0408] Example of a prompt

[0409] "Please generate a list of summer products that customers can easily access."

[0410] As described above, the present invention can provide a more personalized user experience by recognizing the user's emotions and making recommendations based on them. Furthermore, by combining generative AI with an emotion recognition engine, it becomes possible to respond appropriately to the diverse needs of users.

[0411] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0412] Step 1:

[0413] The user inputs instructions into the device. The user inputs instructions via voice or text using smart glasses or a smartphone. These instructions are sent to the device as input data. For example, the instruction "I want new summer clothes" is entered into the device.

[0414] Step 2:

[0415] The device collects the user's emotions. The device uses its camera and microphone to capture the user's facial expressions and tone of voice in real time and sends this to an emotion recognition engine. The emotion recognition engine (e.g., Microsoft Azure's Emotion API) analyzes the collected data and recognizes the user's emotions (e.g., relaxed, excited, stressed). This recognition result is returned to the device as emotion data.

[0416] Step 3:

[0417] The device analyzes the user's instructions and extracts specific tasks. The device uses natural language processing technology (e.g., Google Cloud Natural Language API) to analyze instructions entered via voice or text. The analyzed instructions are sent to the server as text data. For example, the instruction "I'm looking for new summer clothes" is converted into the task "Recommend new summer items."

[0418] Step 4:

[0419] The server collects and preprocesses the necessary data. The server accesses databases (e.g., customer purchase history, inventory information, promotional information) and collects the required data based on the task. The collected data undergoes preprocessing such as imputing missing values, removing outliers, and standardizing data formats. The preprocessed data is then sent to the generative AI.

[0420] Step 5:

[0421] The server uses generative AI to perform the specified task. The generative AI (e.g., OpenAI GPT model) generates a list of optimal products and services based on pre-processed data, taking into account the user's emotions and instructions. This generation process uses the prompt "Generate a list of summer products that are easily accessible to the customer." The generated results are returned to the server as recommendation data.

[0422] Step 6:

[0423] The server sends recommendation data to the device. The server sends the generated results to the device using the HTTPS protocol to ensure data security. The final presented data is then sent to the device.

[0424] Step 7:

[0425] The device presents results based on the user's emotions. Based on the user's emotional data, the device adjusts the presentation method and emphasis. For example, if the user is relaxed, the results are displayed in a calm tone; if they are stressed, the results are displayed more intuitively and concisely. The user then reviews the results and makes a selection of products or services.

[0426] Through these steps, personalized product and service recommendations that take user emotions into consideration are achieved.

[0427] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0428] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0429] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0430] [Second Embodiment]

[0431] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0432] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0433] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0434] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0435] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0436] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0437] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0438] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0439] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0440] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0441] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0442] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0443] The embodiments for carrying out the present invention are described below.

[0444] This system aims to enable users to easily perform advanced data analysis tasks and obtain results quickly and accurately. This allows for the generation of complex reports and predictive analytics without the manual operations and advanced expertise previously required.

[0445] System Overview

[0446] When a user enters a specific instruction into the terminal, the instruction is analyzed and sent to the server. The server collects the necessary data based on the instruction and performs preprocessing. A generative AI then uses the preprocessed data to execute the specified task and presents the results to the user through the terminal.

[0447] Program processing

[0448] 1. User input

[0449] The user enters the instruction "Generate this week's sales report" into the terminal. This input is done via voice recognition or keyboard input.

[0450] 2. Instruction Analysis

[0451] The device receives user instructions and analyzes their content. This analysis uses natural language processing technology to determine the meaning of the instructions.

[0452] 3. Transmission of commands and activation of generative AI

[0453] The terminal sends the analysis results to the server. This is done using network communication.

[0454] The server activates the generative AI based on the received instructions.

[0455] 4. Data Collection

[0456] The server automatically collects the necessary data. For example, it retrieves the required information from customer databases, sales databases, and marketing databases.

[0457] 5. Data preprocessing

[0458] The server preprocesses the collected data. Specifically, this involves data cleaning, normalization, and feature selection. This process prepares the data for analysis.

[0459] 6. Task execution and result generation

[0460] Generative AI uses pre-processed data to perform specified tasks. For example, it can analyze sales trends or perform statistical analysis of customer feedback.

[0461] Generative AI creates reports from generated data and uses text generation technology to produce explanatory text that includes insights.

[0462] 7. Submission and presentation of results

[0463] The server sends the generated report to the terminal. This communication is conducted through a highly secure protocol.

[0464] The terminal analyzes the received report and displays it to the user in an appropriate format. This uses an interactive GUI that includes graphs and text.

[0465] Specific example

[0466] Sales report generation

[0467] 1. The user enters "Generate this week's sales report" into the terminal.

[0468] 2. The terminal analyzes this instruction and sends it to the server.

[0469] 3. The server activates the generative AI and collects and preprocesses the necessary data.

[0470] 4. Generative AI analyzes sales data and generates reports. These reports include monthly sales trends and feedback from key customers.

[0471] 5. The server sends the generated report to the terminal, and the terminal displays it to the user.

[0472] Thus, this system is designed to allow users to easily perform advanced data analysis by automatically collecting, preprocessing, analyzing, generating, and presenting data based on user instructions.

[0473] The following describes the processing flow.

[0474] Step 1:

[0475] The user enters the instruction "Generate this week's sales report" into the terminal.

[0476] The device receives these instructions through voice recognition or text analysis.

[0477] Step 2:

[0478] The device analyzes the user's instructions and extracts the task details from the text.

[0479] The terminal sends the analysis results (task details) to the server.

[0480] Step 3:

[0481] The server analyzes the instructions received from the terminal and determines what type of report needs to be generated.

[0482] The server starts the generative AI and loads the necessary settings.

[0483] Step 4:

[0484] The server collects the necessary data based on instructions. For example, it retrieves data from customer databases, sales databases, marketing databases, etc.

[0485] The server communicates with each database via APIs and data connections to collect necessary information.

[0486] Step 5:

[0487] The server preprocesses the collected data. Specifically, this involves imputing missing values, removing outliers, and standardizing data formats.

[0488] The server prepares the pre-processed data as a dataset for analysis.

[0489] Step 6:

[0490] The generative AI begins its analysis using pre-processed data. Here, it calculates metrics such as sales revenue, conversion rate, and customer feedback.

[0491] The generative AI uses the results to prepare data for generating explanatory text and graphs.

[0492] Step 7:

[0493] The generative AI generates a report based on the analysis results. The report includes insights created using text generation technology, sales trend graphs, and statistical information.

[0494] The generative AI returns the final generated report to the server.

[0495] Step 8:

[0496] The server sends the generated report to the terminal. Communication is usually done using the HTTPS protocol to ensure data security.

[0497] The server records transmission logs for later use in troubleshooting and feedback.

[0498] Step 9:

[0499] The terminal analyzes the received reports and displays them in a user-friendly format. This includes GUI rendering using HTML and CSS.

[0500] The terminal will provide reports in an interactive format, allowing users to quickly obtain the information they need.

[0501] The above outlines the specific processing steps of the system that generates sales reports based on user instructions and presents the results.

[0502] (Example 1)

[0503] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0504] Traditionally, advanced data analysis and complex report generation required users to possess advanced expertise and manual operation. This resulted in significant time and effort, making it inefficient. This invention solves this problem by providing a system that allows users to perform data analysis and report generation easily and quickly without requiring specialized knowledge or manual operation.

[0505] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0506] In this invention, the server includes means for receiving instructions from a user, means for interpreting tasks based on the instructions and analyzing them using natural language processing technology, means for sending the analyzed instructions to the server and activating a generative AI, means for collecting necessary data from a database, cleaning and normalizing the data, and performing preprocessing to select features, means for executing a specified task based on the preprocessed data using the generative AI and generating results, and means for presenting the generated results to the user in an interactive format on a terminal. This enables users to perform advanced data analysis and report generation in a short time without specialized knowledge.

[0507] 1. "Means for receiving instructions from the user" refers to an interface that allows the user to input instructions to the system using methods such as voice recognition or keyboard input.

[0508] 2. "Natural language processing technology" refers to technologies that enable computers to understand and analyze human language, specifically technologies that perform text analysis and semantic analysis.

[0509] 3. "Means for interpreting and analyzing tasks" refers to the process of analyzing instructions received from the user, understanding their content, and determining the appropriate action to take.

[0510] 4. "Generative AI" refers to artificial intelligence models that have been trained to perform specified tasks, and are models that automatically perform data analysis and report generation.

[0511] 5. "Means of data collection" refers to mechanisms for obtaining necessary data from various databases and external data sources.

[0512] 6. "Data preprocessing means" refers to processes such as cleaning, normalization, and feature selection that are performed to prepare collected data into a format suitable for analysis.

[0513] 7. "Means for executing tasks and generating results" refers to the process by which a generative AI performs specified data analysis and report generation tasks based on pre-processed data and creates the results.

[0514] 8. "Means for presenting the generated results to the user on the terminal" refers to an interface for visually displaying the generated analysis results and reports to the user.

[0515] This invention provides a system that allows users to easily perform advanced data analysis and report generation. Specific embodiments for carrying out this invention are described below.

[0516] System Overview

[0517] This system consists of three main components: a user, a terminal, and a server. The user inputs instructions into the terminal via voice recognition or keyboard input. The terminal analyzes the instructions using natural language processing technology and sends them to the server. The server activates a generative AI to collect and preprocess the necessary data, then generates analysis results and reports, which are presented to the user.

[0518] Hardware and software to use

[0519] 1. Terminal

[0520] Hardware: PCs, tablets, smartphones, etc.

[0521] Software: Google Speech-to-Text API (speech recognition), TensorFlow, NLTK (natural language processing)

[0522] 2. Server

[0523] Hardware: Cloud servers, on-premises servers

[0524] Software: MySQL / PostgreSQL (databases), Pandas, Scikit-learn (data preprocessing), GPT-4 (generative AI model)

[0525] 3. Communications

[0526] Protocol: HTTPS (Secure communication)

[0527] Specific examples of actions

[0528] Sales report generation

[0529] 1. The user enters "Generate this week's sales report" into the terminal.

[0530] Example prompt: "Generate this week's sales report."

[0531] 2. The device receives instructions using speech recognition technology (Google Speech-to-Text API) or keyboard input.

[0532] In the case of voice input, the speech is converted to text.

[0533] 3. The device analyzes the instructions using TensorFlow or NLTK.

[0534] Identify tasks based on the keyword "sales report".

[0535] 4. The terminal sends the analyzed commands to the server using the HTTPS protocol.

[0536] 5. The server starts the generative AI model (GPT-4) based on the instructions.

[0537] 6. The server collects sales data and customer data from MySQL or PostgreSQL.

[0538] Execute the SQL query and extract the necessary data.

[0539] 7. The server performs data preprocessing using Pandas or Scikit-learn.

[0540] Cleaning (imputing missing values ​​and deleting invalid data)

[0541] Data normalization (converting values ​​to a scale of 0 to 1)

[0542] Feature selection (select important variables)

[0543] 8. The generative AI model performs the specified task based on the pre-processed data and generates the results.

[0544] We analyze sales trends and customer feedback statistically, and then use text generation technology to generate explanatory text that includes insights.

[0545] 9. The server sends the generated report to the terminal using the HTTPS protocol.

[0546] 10. The terminal presents the received report to the user in an interactive format.

[0547] This program displays graphs using D3.js or Chart.js, along with text styled using HTML and CSS.

[0548] Effects of implementation

[0549] This system allows users to easily perform data analysis and generate reports without requiring advanced expertise. This improves the quality of data-driven decision-making and significantly enhances operational efficiency.

[0550] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0551] Step 1:

[0552] The user enters "Generate this week's sales report" into the terminal. Input is done via voice recognition or keyboard.

[0553] Input: User instruction ("Generate this week's sales report")

[0554] Output: Instruction text displayed on the terminal

[0555] Step 2:

[0556] The device receives user instructions using speech recognition technology (Google Speech-to-Text API) or keyboard input. The received instructions are then analyzed using natural language processing technology (TensorFlow, NLTK, etc.).

[0557] Input: Speech recognition result or keyboard input text

[0558] Output: Analyzed instructions (report generation task)

[0559] Step 3:

[0560] The terminal transfers the analysis results to the server. This communication is securely performed using the HTTPS protocol.

[0561] Input: Analyzed instructions

[0562] Output: Parsing instructions to the server

[0563] Step 4:

[0564] The server launches a generative AI model (GPT-4) based on the analysis results. The generative AI then prepares to execute the specified task.

[0565] Input: Analyzed instructions

[0566] Output: Preparation for starting a generative AI

[0567] Step 5:

[0568] The server collects the necessary data. Specifically, it executes SQL queries to retrieve sales data and customer data from databases (such as MySQL and PostgreSQL).

[0569] Input: Data to be collected as instructed

[0570] Output: Acquired dataset (sales data, customer data, etc.)

[0571] Step 6:

[0572] The server preprocesses the collected data. It uses Pandas and Scikit-learn to perform tasks such as data cleaning, normalization, and feature selection.

[0573] Input: Acquired dataset

[0574] Output: Preprocessed data

[0575] Step 7:

[0576] Generative AI models perform specified tasks based on pre-processed data and generate results. For example, they can analyze sales trends or perform statistical analysis of customer feedback.

[0577] Input: Preprocessed data

[0578] Output: Analysis results and report content

[0579] Step 8:

[0580] The server sends the generated results to the terminal. This communication also uses the HTTPS protocol.

[0581] Input: Generated results (report content)

[0582] Output: Sending results to the terminal

[0583] Step 9:

[0584] The terminal analyzes the received report and displays it to the user in an interactive format. It visualizes the data using D3.js and Chart.js, and styles it with HTML and CSS.

[0585] Input: Report results from the server

[0586] Output: Interactive report display on the user's device.

[0587] (Application Example 1)

[0588] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0589] Current factory operations generate a large amount of data, but analyzing and reporting on it requires advanced expertise and manual operation, making efficient data utilization difficult. Furthermore, there is a need for real-time monitoring of operating conditions and rapid report generation, but there is a lack of effective means to achieve this.

[0590] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0591] In this invention, the server includes means for receiving instructions from a user, means for interpreting and analyzing specific tasks based on the instructions, means for collecting and pre-processing necessary data, means for executing specified tasks using generative AI and generating results, means for presenting the generated results to the user, means for creating an operational status report from the generated data, and means including an interactive interface for displaying the operational status report. This enables the user to easily perform advanced data analysis, generate status reports in real time, and respond quickly.

[0592] "Means of receiving instructions from the user" refers to an interface that receives instructions from the user via voice input or text input, and a system that interprets that input data.

[0593] "A means of interpreting and analyzing a specific task based on instructions" refers to a system that uses natural language processing technology to identify a task from user input and analyze its content.

[0594] "Means for collecting and pre-processing necessary data" refers to processes and systems that collect appropriate data from sensors, databases, etc., and then clean and normalize that data.

[0595] "A means of executing a specified task and generating results using generative AI" refers to a system that utilizes generative artificial intelligence to analyze pre-processed data and generate the necessary results and reports.

[0596] "Means of presenting generated results to the user" refers to a system that displays generated data and reports to the user through an interface.

[0597] "Means for creating operational status reports from generated data" refers to a system that automatically generates reports on operational status based on collected and pre-processed data.

[0598] "Means including an interactive interface for displaying operational status reports" refers to an interface that visually displays the generated reports and allows users to interact with them.

[0599] This invention relates to a system that enables the efficient collection and analysis of data within a factory, as well as the automation of report generation. The aim of this system is to allow users to easily perform advanced data analysis tasks and obtain results quickly and accurately.

[0600] System Overview

[0601] First, the user gives instructions to the robot via voice or text input. For example, an instruction such as "Generate this week's operational status report" is conceivable. The robot uses a voice recognition system to convert these instructions into text and analyzes its content. Specifically, it uses natural language processing technology to determine the meaning of the instructions.

[0602] Hardware and software

[0603] The hardware used in this process includes a voice input device for recognizing the user's voice, the robot body, and a central server. The software includes a voice recognition system (e.g., Dialogflow or other natural language processing (NLP) techniques), Python libraries for data collection and preprocessing (e.g., Pandas, NumPy), and a model as a generative AI (e.g., GPT-4).

[0604] Data collection and preprocessing

[0605] The server collects necessary data from various sensors and databases. PLCs (Programmable Logic Controllers) and IoT devices are used for data collection. The collected data is cleaned and normalized using Python's Pandas and NumPy libraries.

[0606] Report generation

[0607] Generative AI (e.g., GPT-4) uses pre-processed data to perform a specified task and generate results. An operational status report is automatically created from the generated data. This report includes uptime, downtime, and other key metrics.

[0608] Presentation of results

[0609] The generated report is sent from the server to the robot and displayed to the user through the robot's interface. An interactive GUI is used for display, and data visualization using tools such as Tableau or D3.js is envisioned.

[0610] Specific example

[0611] A concrete example is shown below. When a user gives a voice command to the robot, "Generate this week's operational status report," the robot analyzes this command and sends a request to a central server. The server collects the necessary data from various sensors and databases and preprocesses it using Python's Pandas and NumPy. The preprocessed data is then analyzed by GPT-4, and an operational status report is generated. The generated report is sent to the robot and displayed to the user in an interactive format.

[0612] Example of a prompt

[0613] User: "Generate this week's operational status report."

[0614] Robot: "Analyzing the command..."

[0615] Robot: "Collecting data..."

[0616] Robot: "We are pre-processing the data..."

[0617] Robot: "Generating report..."

[0618] Robot: "Here is this week's operational report. Operating hours and downtime are as follows..."

[0619] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0620] Step 1:

[0621] Receiving and analyzing user instructions

[0622] The user instructs the robot via voice or text input, "Generate this week's operational status report." The robot uses a speech recognition system to convert the speech into text and natural language processing technology to analyze the meaning of the instruction. Specifically, a voice input device captures the user's voice, and speech recognition software converts it into text data. Next, a natural language processing engine analyzes the instruction and recognizes the specific task (in this case, generating an operational status report).

[0623] Input: User's voice instructions

[0624] Output: Analyzed text instructions

[0625] Step 2:

[0626] Transmission of instructions to the server

[0627] The terminal sends the analysis results to the server. This communication is secure using the HTTPS protocol. Specifically, the robot sends the analyzed text data to the server via the network communication module. The server receives this instruction and proceeds to the next data collection process.

[0628] Input: Parsed text instructions

[0629] Output: Instruction data to the server

[0630] Step 3:

[0631] Data collection

[0632] The server collects the necessary data from sensors and databases within the factory. PLCs and IoT devices are used to collect data from sensors. The server sends data requests back to each device and receives operational status data in return.

[0633] Input: Instruction data to the server

[0634] Output: Collected raw data

[0635] Step 4:

[0636] Data preprocessing

[0637] The server preprocesses the collected data. Preprocessing includes data cleaning (imputing missing values ​​and removing outliers), normalization, and feature selection. Specifically, it uses Python's Pandas and NumPy to generate dataframes and perform cleanup and standardization operations.

[0638] Input: Collected raw data

[0639] Output: Preprocessed data

[0640] Step 5:

[0641] Report generation

[0642] The server uses pre-processed data to execute specified tasks using a generative AI (e.g., GPT-4) and generate reports. The generative AI analyzes the input data and automatically creates operational status reports. Specifically, the generative AI model takes in pre-processed data and performs statistics on uptime and downtime analysis.

[0643] Input: Preprocessed data

[0644] Output: Generated operational status report

[0645] Step 6:

[0646] Presentation of results

[0647] The server sends the generated report to the robot, which then presents it to the user. An interactive GUI is used for the display, allowing the user to visually check the operational status. Specifically, visualization tools such as Tableau and D3.js are used.

[0648] Input: Generated operational status report

[0649] Output: An interactive report displayed to the user.

[0650] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0651] The present invention will now describe embodiments for carrying out the present invention. The present invention is a system that combines user emotion recognition technology, enabling users to easily perform advanced data analysis tasks and obtain results quickly and accurately.

[0652] System Overview

[0653] When a user enters a specific instruction into the device, the instruction is analyzed and sent to the server. The server collects the necessary data based on the instruction and performs preprocessing. A generative AI uses the preprocessed data to execute the specified task and presents the results to the user through the device. In addition, an emotion engine recognizes the user's emotions and reflects them in the generated results and how they are presented.

[0654] Program processing

[0655] 1. User input

[0656] The user inputs a command into the terminal, such as "Generate this week's sales report." This input is done via voice recognition or keyboard input.

[0657] 2. Instruction Analysis

[0658] The device analyzes the user's instructions and extracts task details from the text. Natural language processing technology is used for the analysis.

[0659] The terminal sends the analysis results (task details) to the server.

[0660] 3. Transmission of commands and activation of generative AI

[0661] The terminal sends the analysis results to the server. Network protocols are used for communication.

[0662] The server analyzes the received instructions and determines what type of report needs to be generated.

[0663] The server starts the generative AI and loads the necessary settings.

[0664] 4. Activation of the Emotional Engine

[0665] The device collects the user's voice and facial expressions through its camera and microphone and transmits them to the emotion engine.

[0666] The emotion engine analyzes the user's emotions and sends the results to the server.

[0667] 5. Data Collection

[0668] The server automatically collects the necessary data, including data retrieval from customer databases, sales databases, and marketing databases.

[0669] The server communicates with each database via APIs and data connections to collect necessary information.

[0670] 6. Data preprocessing

[0671] The server preprocesses the collected data, performing tasks such as imputing missing values, removing outliers, and standardizing data formats.

[0672] The server prepares the pre-processed data as a dataset for analysis.

[0673] 7. Task execution and result generation

[0674] The generative AI begins analysis using pre-processed data. For example, it calculates metrics such as sales revenue, conversion rate, and customer feedback.

[0675] Generative AI creates reports from generated data, and text generation technology generates explanatory text that includes insights.

[0676] 8. Submission and presentation of results

[0677] The server sends the generated report to the terminal. This communication uses the HTTPS protocol to ensure data security.

[0678] The device analyzes the received reports and adjusts the presentation method based on the user's perceived emotions. The format and emphasis of the presentation will differ depending on whether the user is relaxed or stressed.

[0679] Specific example

[0680] Sales report generation and sentiment feedback

[0681] 1. The user enters "Generate this week's sales report" into the terminal.

[0682] 2. The terminal analyzes the instructions and sends the information to the server.

[0683] 3. The server activates the generative AI and collects and preprocesses the necessary data.

[0684] 4. The emotion engine analyzes the user's emotions and sends the results to the server. For example, it might determine that the user is feeling stressed.

[0685] 5. Generative AI analyzes sales data and generates reports. These reports include sales trends and feedback from key customers.

[0686] 6. The server sends the generated effet and sentiment information to the terminal, which then displays it to the user.

[0687] Thus, this system automatically performs tasks from data collection and preprocessing to analysis, result generation, and presentation based on user instructions. Furthermore, it can improve the user experience by recognizing user emotions and adjusting the way results are presented accordingly.

[0688] The following describes the processing flow.

[0689] Step 1:

[0690] The user enters the instruction "Generate this week's sales report" into the terminal.

[0691] The device receives these instructions via voice recognition or text input.

[0692] Step 2:

[0693] The terminal analyzes the user's instructions. Natural language processing technology is used to extract task details (sales report generation) from the text.

[0694] The terminal sends the analysis results (task details) to the server.

[0695] Step 3:

[0696] The device collects the user's voice and facial expression data through its camera and microphone.

[0697] The device sends the collected data to the emotion engine.

[0698] Step 4:

[0699] The emotion engine analyzes voice and facial expression data to determine the user's emotional state (e.g., relaxed, stressed).

[0700] The emotion engine sends the analysis results (emotional information) to the server.

[0701] Step 5:

[0702] The server analyzes the instructions and emotional information received from the terminal.

[0703] The server starts the generative AI and loads the necessary settings.

[0704] Step 6:

[0705] The server automatically collects the necessary data based on instructions. This includes retrieving data from customer databases, sales databases, marketing databases, and so on.

[0706] The server communicates with each database via APIs and data connections to collect necessary information.

[0707] Step 7:

[0708] The server performs preprocessing on the collected data. Specifically, this includes data cleaning (imputing missing values ​​and removing outliers), normalization, and selecting necessary attributes.

[0709] The server prepares the pre-processed data into a format that can be used by generative AI for analysis.

[0710] Step 8:

[0711] A generative AI performs tasks using pre-processed data. Here, it calculates metrics such as sales revenue, conversion rate, and customer feedback.

[0712] Generative AI prepares data to generate explanatory text and graphs based on the analysis results.

[0713] Step 9:

[0714] The generative AI uses text generation technology to create a report based on the analysis results. The report includes insights, sales trend graphs, and statistical information.

[0715] The generative AI returns the final generated report to the server.

[0716] Step 10:

[0717] The server sends the generated report and sentiment information to the terminal. The communication uses the HTTPS protocol to ensure data security.

[0718] The server records transmission logs for later use in troubleshooting and feedback.

[0719] Step 11:

[0720] Based on the reports received by the device, the presentation method is adjusted according to the user's perceived emotions. For example, if the user is feeling stressed, the results are displayed in a concise and easy-to-understand format.

[0721] The terminal displays the generated report on the screen, allowing the user to review it.

[0722] Following the steps described above, the present invention automatically performs data collection, preprocessing, analysis, result generation, and presentation based on user instructions. Furthermore, it improves the user experience by recognizing the user's emotions and adjusting the result presentation method accordingly.

[0723] (Example 2)

[0724] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0725] Conventional data analysis systems often required numerous manual operations from the user's input of specific instructions to obtaining results, resulting in lengthy result generation times. Furthermore, they struggled to respond flexibly to the user's emotional state, sometimes leading to a diminished user experience. To address these challenges, it is necessary to provide a system that allows users to easily, quickly, and accurately obtain data analysis results while also presenting appropriate information based on their emotional state.

[0726] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0727] In this invention, the server includes means for receiving instructions from a user, means for interpreting and analyzing specific tasks based on the instructions, means for collecting and pre-processing necessary data, means for executing a specified task using generative artificial intelligence and generating results, means for recognizing the user's emotions, means for reflecting the user's emotions in the generated results and how they are presented, and means for presenting the generated results to the user. This makes it possible to automate everything from receiving instructions from the user to data collection, analysis, result generation, and flexible information presentation based on emotions, thereby improving the user experience.

[0728] A "user" is a person or entity that operates a system and provides instructions or inputs.

[0729] "Instructions" refer to the specific operations or commands that a user performs on the system.

[0730] A "task" is a specific task or process that a system should perform based on instructions.

[0731] "Analysis" refers to data processing and analysis that interprets user instructions and translates them into specific tasks.

[0732] "Data" is a collection of information that a system needs to perform a task.

[0733] "Data collection" is the process of obtaining necessary data from various data sources.

[0734] "Preprocessing" refers to the process of preparing collected data for analysis or task execution.

[0735] "Generative artificial intelligence" refers to artificial intelligence technology used to perform specified tasks and generate results.

[0736] "Results" refer to the output or outcome obtained when a generative artificial intelligence performs a task.

[0737] "Emotions" are internal reactions that indicate a user's psychological state or feedback.

[0738] "Recognition" is the process of analyzing the user's voice and facial expressions to determine their emotions.

[0739] "Presentation" refers to the act of showing the generated results to the user visually or audibly.

[0740] The system of this invention performs advanced data analysis tasks based on user instructions and provides the results quickly and accurately. Furthermore, it can recognize the user's emotions and adjust the way the results are presented accordingly. This system is comprised of the following key hardware and software components.

[0741] User input

[0742] When users input instructions into the terminal, they use voice recognition technology (e.g., a voice recognition API) or keyboard input. The input is a prompt message such as "Generate this week's sales report."

[0743] Instruction parsing

[0744] The device uses speech recognition technology (speech recognition API) to convert voice input into text in order to analyze user instructions. Next, it uses natural language processing technology (natural language processing API) to analyze the input text and extract the task details. The analysis results are stored in a database and sent from the device to the server.

[0745] Transmission of commands and activation of generative AI

[0746] The HTTPS protocol is used when the terminal sends the analyzed instructions to the server. Based on the received analysis results, the server determines what type of report to generate. The server then activates the generative artificial intelligence (generative artificial intelligence API) and loads the necessary settings.

[0747] Emotional engine activation

[0748] The device collects the user's voice and facial expression data through its camera and microphone and sends it to an emotion engine (emotion recognition API). The emotion engine analyzes the user's emotions and sends the results to a server.

[0749] Data collection

[0750] When the server automatically collects the necessary data, it retrieves data from external data sources such as customer databases, sales databases, and marketing databases through APIs and data connections.

[0751] Data preprocessing

[0752] The server preprocesses the collected data. This includes imputing missing values, removing outliers, and standardizing data formats, preparing the dataset for analysis.

[0753] Task execution and result generation

[0754] Generative artificial intelligence begins analysis using pre-processed data. For example, it calculates metrics such as sales revenue, conversion rate, and customer feedback. A report is created from the generated data, and explanatory text containing insights is generated using text generation technology.

[0755] Sending and presenting results

[0756] The server sends the generated report to the terminal using the HTTPS protocol. The terminal analyzes the received report and adjusts the presentation method based on the user's emotion recognition results. For example, if the user is stressed, important information is highlighted and displayed clearly. Conversely, if the user is relaxed, detailed information is also presented.

[0757] Specific example

[0758] Sales report generation and sentiment feedback

[0759] The user enters a command into the terminal, such as "Generate this week's sales report." The terminal analyzes this command and sends it to the server. The server collects the necessary data and preprocesses it. The emotion engine analyzes the user's emotions and sends the results to the server. For example, if it determines that the user is feeling stressed, it appropriately highlights important information in the report. Generative artificial intelligence analyzes the sales data and generates the report. The server then sends the generated report to the terminal, which displays it to the user.

[0760] This allows users to obtain data analysis results quickly and accurately, and enables the presentation of information tailored to the user's emotional state.

[0761] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0762] Step 1:

[0763] The user inputs the command "Generate this week's sales report" into the device. Speech recognition technology (such as a speech recognition API) is used to convert the voice input into text. The input data is output in text format as "Generate this week's sales report".

[0764] Step 2:

[0765] The terminal analyzes the input text using natural language processing technology (such as a natural language processing API). Specifically, it extracts the keywords "generate," "sales report," and "this week," and recognizes them as tasks. The analysis results are output as JSON data containing the task details, such as "Generate Sales Report."

[0766] Step 3:

[0767] The terminal sends the analysis results to the server using the HTTPS protocol. The transmitted data is in JSON format and contains task details. The server checks the received instructions and determines which type of task to execute. Based on this determination, the instruction "Generate Sales Report" is issued.

[0768] Step 4:

[0769] The device collects user voice and facial expression data through its camera and microphone and sends it to an emotion engine (emotion recognition API). For example, it analyzes stress levels from the user's face. The emotion engine analyzes the emotional state and outputs the result to the server as data such as "the user is feeling stressed."

[0770] Step 5:

[0771] The server collects the necessary data based on the "Sales Report Generation" task. For example, it retrieves data from external data sources such as customer databases, sales databases, and marketing databases via APIs. The collected data includes customer information and sales data. This data is aggregated on the server in an integrated format.

[0772] Step 6:

[0773] The server preprocesses the collected data. Specifically, it imputes missing values, removes outliers, and standardizes the data format. The preprocessed dataset is output in a format suitable for analysis.

[0774] Step 7:

[0775] Generative artificial intelligence performs tasks using pre-processed data. Specifically, it calculates metrics such as sales trends, conversion rates, and customer feedback. Based on the analysis results, the generative AI creates a report and generates explanatory text containing insights using text generation technology. The generated report is output in text and graph formats.

[0776] Step 8:

[0777] The server sends the generated report to the terminal using the HTTPS protocol. The transmitted data includes not only the report information but also the user's emotional state obtained from the emotion engine.

[0778] Step 9:

[0779] The device analyzes the received report and adjusts the presentation method based on the user's emotional state. For example, if the user is feeling stressed, important parts of the report will be highlighted. The presentation method ensures that the information is presented in a format that is easiest for the user to understand.

[0780] Example prompt statements

[0781] "Generate this week's sales report."

[0782] "A compilation of the latest customer feedback"

[0783] "Tell me your sales forecast for next week."

[0784] The above outlines the detailed processing steps of the system. This process allows users to obtain data analysis results quickly and accurately, and further facilitates the presentation of information tailored to the user's emotional state.

[0785] (Application Example 2)

[0786] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0787] Traditional in-store customer service struggled to recognize customer emotions in real time and suggest appropriate products and services based on those emotions. Furthermore, the lack of systems to provide feedback and recommendations tailored to customer emotional states prevented improvements in the customer experience. This resulted in problems such as decreased customer satisfaction and reduced purchasing intent.

[0788] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving instructions from the user, means for interpreting and analyzing specific tasks based on the instructions, and means for recognizing the user's emotions and adjusting the method of presenting the results generated based on those emotions. This makes it possible to recognize customer emotions in real time in a physical store and propose appropriate products and services based on those emotions.

[0789] "Means for receiving instructions from the user" refers to an interface for receiving voice or text instructions entered by the user through a device.

[0790] "Means for interpreting and analyzing specific tasks based on instructions" refers to a process that uses natural language processing technology to analyze the user's instructions and determine which task to execute.

[0791] "Means for collecting and pre-processing necessary data" refers to data processing functions that collect necessary information from diverse data sources and perform pre-processing such as data cleansing and formatting standardization.

[0792] "A means of executing a specified task and generating results using generative AI" refers to an algorithm that uses a generative AI model to perform a specified task from pre-processed data and generate results.

[0793] "Means for recognizing user emotions and adjusting the presentation method of results generated based on those emotions" refers to technology that analyzes user emotions from facial expressions and voice, and presents the results in the most optimal format based on the analysis results.

[0794] "Means of presenting the generated results to the user" refers to UI / UX functions for displaying the generated results on the user's device.

[0795] A description of embodiments for carrying out the present invention will be provided.

[0796] System Configuration

[0797] This system can recognize user emotions and provide personalized product and service recommendations based on those emotions. The system consists of the following components:

[0798] 1. Devices: These include user devices such as smart glasses and smartphones. They are used to collect user input and emotional data.

[0799] 2. Server: The cloud server performs instruction analysis, data collection, preprocessing, activation of generative AI, result generation, and result presentation.

[0800] 3. Emotion Recognition Engine: This engine analyzes data collected by the device through the camera and microphone to recognize the user's emotions. For example, Microsoft Azure's Emotion API is used for this engine.

[0801] 4. Generative AI: AI modules that perform tasks using pre-processed data and generate results. For example, OpenAI's GPT model is used.

[0802] Data collection and analysis

[0803] The device collects the user's voice and facial expression data in real time. This data is sent to an emotion recognition engine to recognize the user's emotions. The emotion-recognized data is sent to a server and analyzed along with the user's instructions.

[0804] The analysis uses natural language processing technology (e.g., Google Cloud Natural Language API) to convert the instructions into tasks. For example, if a user enters "I'm looking for new summer clothes," this instruction is parsed into the task "Recommend new summer items."

[0805] Data collection and preprocessing

[0806] The server collects the necessary data from the database based on the analyzed task. This data may include customer purchase history, inventory information, and promotional information. The collected data is preprocessed (data cleansing, formatting, etc.) and prepared in a format suitable for generative AI.

[0807] Task execution using generative AI

[0808] A generative AI analyzes pre-processed data to generate optimal product and service recommendations based on the user's emotions and preferences. The generated results are then sent back to the terminal via the server.

[0809] Presentation of results

[0810] The device displays results generated based on the user's emotions. It adjusts the display method and emphasis depending on whether the user is relaxed or stressed. For example, it highlights relaxing items for tense users and recommends stimulating items for excited users.

[0811] Specific example

[0812] For example, if a customer voice-inputs "I'm looking for new summer clothes" into smart glasses, the emotion recognition engine detects that the customer is relaxed. The system then generates a list of the latest summer items and displays them in a relaxed and easy-to-view format.

[0813] Example of a prompt

[0814] "Please generate a list of summer products that customers can easily access."

[0815] As described above, the present invention can provide a more personalized user experience by recognizing the user's emotions and making recommendations based on them. Furthermore, by combining generative AI with an emotion recognition engine, it becomes possible to respond appropriately to the diverse needs of users.

[0816] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0817] Step 1:

[0818] The user inputs instructions into the device. The user inputs instructions via voice or text using smart glasses or a smartphone. These instructions are sent to the device as input data. For example, the instruction "I want new summer clothes" is entered into the device.

[0819] Step 2:

[0820] The device collects the user's emotions. The device uses its camera and microphone to capture the user's facial expressions and tone of voice in real time and sends this to an emotion recognition engine. The emotion recognition engine (e.g., Microsoft Azure's Emotion API) analyzes the collected data and recognizes the user's emotions (e.g., relaxed, excited, stressed). This recognition result is returned to the device as emotion data.

[0821] Step 3:

[0822] The device analyzes the user's instructions and extracts specific tasks. The device uses natural language processing technology (e.g., Google Cloud Natural Language API) to analyze instructions entered via voice or text. The analyzed instructions are sent to the server as text data. For example, the instruction "I'm looking for new summer clothes" is converted into the task "Recommend new summer items."

[0823] Step 4:

[0824] The server collects and preprocesses the necessary data. The server accesses databases (e.g., customer purchase history, inventory information, promotional information) and collects the required data based on the task. The collected data undergoes preprocessing such as imputing missing values, removing outliers, and standardizing data formats. The preprocessed data is then sent to the generative AI.

[0825] Step 5:

[0826] The server uses generative AI to perform the specified task. The generative AI (e.g., OpenAI GPT model) generates a list of optimal products and services based on pre-processed data, taking into account the user's emotions and instructions. This generation process uses the prompt "Generate a list of summer products that are easily accessible to the customer." The generated results are returned to the server as recommendation data.

[0827] Step 6:

[0828] The server sends recommendation data to the device. The server sends the generated results to the device using the HTTPS protocol to ensure data security. The final presented data is then sent to the device.

[0829] Step 7:

[0830] The device presents results based on the user's emotions. Based on the user's emotional data, the device adjusts the presentation method and emphasis. For example, if the user is relaxed, the results are displayed in a calm tone; if they are stressed, the results are displayed more intuitively and concisely. The user then reviews the results and makes a selection of products or services.

[0831] Through these steps, personalized product and service recommendations that take user emotions into consideration are achieved.

[0832] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0833] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0834] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0835] [Third Embodiment]

[0836] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0837] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0838] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0839] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0840] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0841] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0842] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0843] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0844] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0845] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0846] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0847] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0848] The embodiments for carrying out the present invention are described below.

[0849] This system aims to enable users to easily perform advanced data analysis tasks and obtain results quickly and accurately. This allows for the generation of complex reports and predictive analytics without the manual operations and advanced expertise previously required.

[0850] System Overview

[0851] When a user enters a specific instruction into the terminal, the instruction is analyzed and sent to the server. The server collects the necessary data based on the instruction and performs preprocessing. A generative AI then uses the preprocessed data to execute the specified task and presents the results to the user through the terminal.

[0852] Program processing

[0853] 1. User input

[0854] The user enters the instruction "Generate this week's sales report" into the terminal. This input is done via voice recognition or keyboard input.

[0855] 2. Instruction Analysis

[0856] The device receives user instructions and analyzes their content. This analysis uses natural language processing technology to determine the meaning of the instructions.

[0857] 3. Transmission of commands and activation of generative AI

[0858] The terminal sends the analysis results to the server. This is done using network communication.

[0859] The server activates the generative AI based on the received instructions.

[0860] 4. Data Collection

[0861] The server automatically collects the necessary data. For example, it retrieves the required information from customer databases, sales databases, and marketing databases.

[0862] 5. Data preprocessing

[0863] The server preprocesses the collected data. Specifically, this involves data cleaning, normalization, and feature selection. This process prepares the data for analysis.

[0864] 6. Task execution and result generation

[0865] Generative AI uses pre-processed data to perform specified tasks. For example, it can analyze sales trends or perform statistical analysis of customer feedback.

[0866] Generative AI creates reports from generated data and uses text generation technology to produce explanatory text that includes insights.

[0867] 7. Submission and presentation of results

[0868] The server sends the generated report to the terminal. This communication is conducted through a highly secure protocol.

[0869] The terminal analyzes the received report and displays it to the user in an appropriate format. This uses an interactive GUI that includes graphs and text.

[0870] Specific example

[0871] Sales report generation

[0872] 1. The user enters "Generate this week's sales report" into the terminal.

[0873] 2. The terminal analyzes this instruction and sends it to the server.

[0874] 3. The server activates the generative AI and collects and preprocesses the necessary data.

[0875] 4. Generative AI analyzes sales data and generates reports. These reports include monthly sales trends and feedback from key customers.

[0876] 5. The server sends the generated report to the terminal, and the terminal displays it to the user.

[0877] Thus, this system is designed to allow users to easily perform advanced data analysis by automatically collecting, preprocessing, analyzing, generating, and presenting data based on user instructions.

[0878] The following describes the processing flow.

[0879] Step 1:

[0880] The user enters the instruction "Generate this week's sales report" into the terminal.

[0881] The device receives these instructions through voice recognition or text analysis.

[0882] Step 2:

[0883] The device analyzes the user's instructions and extracts the task details from the text.

[0884] The terminal sends the analysis results (task details) to the server.

[0885] Step 3:

[0886] The server analyzes the instructions received from the terminal and determines what type of report needs to be generated.

[0887] The server starts the generative AI and loads the necessary settings.

[0888] Step 4:

[0889] The server collects the necessary data based on instructions. For example, it retrieves data from customer databases, sales databases, marketing databases, etc.

[0890] The server communicates with each database via APIs and data connections to collect necessary information.

[0891] Step 5:

[0892] The server preprocesses the collected data. Specifically, this involves imputing missing values, removing outliers, and standardizing data formats.

[0893] The server prepares the pre-processed data as a dataset for analysis.

[0894] Step 6:

[0895] The generative AI begins its analysis using pre-processed data. Here, it calculates metrics such as sales revenue, conversion rate, and customer feedback.

[0896] The generative AI uses the results to prepare data for generating explanatory text and graphs.

[0897] Step 7:

[0898] The generative AI generates a report based on the analysis results. The report includes insights created using text generation technology, sales trend graphs, and statistical information.

[0899] The generative AI returns the final generated report to the server.

[0900] Step 8:

[0901] The server sends the generated report to the terminal. Communication is usually done using the HTTPS protocol to ensure data security.

[0902] The server records transmission logs for later use in troubleshooting and feedback.

[0903] Step 9:

[0904] The terminal analyzes the received reports and displays them in a user-friendly format. This includes GUI rendering using HTML and CSS.

[0905] The terminal will provide reports in an interactive format, allowing users to quickly obtain the information they need.

[0906] The above outlines the specific processing steps of the system that generates sales reports based on user instructions and presents the results.

[0907] (Example 1)

[0908] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0909] Traditionally, advanced data analysis and complex report generation required users to possess advanced expertise and manual operation. This resulted in significant time and effort, making it inefficient. This invention solves this problem by providing a system that allows users to perform data analysis and report generation easily and quickly without requiring specialized knowledge or manual operation.

[0910] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0911] In this invention, the server includes means for receiving instructions from a user, means for interpreting tasks based on the instructions and analyzing them using natural language processing technology, means for sending the analyzed instructions to the server and activating a generative AI, means for collecting necessary data from a database, cleaning and normalizing the data, and performing preprocessing to select features, means for executing a specified task based on the preprocessed data using the generative AI and generating results, and means for presenting the generated results to the user in an interactive format on a terminal. This enables users to perform advanced data analysis and report generation in a short time without specialized knowledge.

[0912] 1. "Means for receiving instructions from the user" refers to an interface that allows the user to input instructions to the system using methods such as voice recognition or keyboard input.

[0913] 2. "Natural language processing technology" refers to technologies that enable computers to understand and analyze human language, specifically technologies that perform text analysis and semantic analysis.

[0914] 3. "Means for interpreting and analyzing tasks" refers to the process of analyzing instructions received from the user, understanding their content, and determining the appropriate action to take.

[0915] 4. "Generative AI" refers to artificial intelligence models that have been trained to perform specified tasks, and are models that automatically perform data analysis and report generation.

[0916] 5. "Means of data collection" refers to mechanisms for obtaining necessary data from various databases and external data sources.

[0917] 6. "Data preprocessing means" refers to processes such as cleaning, normalization, and feature selection that are performed to prepare collected data into a format suitable for analysis.

[0918] 7. "Means for executing tasks and generating results" refers to the process by which a generative AI performs specified data analysis and report generation tasks based on pre-processed data and creates the results.

[0919] 8. "Means for presenting the generated results to the user on the terminal" refers to an interface for visually displaying the generated analysis results and reports to the user.

[0920] This invention provides a system that allows users to easily perform advanced data analysis and report generation. Specific embodiments for carrying out this invention are described below.

[0921] System Overview

[0922] This system consists of three main components: a user, a terminal, and a server. The user inputs instructions into the terminal via voice recognition or keyboard input. The terminal analyzes the instructions using natural language processing technology and sends them to the server. The server activates a generative AI to collect and preprocess the necessary data, then generates analysis results and reports, which are presented to the user.

[0923] Hardware and software to use

[0924] 1. Terminal

[0925] Hardware: PCs, tablets, smartphones, etc.

[0926] Software: Google Speech-to-Text API (speech recognition), TensorFlow, NLTK (natural language processing)

[0927] 2. Server

[0928] Hardware: Cloud servers, on-premises servers

[0929] Software: MySQL / PostgreSQL (databases), Pandas, Scikit-learn (data preprocessing), GPT-4 (generative AI model)

[0930] 3. Communications

[0931] Protocol: HTTPS (Secure communication)

[0932] Specific examples of actions

[0933] Sales report generation

[0934] 1. The user enters "Generate this week's sales report" into the terminal.

[0935] Example prompt: "Generate this week's sales report."

[0936] 2. The device receives instructions using speech recognition technology (Google Speech-to-Text API) or keyboard input.

[0937] In the case of voice input, the speech is converted to text.

[0938] 3. The device analyzes the instructions using TensorFlow or NLTK.

[0939] Identify tasks based on the keyword "sales report".

[0940] 4. The terminal sends the analyzed commands to the server using the HTTPS protocol.

[0941] 5. The server starts the generative AI model (GPT-4) based on the instructions.

[0942] 6. The server collects sales data and customer data from MySQL or PostgreSQL.

[0943] Execute the SQL query and extract the necessary data.

[0944] 7. The server performs data preprocessing using Pandas or Scikit-learn.

[0945] Cleaning (imputing missing values ​​and deleting invalid data)

[0946] Data normalization (converting values ​​to a scale of 0 to 1)

[0947] Feature selection (select important variables)

[0948] 8. The generative AI model performs the specified task based on the pre-processed data and generates the results.

[0949] We analyze sales trends and customer feedback statistically, and then use text generation technology to generate explanatory text that includes insights.

[0950] 9. The server sends the generated report to the terminal using the HTTPS protocol.

[0951] 10. The terminal presents the received report to the user in an interactive format.

[0952] This program displays graphs using D3.js or Chart.js, along with text styled using HTML and CSS.

[0953] Effects of implementation

[0954] This system allows users to easily perform data analysis and generate reports without requiring advanced expertise. This improves the quality of data-driven decision-making and significantly enhances operational efficiency.

[0955] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0956] Step 1:

[0957] The user enters "Generate this week's sales report" into the terminal. Input is done via voice recognition or keyboard.

[0958] Input: User instruction ("Generate this week's sales report")

[0959] Output: Instruction text displayed on the terminal

[0960] Step 2:

[0961] The device receives user instructions using speech recognition technology (Google Speech-to-Text API) or keyboard input. The received instructions are then analyzed using natural language processing technology (TensorFlow, NLTK, etc.).

[0962] Input: Speech recognition result or keyboard input text

[0963] Output: Analyzed instructions (report generation task)

[0964] Step 3:

[0965] The terminal transfers the analysis results to the server. This communication is securely performed using the HTTPS protocol.

[0966] Input: Analyzed instructions

[0967] Output: Parsing instructions to the server

[0968] Step 4:

[0969] The server launches a generative AI model (GPT-4) based on the analysis results. The generative AI then prepares to execute the specified task.

[0970] Input: Analyzed instructions

[0971] Output: Preparation for starting a generative AI

[0972] Step 5:

[0973] The server collects the necessary data. Specifically, it executes SQL queries to retrieve sales data and customer data from databases (such as MySQL and PostgreSQL).

[0974] Input: Data to be collected as instructed

[0975] Output: Acquired dataset (sales data, customer data, etc.)

[0976] Step 6:

[0977] The server preprocesses the collected data. It uses Pandas and Scikit-learn to perform tasks such as data cleaning, normalization, and feature selection.

[0978] Input: Acquired dataset

[0979] Output: Preprocessed data

[0980] Step 7:

[0981] Generative AI models perform specified tasks based on pre-processed data and generate results. For example, they can analyze sales trends or perform statistical analysis of customer feedback.

[0982] Input: Preprocessed data

[0983] Output: Analysis results and report content

[0984] Step 8:

[0985] The server sends the generated results to the terminal. This communication also uses the HTTPS protocol.

[0986] Input: Generated results (report content)

[0987] Output: Sending results to the terminal

[0988] Step 9:

[0989] The terminal analyzes the received report and displays it to the user in an interactive format. It visualizes the data using D3.js and Chart.js, and styles it with HTML and CSS.

[0990] Input: Report results from the server

[0991] Output: Interactive report display on the user's device.

[0992] (Application Example 1)

[0993] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0994] Current factory operations generate a large amount of data, but analyzing and reporting on it requires advanced expertise and manual operation, making efficient data utilization difficult. Furthermore, there is a need for real-time monitoring of operating conditions and rapid report generation, but there is a lack of effective means to achieve this.

[0995] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0996] In this invention, the server includes means for receiving instructions from a user, means for interpreting and analyzing specific tasks based on the instructions, means for collecting and pre-processing necessary data, means for executing specified tasks using generative AI and generating results, means for presenting the generated results to the user, means for creating an operational status report from the generated data, and means including an interactive interface for displaying the operational status report. This enables the user to easily perform advanced data analysis, generate status reports in real time, and respond quickly.

[0997] "Means of receiving instructions from the user" refers to an interface that receives instructions from the user via voice input or text input, and a system that interprets that input data.

[0998] "A means of interpreting and analyzing a specific task based on instructions" refers to a system that uses natural language processing technology to identify a task from user input and analyze its content.

[0999] "Means for collecting and pre-processing necessary data" refers to processes and systems that collect appropriate data from sensors, databases, etc., and then clean and normalize that data.

[1000] "A means of executing a specified task and generating results using generative AI" refers to a system that utilizes generative artificial intelligence to analyze pre-processed data and generate the necessary results and reports.

[1001] "Means of presenting generated results to the user" refers to a system that displays generated data and reports to the user through an interface.

[1002] "Means for creating operational status reports from generated data" refers to a system that automatically generates reports on operational status based on collected and pre-processed data.

[1003] "Means including an interactive interface for displaying operational status reports" refers to an interface that visually displays the generated reports and allows users to interact with them.

[1004] This invention relates to a system that enables the efficient collection and analysis of data within a factory, as well as the automation of report generation. The aim of this system is to allow users to easily perform advanced data analysis tasks and obtain results quickly and accurately.

[1005] System Overview

[1006] First, the user gives instructions to the robot via voice or text input. For example, an instruction such as "Generate this week's operational status report" is conceivable. The robot uses a voice recognition system to convert these instructions into text and analyzes its content. Specifically, it uses natural language processing technology to determine the meaning of the instructions.

[1007] Hardware and software

[1008] The hardware used in this process includes a voice input device for recognizing the user's voice, the robot body, and a central server. The software includes a voice recognition system (e.g., Dialogflow or other natural language processing (NLP) techniques), Python libraries for data collection and preprocessing (e.g., Pandas, NumPy), and a model as a generative AI (e.g., GPT-4).

[1009] Data collection and preprocessing

[1010] The server collects necessary data from various sensors and databases. PLCs (Programmable Logic Controllers) and IoT devices are used for data collection. The collected data is cleaned and normalized using Python's Pandas and NumPy libraries.

[1011] Report generation

[1012] Generative AI (e.g., GPT-4) uses pre-processed data to perform a specified task and generate results. An operational status report is automatically created from the generated data. This report includes uptime, downtime, and other key metrics.

[1013] Presentation of results

[1014] The generated report is sent from the server to the robot and displayed to the user through the robot's interface. An interactive GUI is used for display, and data visualization using tools such as Tableau or D3.js is envisioned.

[1015] Specific example

[1016] A concrete example is shown below. When a user gives a voice command to the robot, "Generate this week's operational status report," the robot analyzes this command and sends a request to a central server. The server collects the necessary data from various sensors and databases and preprocesses it using Python's Pandas and NumPy. The preprocessed data is then analyzed by GPT-4, and an operational status report is generated. The generated report is sent to the robot and displayed to the user in an interactive format.

[1017] Example of a prompt

[1018] User: "Generate this week's operational status report."

[1019] Robot: "Analyzing the command..."

[1020] Robot: "Collecting data..."

[1021] Robot: "We are pre-processing the data..."

[1022] Robot: "Generating report..."

[1023] Robot: "Here is this week's operational report. Operating hours and downtime are as follows..."

[1024] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1025] Step 1:

[1026] Receiving and analyzing user instructions

[1027] The user instructs the robot via voice or text input, "Generate this week's operational status report." The robot uses a speech recognition system to convert the speech into text and natural language processing technology to analyze the meaning of the instruction. Specifically, a voice input device captures the user's voice, and speech recognition software converts it into text data. Next, a natural language processing engine analyzes the instruction and recognizes the specific task (in this case, generating an operational status report).

[1028] Input: User's voice instructions

[1029] Output: Analyzed text instructions

[1030] Step 2:

[1031] Transmission of instructions to the server

[1032] The terminal sends the analysis results to the server. This communication is secure using the HTTPS protocol. Specifically, the robot sends the analyzed text data to the server via the network communication module. The server receives this instruction and proceeds to the next data collection process.

[1033] Input: Parsed text instructions

[1034] Output: Instruction data to the server

[1035] Step 3:

[1036] Data collection

[1037] The server collects the necessary data from sensors and databases within the factory. PLCs and IoT devices are used to collect data from sensors. The server sends data requests back to each device and receives operational status data in return.

[1038] Input: Instruction data to the server

[1039] Output: Collected raw data

[1040] Step 4:

[1041] Data preprocessing

[1042] The server preprocesses the collected data. Preprocessing includes data cleaning (imputing missing values ​​and removing outliers), normalization, and feature selection. Specifically, it uses Python's Pandas and NumPy to generate dataframes and perform cleanup and standardization operations.

[1043] Input: Collected raw data

[1044] Output: Preprocessed data

[1045] Step 5:

[1046] Report generation

[1047] The server uses pre-processed data to execute specified tasks using a generative AI (e.g., GPT-4) and generate reports. The generative AI analyzes the input data and automatically creates operational status reports. Specifically, the generative AI model takes in pre-processed data and performs statistics on uptime and downtime analysis.

[1048] Input: Preprocessed data

[1049] Output: Generated operational status report

[1050] Step 6:

[1051] Presentation of results

[1052] The server sends the generated report to the robot, which then presents it to the user. An interactive GUI is used for the display, allowing the user to visually check the operational status. Specifically, visualization tools such as Tableau and D3.js are used.

[1053] Input: Generated operational status report

[1054] Output: An interactive report displayed to the user.

[1055] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1056] The present invention will now describe embodiments for carrying out the present invention. The present invention is a system that combines user emotion recognition technology, enabling users to easily perform advanced data analysis tasks and obtain results quickly and accurately.

[1057] System Overview

[1058] When a user enters a specific instruction into the device, the instruction is analyzed and sent to the server. The server collects the necessary data based on the instruction and performs preprocessing. A generative AI uses the preprocessed data to execute the specified task and presents the results to the user through the device. In addition, an emotion engine recognizes the user's emotions and reflects them in the generated results and how they are presented.

[1059] Program processing

[1060] 1. User input

[1061] The user inputs a command into the terminal, such as "Generate this week's sales report." This input is done via voice recognition or keyboard input.

[1062] 2. Instruction Analysis

[1063] The device analyzes the user's instructions and extracts task details from the text. Natural language processing technology is used for the analysis.

[1064] The terminal sends the analysis results (task details) to the server.

[1065] 3. Transmission of commands and activation of generative AI

[1066] The terminal sends the analysis results to the server. Network protocols are used for communication.

[1067] The server analyzes the received instructions and determines what type of report needs to be generated.

[1068] The server starts the generative AI and loads the necessary settings.

[1069] 4. Activation of the Emotional Engine

[1070] The device collects the user's voice and facial expressions through its camera and microphone and transmits them to the emotion engine.

[1071] The emotion engine analyzes the user's emotions and sends the results to the server.

[1072] 5. Data Collection

[1073] The server automatically collects the necessary data, including data retrieval from customer databases, sales databases, and marketing databases.

[1074] The server communicates with each database via APIs and data connections to collect necessary information.

[1075] 6. Data preprocessing

[1076] The server preprocesses the collected data, performing tasks such as imputing missing values, removing outliers, and standardizing data formats.

[1077] The server prepares the pre-processed data as a dataset for analysis.

[1078] 7. Task execution and result generation

[1079] The generative AI begins analysis using pre-processed data. For example, it calculates metrics such as sales revenue, conversion rate, and customer feedback.

[1080] Generative AI creates reports from generated data, and text generation technology generates explanatory text that includes insights.

[1081] 8. Submission and presentation of results

[1082] The server sends the generated report to the terminal. This communication uses the HTTPS protocol to ensure data security.

[1083] The device analyzes the received reports and adjusts the presentation method based on the user's perceived emotions. The format and emphasis of the presentation will differ depending on whether the user is relaxed or stressed.

[1084] Specific example

[1085] Sales report generation and sentiment feedback

[1086] 1. The user enters "Generate this week's sales report" into the terminal.

[1087] 2. The terminal analyzes the instructions and sends the information to the server.

[1088] 3. The server activates the generative AI and collects and preprocesses the necessary data.

[1089] 4. The emotion engine analyzes the user's emotions and sends the results to the server. For example, it might determine that the user is feeling stressed.

[1090] 5. Generative AI analyzes sales data and generates reports. These reports include sales trends and feedback from key customers.

[1091] 6. The server sends the generated effet and sentiment information to the terminal, which then displays it to the user.

[1092] Thus, this system automatically performs tasks from data collection and preprocessing to analysis, result generation, and presentation based on user instructions. Furthermore, it can improve the user experience by recognizing user emotions and adjusting the way results are presented accordingly.

[1093] The following describes the processing flow.

[1094] Step 1:

[1095] The user enters the instruction "Generate this week's sales report" into the terminal.

[1096] The device receives these instructions via voice recognition or text input.

[1097] Step 2:

[1098] The terminal analyzes the user's instructions. Natural language processing technology is used to extract task details (sales report generation) from the text.

[1099] The terminal sends the analysis results (task details) to the server.

[1100] Step 3:

[1101] The device collects the user's voice and facial expression data through its camera and microphone.

[1102] The device sends the collected data to the emotion engine.

[1103] Step 4:

[1104] The emotion engine analyzes voice and facial expression data to determine the user's emotional state (e.g., relaxed, stressed).

[1105] The emotion engine sends the analysis results (emotional information) to the server.

[1106] Step 5:

[1107] The server analyzes the instructions and emotional information received from the terminal.

[1108] The server starts the generative AI and loads the necessary settings.

[1109] Step 6:

[1110] The server automatically collects the necessary data based on instructions. This includes retrieving data from customer databases, sales databases, marketing databases, and so on.

[1111] The server communicates with each database via APIs and data connections to collect necessary information.

[1112] Step 7:

[1113] The server performs preprocessing on the collected data. Specifically, this includes data cleaning (imputing missing values ​​and removing outliers), normalization, and selecting necessary attributes.

[1114] The server prepares the pre-processed data into a format that can be used by generative AI for analysis.

[1115] Step 8:

[1116] A generative AI performs tasks using pre-processed data. Here, it calculates metrics such as sales revenue, conversion rate, and customer feedback.

[1117] Generative AI prepares data to generate explanatory text and graphs based on the analysis results.

[1118] Step 9:

[1119] The generative AI uses text generation technology to create a report based on the analysis results. The report includes insights, sales trend graphs, and statistical information.

[1120] The generative AI returns the final generated report to the server.

[1121] Step 10:

[1122] The server sends the generated report and sentiment information to the terminal. The communication uses the HTTPS protocol to ensure data security.

[1123] The server records transmission logs for later use in troubleshooting and feedback.

[1124] Step 11:

[1125] Based on the reports received by the device, the presentation method is adjusted according to the user's perceived emotions. For example, if the user is feeling stressed, the results are displayed in a concise and easy-to-understand format.

[1126] The terminal displays the generated report on the screen, allowing the user to review it.

[1127] Following the steps described above, the present invention automatically performs data collection, preprocessing, analysis, result generation, and presentation based on user instructions. Furthermore, it improves the user experience by recognizing the user's emotions and adjusting the result presentation method accordingly.

[1128] (Example 2)

[1129] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1130] Conventional data analysis systems often required numerous manual operations from the user's input of specific instructions to obtaining results, resulting in lengthy result generation times. Furthermore, they struggled to respond flexibly to the user's emotional state, sometimes leading to a diminished user experience. To address these challenges, it is necessary to provide a system that allows users to easily, quickly, and accurately obtain data analysis results while also presenting appropriate information based on their emotional state.

[1131] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1132] In this invention, the server includes means for receiving instructions from a user, means for interpreting and analyzing specific tasks based on the instructions, means for collecting and pre-processing necessary data, means for executing a specified task using generative artificial intelligence and generating results, means for recognizing the user's emotions, means for reflecting the user's emotions in the generated results and how they are presented, and means for presenting the generated results to the user. This makes it possible to automate everything from receiving instructions from the user to data collection, analysis, result generation, and flexible information presentation based on emotions, thereby improving the user experience.

[1133] A "user" is a person or entity that operates a system and provides instructions or inputs.

[1134] "Instructions" refer to the specific operations or commands that a user performs on the system.

[1135] A "task" is a specific task or process that a system should perform based on instructions.

[1136] "Analysis" refers to data processing and analysis that interprets user instructions and translates them into specific tasks.

[1137] "Data" is a collection of information that a system needs to perform a task.

[1138] "Data collection" is the process of obtaining necessary data from various data sources.

[1139] "Preprocessing" refers to the process of preparing collected data for analysis or task execution.

[1140] "Generative artificial intelligence" refers to artificial intelligence technology used to perform specified tasks and generate results.

[1141] "Results" refer to the output or outcome obtained when a generative artificial intelligence performs a task.

[1142] "Emotions" are internal reactions that indicate a user's psychological state or feedback.

[1143] "Recognition" is the process of analyzing the user's voice and facial expressions to determine their emotions.

[1144] "Presentation" refers to the act of showing the generated results to the user visually or audibly.

[1145] The system of this invention performs advanced data analysis tasks based on user instructions and provides the results quickly and accurately. Furthermore, it can recognize the user's emotions and adjust the way the results are presented accordingly. This system is comprised of the following key hardware and software components.

[1146] User input

[1147] When users input instructions into the terminal, they use voice recognition technology (e.g., a voice recognition API) or keyboard input. The input is a prompt message such as "Generate this week's sales report."

[1148] Instruction parsing

[1149] The device uses speech recognition technology (speech recognition API) to convert voice input into text in order to analyze user instructions. Next, it uses natural language processing technology (natural language processing API) to analyze the input text and extract the task details. The analysis results are stored in a database and sent from the device to the server.

[1150] Transmission of commands and activation of generative AI

[1151] The HTTPS protocol is used when the terminal sends the analyzed instructions to the server. Based on the received analysis results, the server determines what type of report to generate. The server then activates the generative artificial intelligence (generative artificial intelligence API) and loads the necessary settings.

[1152] Emotional engine activation

[1153] The device collects the user's voice and facial expression data through its camera and microphone and sends it to an emotion engine (emotion recognition API). The emotion engine analyzes the user's emotions and sends the results to a server.

[1154] Data collection

[1155] When the server automatically collects the necessary data, it retrieves data from external data sources such as customer databases, sales databases, and marketing databases through APIs and data connections.

[1156] Data preprocessing

[1157] The server preprocesses the collected data. This includes imputing missing values, removing outliers, and standardizing data formats, preparing the dataset for analysis.

[1158] Task execution and result generation

[1159] Generative artificial intelligence begins analysis using pre-processed data. For example, it calculates metrics such as sales revenue, conversion rate, and customer feedback. A report is created from the generated data, and explanatory text containing insights is generated using text generation technology.

[1160] Sending and presenting results

[1161] The server sends the generated report to the terminal using the HTTPS protocol. The terminal analyzes the received report and adjusts the presentation method based on the user's emotion recognition results. For example, if the user is stressed, important information is highlighted and displayed clearly. Conversely, if the user is relaxed, detailed information is also presented.

[1162] Specific example

[1163] Sales report generation and sentiment feedback

[1164] The user enters a command into the terminal, such as "Generate this week's sales report." The terminal analyzes this command and sends it to the server. The server collects the necessary data and preprocesses it. The emotion engine analyzes the user's emotions and sends the results to the server. For example, if it determines that the user is feeling stressed, it appropriately highlights important information in the report. Generative artificial intelligence analyzes the sales data and generates the report. The server then sends the generated report to the terminal, which displays it to the user.

[1165] This allows users to obtain data analysis results quickly and accurately, and enables the presentation of information tailored to the user's emotional state.

[1166] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1167] Step 1:

[1168] The user inputs the command "Generate this week's sales report" into the device. Speech recognition technology (such as a speech recognition API) is used to convert the voice input into text. The input data is output in text format as "Generate this week's sales report".

[1169] Step 2:

[1170] The terminal analyzes the input text using natural language processing technology (such as a natural language processing API). Specifically, it extracts the keywords "generate," "sales report," and "this week," and recognizes them as tasks. The analysis results are output as JSON data containing the task details, such as "Generate Sales Report."

[1171] Step 3:

[1172] The terminal sends the analysis results to the server using the HTTPS protocol. The transmitted data is in JSON format and contains task details. The server checks the received instructions and determines which type of task to execute. Based on this determination, the instruction "Generate Sales Report" is issued.

[1173] Step 4:

[1174] The device collects user voice and facial expression data through its camera and microphone and sends it to an emotion engine (emotion recognition API). For example, it analyzes stress levels from the user's face. The emotion engine analyzes the emotional state and outputs the result to the server as data such as "the user is feeling stressed."

[1175] Step 5:

[1176] The server collects the necessary data based on the "Sales Report Generation" task. For example, it retrieves data from external data sources such as customer databases, sales databases, and marketing databases via APIs. The collected data includes customer information and sales data. This data is aggregated on the server in an integrated format.

[1177] Step 6:

[1178] The server preprocesses the collected data. Specifically, it imputes missing values, removes outliers, and standardizes the data format. The preprocessed dataset is output in a format suitable for analysis.

[1179] Step 7:

[1180] Generative artificial intelligence performs tasks using pre-processed data. Specifically, it calculates metrics such as sales trends, conversion rates, and customer feedback. Based on the analysis results, the generative AI creates a report and generates explanatory text containing insights using text generation technology. The generated report is output in text and graph formats.

[1181] Step 8:

[1182] The server sends the generated report to the terminal using the HTTPS protocol. The transmitted data includes not only the report information but also the user's emotional state obtained from the emotion engine.

[1183] Step 9:

[1184] The device analyzes the received report and adjusts the presentation method based on the user's emotional state. For example, if the user is feeling stressed, important parts of the report will be highlighted. The presentation method ensures that the information is presented in a format that is easiest for the user to understand.

[1185] Example prompt statements

[1186] "Generate this week's sales report."

[1187] "A compilation of the latest customer feedback"

[1188] "Tell me your sales forecast for next week."

[1189] The above outlines the detailed processing steps of the system. This process allows users to obtain data analysis results quickly and accurately, and further facilitates the presentation of information tailored to the user's emotional state.

[1190] (Application Example 2)

[1191] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1192] Traditional in-store customer service struggled to recognize customer emotions in real time and suggest appropriate products and services based on those emotions. Furthermore, the lack of systems to provide feedback and recommendations tailored to customer emotional states prevented improvements in the customer experience. This resulted in problems such as decreased customer satisfaction and reduced purchasing intent.

[1193] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving instructions from the user, means for interpreting and analyzing specific tasks based on the instructions, and means for recognizing the user's emotions and adjusting the method of presenting the results generated based on those emotions. This makes it possible to recognize customer emotions in real time in a physical store and propose appropriate products and services based on those emotions.

[1194] "Means for receiving instructions from the user" refers to an interface for receiving voice or text instructions entered by the user through a device.

[1195] "Means for interpreting and analyzing specific tasks based on instructions" refers to a process that uses natural language processing technology to analyze the user's instructions and determine which task to execute.

[1196] "Means for collecting and pre-processing necessary data" refers to data processing functions that collect necessary information from diverse data sources and perform pre-processing such as data cleansing and formatting standardization.

[1197] "A means of executing a specified task and generating results using generative AI" refers to an algorithm that uses a generative AI model to perform a specified task from pre-processed data and generate results.

[1198] "Means for recognizing user emotions and adjusting the presentation method of results generated based on those emotions" refers to technology that analyzes user emotions from facial expressions and voice, and presents the results in the most optimal format based on the analysis results.

[1199] "Means of presenting the generated results to the user" refers to UI / UX functions for displaying the generated results on the user's device.

[1200] A description of embodiments for carrying out the present invention will be provided.

[1201] System Configuration

[1202] This system can recognize user emotions and provide personalized product and service recommendations based on those emotions. The system consists of the following components:

[1203] 1. Devices: These include user devices such as smart glasses and smartphones. They are used to collect user input and emotional data.

[1204] 2. Server: The cloud server performs instruction analysis, data collection, preprocessing, activation of generative AI, result generation, and result presentation.

[1205] 3. Emotion Recognition Engine: This engine analyzes data collected by the device through the camera and microphone to recognize the user's emotions. For example, Microsoft Azure's Emotion API is used for this engine.

[1206] 4. Generative AI: AI modules that perform tasks using pre-processed data and generate results. For example, OpenAI's GPT model is used.

[1207] Data collection and analysis

[1208] The device collects the user's voice and facial expression data in real time. This data is sent to an emotion recognition engine to recognize the user's emotions. The emotion-recognized data is sent to a server and analyzed along with the user's instructions.

[1209] The analysis uses natural language processing technology (e.g., Google Cloud Natural Language API) to convert the instructions into tasks. For example, if a user enters "I'm looking for new summer clothes," this instruction is parsed into the task "Recommend new summer items."

[1210] Data collection and preprocessing

[1211] The server collects the necessary data from the database based on the analyzed task. This data may include customer purchase history, inventory information, and promotional information. The collected data is preprocessed (data cleansing, formatting, etc.) and prepared in a format suitable for generative AI.

[1212] Task execution using generative AI

[1213] A generative AI analyzes pre-processed data to generate optimal product and service recommendations based on the user's emotions and preferences. The generated results are then sent back to the terminal via the server.

[1214] Presentation of results

[1215] The device displays results generated based on the user's emotions. It adjusts the display method and emphasis depending on whether the user is relaxed or stressed. For example, it highlights relaxing items for tense users and recommends stimulating items for excited users.

[1216] Specific example

[1217] For example, if a customer voice-inputs "I'm looking for new summer clothes" into smart glasses, the emotion recognition engine detects that the customer is relaxed. The system then generates a list of the latest summer items and displays them in a relaxed and easy-to-view format.

[1218] Example of a prompt

[1219] "Please generate a list of summer products that customers can easily access."

[1220] As described above, the present invention can provide a more personalized user experience by recognizing the user's emotions and making recommendations based on them. Furthermore, by combining generative AI with an emotion recognition engine, it becomes possible to respond appropriately to the diverse needs of users.

[1221] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1222] Step 1:

[1223] The user inputs instructions into the device. The user inputs instructions via voice or text using smart glasses or a smartphone. These instructions are sent to the device as input data. For example, the instruction "I want new summer clothes" is entered into the device.

[1224] Step 2:

[1225] The device collects the user's emotions. The device uses its camera and microphone to capture the user's facial expressions and tone of voice in real time and sends this to an emotion recognition engine. The emotion recognition engine (e.g., Microsoft Azure's Emotion API) analyzes the collected data and recognizes the user's emotions (e.g., relaxed, excited, stressed). This recognition result is returned to the device as emotion data.

[1226] Step 3:

[1227] The device analyzes the user's instructions and extracts specific tasks. The device uses natural language processing technology (e.g., Google Cloud Natural Language API) to analyze instructions entered via voice or text. The analyzed instructions are sent to the server as text data. For example, the instruction "I'm looking for new summer clothes" is converted into the task "Recommend new summer items."

[1228] Step 4:

[1229] The server collects and preprocesses the necessary data. The server accesses databases (e.g., customer purchase history, inventory information, promotional information) and collects the required data based on the task. The collected data undergoes preprocessing such as imputing missing values, removing outliers, and standardizing data formats. The preprocessed data is then sent to the generative AI.

[1230] Step 5:

[1231] The server uses generative AI to perform the specified task. The generative AI (e.g., OpenAI GPT model) generates a list of optimal products and services based on pre-processed data, taking into account the user's emotions and instructions. This generation process uses the prompt "Generate a list of summer products that are easily accessible to the customer." The generated results are returned to the server as recommendation data.

[1232] Step 6:

[1233] The server sends recommendation data to the device. The server sends the generated results to the device using the HTTPS protocol to ensure data security. The final presented data is then sent to the device.

[1234] Step 7:

[1235] The device presents results based on the user's emotions. Based on the user's emotional data, the device adjusts the presentation method and emphasis. For example, if the user is relaxed, the results are displayed in a calm tone; if they are stressed, the results are displayed more intuitively and concisely. The user then reviews the results and makes a selection of products or services.

[1236] Through these steps, personalized product and service recommendations that take user emotions into consideration are achieved.

[1237] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1238] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1239] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1240] [Fourth Embodiment]

[1241] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1242] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1243] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1244] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1245] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1246] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1247] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1248] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1249] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1250] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1251] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1252] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1253] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1254] The embodiments for carrying out the present invention are described below.

[1255] This system aims to enable users to easily perform advanced data analysis tasks and obtain results quickly and accurately. This allows for the generation of complex reports and predictive analytics without the manual operations and advanced expertise previously required.

[1256] System Overview

[1257] When a user enters a specific instruction into the terminal, the instruction is analyzed and sent to the server. The server collects the necessary data based on the instruction and performs preprocessing. A generative AI then uses the preprocessed data to execute the specified task and presents the results to the user through the terminal.

[1258] Program processing

[1259] 1. User input

[1260] The user enters the instruction "Generate this week's sales report" into the terminal. This input is done via voice recognition or keyboard input.

[1261] 2. Instruction Analysis

[1262] The device receives user instructions and analyzes their content. This analysis uses natural language processing technology to determine the meaning of the instructions.

[1263] 3. Transmission of commands and activation of generative AI

[1264] The terminal sends the analysis results to the server. This is done using network communication.

[1265] The server activates the generative AI based on the received instructions.

[1266] 4. Data Collection

[1267] The server automatically collects the necessary data. For example, it retrieves the required information from customer databases, sales databases, and marketing databases.

[1268] 5. Data preprocessing

[1269] The server preprocesses the collected data. Specifically, this involves data cleaning, normalization, and feature selection. This process prepares the data for analysis.

[1270] 6. Task execution and result generation

[1271] Generative AI uses pre-processed data to perform specified tasks. For example, it can analyze sales trends or perform statistical analysis of customer feedback.

[1272] Generative AI creates reports from generated data and uses text generation technology to produce explanatory text that includes insights.

[1273] 7. Submission and presentation of results

[1274] The server sends the generated report to the terminal. This communication is conducted through a highly secure protocol.

[1275] The terminal analyzes the received report and displays it to the user in an appropriate format. This uses an interactive GUI that includes graphs and text.

[1276] Specific example

[1277] Sales report generation

[1278] 1. The user enters "Generate this week's sales report" into the terminal.

[1279] 2. The terminal analyzes this instruction and sends it to the server.

[1280] 3. The server activates the generative AI and collects and preprocesses the necessary data.

[1281] 4. Generative AI analyzes sales data and generates reports. These reports include monthly sales trends and feedback from key customers.

[1282] 5. The server sends the generated report to the terminal, and the terminal displays it to the user.

[1283] Thus, this system is designed to allow users to easily perform advanced data analysis by automatically collecting, preprocessing, analyzing, generating, and presenting data based on user instructions.

[1284] The following describes the processing flow.

[1285] Step 1:

[1286] The user enters the instruction "Generate this week's sales report" into the terminal.

[1287] The device receives these instructions through voice recognition or text analysis.

[1288] Step 2:

[1289] The device analyzes the user's instructions and extracts the task details from the text.

[1290] The terminal sends the analysis results (task details) to the server.

[1291] Step 3:

[1292] The server analyzes the instructions received from the terminal and determines what type of report needs to be generated.

[1293] The server starts the generative AI and loads the necessary settings.

[1294] Step 4:

[1295] The server collects the necessary data based on instructions. For example, it retrieves data from customer databases, sales databases, marketing databases, etc.

[1296] The server communicates with each database via APIs and data connections to collect necessary information.

[1297] Step 5:

[1298] The server preprocesses the collected data. Specifically, this involves imputing missing values, removing outliers, and standardizing data formats.

[1299] The server prepares the pre-processed data as a dataset for analysis.

[1300] Step 6:

[1301] The generative AI begins its analysis using pre-processed data. Here, it calculates metrics such as sales revenue, conversion rate, and customer feedback.

[1302] The generative AI uses the results to prepare data for generating explanatory text and graphs.

[1303] Step 7:

[1304] The generative AI generates a report based on the analysis results. The report includes insights created using text generation technology, sales trend graphs, and statistical information.

[1305] The generative AI returns the final generated report to the server.

[1306] Step 8:

[1307] The server sends the generated report to the terminal. Communication is usually done using the HTTPS protocol to ensure data security.

[1308] The server records transmission logs for later use in troubleshooting and feedback.

[1309] Step 9:

[1310] The terminal analyzes the received reports and displays them in a user-friendly format. This includes GUI rendering using HTML and CSS.

[1311] The terminal will provide reports in an interactive format, allowing users to quickly obtain the information they need.

[1312] The above outlines the specific processing steps of the system that generates sales reports based on user instructions and presents the results.

[1313] (Example 1)

[1314] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1315] Traditionally, advanced data analysis and complex report generation required users to possess advanced expertise and manual operation. This resulted in significant time and effort, making it inefficient. This invention solves this problem by providing a system that allows users to perform data analysis and report generation easily and quickly without requiring specialized knowledge or manual operation.

[1316] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1317] In this invention, the server includes means for receiving instructions from a user, means for interpreting tasks based on the instructions and analyzing them using natural language processing technology, means for sending the analyzed instructions to the server and activating a generative AI, means for collecting necessary data from a database, cleaning and normalizing the data, and performing preprocessing to select features, means for executing a specified task based on the preprocessed data using the generative AI and generating results, and means for presenting the generated results to the user in an interactive format on a terminal. This enables users to perform advanced data analysis and report generation in a short time without specialized knowledge.

[1318] 1. "Means for receiving instructions from the user" refers to an interface that allows the user to input instructions to the system using methods such as voice recognition or keyboard input.

[1319] 2. "Natural language processing technology" refers to technologies that enable computers to understand and analyze human language, specifically technologies that perform text analysis and semantic analysis.

[1320] 3. "Means for interpreting and analyzing tasks" refers to the process of analyzing instructions received from the user, understanding their content, and determining the appropriate action to take.

[1321] 4. "Generative AI" refers to artificial intelligence models that have been trained to perform specified tasks, and are models that automatically perform data analysis and report generation.

[1322] 5. "Means of data collection" refers to mechanisms for obtaining necessary data from various databases and external data sources.

[1323] 6. "Data preprocessing means" refers to processes such as cleaning, normalization, and feature selection that are performed to prepare collected data into a format suitable for analysis.

[1324] 7. "Means for executing tasks and generating results" refers to the process by which a generative AI performs specified data analysis and report generation tasks based on pre-processed data and creates the results.

[1325] 8. "Means for presenting the generated results to the user on the terminal" refers to an interface for visually displaying the generated analysis results and reports to the user.

[1326] This invention provides a system that allows users to easily perform advanced data analysis and report generation. Specific embodiments for carrying out this invention are described below.

[1327] System Overview

[1328] This system consists of three main components: a user, a terminal, and a server. The user inputs instructions into the terminal via voice recognition or keyboard input. The terminal analyzes the instructions using natural language processing technology and sends them to the server. The server activates a generative AI to collect and preprocess the necessary data, then generates analysis results and reports, which are presented to the user.

[1329] Hardware and software to use

[1330] 1. Terminal

[1331] Hardware: PCs, tablets, smartphones, etc.

[1332] Software: Google Speech-to-Text API (speech recognition), TensorFlow, NLTK (natural language processing)

[1333] 2. Server

[1334] Hardware: Cloud servers, on-premises servers

[1335] Software: MySQL / PostgreSQL (databases), Pandas, Scikit-learn (data preprocessing), GPT-4 (generative AI model)

[1336] 3. Communications

[1337] Protocol: HTTPS (Secure communication)

[1338] Specific examples of actions

[1339] Sales report generation

[1340] 1. The user enters "Generate this week's sales report" into the terminal.

[1341] Example prompt: "Generate this week's sales report."

[1342] 2. The device receives instructions using speech recognition technology (Google Speech-to-Text API) or keyboard input.

[1343] In the case of voice input, the speech is converted to text.

[1344] 3. The device analyzes the instructions using TensorFlow or NLTK.

[1345] Identify tasks based on the keyword "sales report".

[1346] 4. The terminal sends the analyzed commands to the server using the HTTPS protocol.

[1347] 5. The server starts the generative AI model (GPT-4) based on the instructions.

[1348] 6. The server collects sales data and customer data from MySQL or PostgreSQL.

[1349] Execute the SQL query and extract the necessary data.

[1350] 7. The server performs data preprocessing using Pandas or Scikit-learn.

[1351] Cleaning (imputing missing values ​​and deleting invalid data)

[1352] Data normalization (converting values ​​to a scale of 0 to 1)

[1353] Feature selection (select important variables)

[1354] 8. The generative AI model performs the specified task based on the pre-processed data and generates the results.

[1355] We analyze sales trends and customer feedback statistically, and then use text generation technology to generate explanatory text that includes insights.

[1356] 9. The server sends the generated report to the terminal using the HTTPS protocol.

[1357] 10. The terminal presents the received report to the user in an interactive format.

[1358] This program displays graphs using D3.js or Chart.js, along with text styled using HTML and CSS.

[1359] Effects of implementation

[1360] This system allows users to easily perform data analysis and generate reports without requiring advanced expertise. This improves the quality of data-driven decision-making and significantly enhances operational efficiency.

[1361] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1362] Step 1:

[1363] The user enters "Generate this week's sales report" into the terminal. Input is done via voice recognition or keyboard.

[1364] Input: User instruction ("Generate this week's sales report")

[1365] Output: Instruction text displayed on the terminal

[1366] Step 2:

[1367] The device receives user instructions using speech recognition technology (Google Speech-to-Text API) or keyboard input. The received instructions are then analyzed using natural language processing technology (TensorFlow, NLTK, etc.).

[1368] Input: Speech recognition result or keyboard input text

[1369] Output: Analyzed instructions (report generation task)

[1370] Step 3:

[1371] The terminal transfers the analysis results to the server. This communication is securely performed using the HTTPS protocol.

[1372] Input: Analyzed instructions

[1373] Output: Parsing instructions to the server

[1374] Step 4:

[1375] The server launches a generative AI model (GPT-4) based on the analysis results. The generative AI then prepares to execute the specified task.

[1376] Input: Analyzed instructions

[1377] Output: Preparation for starting a generative AI

[1378] Step 5:

[1379] The server collects the necessary data. Specifically, it executes SQL queries to retrieve sales data and customer data from databases (such as MySQL and PostgreSQL).

[1380] Input: Data to be collected as instructed

[1381] Output: Acquired dataset (sales data, customer data, etc.)

[1382] Step 6:

[1383] The server preprocesses the collected data. It uses Pandas and Scikit-learn to perform tasks such as data cleaning, normalization, and feature selection.

[1384] Input: Acquired dataset

[1385] Output: Preprocessed data

[1386] Step 7:

[1387] Generative AI models perform specified tasks based on pre-processed data and generate results. For example, they can analyze sales trends or perform statistical analysis of customer feedback.

[1388] Input: Preprocessed data

[1389] Output: Analysis results and report content

[1390] Step 8:

[1391] The server sends the generated results to the terminal. This communication also uses the HTTPS protocol.

[1392] Input: Generated results (report content)

[1393] Output: Sending results to the terminal

[1394] Step 9:

[1395] The terminal analyzes the received report and displays it to the user in an interactive format. It visualizes the data using D3.js and Chart.js, and styles it with HTML and CSS.

[1396] Input: Report results from the server

[1397] Output: Interactive report display on the user's device.

[1398] (Application Example 1)

[1399] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1400] Current factory operations generate a large amount of data, but analyzing and reporting on it requires advanced expertise and manual operation, making efficient data utilization difficult. Furthermore, there is a need for real-time monitoring of operating conditions and rapid report generation, but there is a lack of effective means to achieve this.

[1401] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1402] In this invention, the server includes means for receiving instructions from a user, means for interpreting and analyzing specific tasks based on the instructions, means for collecting and pre-processing necessary data, means for executing specified tasks using generative AI and generating results, means for presenting the generated results to the user, means for creating an operational status report from the generated data, and means including an interactive interface for displaying the operational status report. This enables the user to easily perform advanced data analysis, generate status reports in real time, and respond quickly.

[1403] "Means of receiving instructions from the user" refers to an interface that receives instructions from the user via voice input or text input, and a system that interprets that input data.

[1404] "A means of interpreting and analyzing a specific task based on instructions" refers to a system that uses natural language processing technology to identify a task from user input and analyze its content.

[1405] "Means for collecting and pre-processing necessary data" refers to processes and systems that collect appropriate data from sensors, databases, etc., and then clean and normalize that data.

[1406] "A means of executing a specified task and generating results using generative AI" refers to a system that utilizes generative artificial intelligence to analyze pre-processed data and generate the necessary results and reports.

[1407] "Means of presenting generated results to the user" refers to a system that displays generated data and reports to the user through an interface.

[1408] "Means for creating operational status reports from generated data" refers to a system that automatically generates reports on operational status based on collected and pre-processed data.

[1409] "Means including an interactive interface for displaying operational status reports" refers to an interface that visually displays the generated reports and allows users to interact with them.

[1410] This invention relates to a system that enables the efficient collection and analysis of data within a factory, as well as the automation of report generation. The aim of this system is to allow users to easily perform advanced data analysis tasks and obtain results quickly and accurately.

[1411] System Overview

[1412] First, the user gives instructions to the robot via voice or text input. For example, an instruction such as "Generate this week's operational status report" is conceivable. The robot uses a voice recognition system to convert these instructions into text and analyzes its content. Specifically, it uses natural language processing technology to determine the meaning of the instructions.

[1413] Hardware and software

[1414] The hardware used in this process includes a voice input device for recognizing the user's voice, the robot body, and a central server. The software includes a voice recognition system (e.g., Dialogflow or other natural language processing (NLP) techniques), Python libraries for data collection and preprocessing (e.g., Pandas, NumPy), and a model as a generative AI (e.g., GPT-4).

[1415] Data collection and preprocessing

[1416] The server collects necessary data from various sensors and databases. PLCs (Programmable Logic Controllers) and IoT devices are used for data collection. The collected data is cleaned and normalized using Python's Pandas and NumPy libraries.

[1417] Report generation

[1418] Generative AI (e.g., GPT-4) uses pre-processed data to perform a specified task and generate results. An operational status report is automatically created from the generated data. This report includes uptime, downtime, and other key metrics.

[1419] Presentation of results

[1420] The generated report is sent from the server to the robot and displayed to the user through the robot's interface. An interactive GUI is used for display, and data visualization using tools such as Tableau or D3.js is envisioned.

[1421] Specific example

[1422] A concrete example is shown below. When a user gives a voice command to the robot, "Generate this week's operational status report," the robot analyzes this command and sends a request to a central server. The server collects the necessary data from various sensors and databases and preprocesses it using Python's Pandas and NumPy. The preprocessed data is then analyzed by GPT-4, and an operational status report is generated. The generated report is sent to the robot and displayed to the user in an interactive format.

[1423] Example of a prompt

[1424] User: "Generate this week's operational status report."

[1425] Robot: "Analyzing the command..."

[1426] Robot: "Collecting data..."

[1427] Robot: "We are pre-processing the data..."

[1428] Robot: "Generating report..."

[1429] Robot: "Here is this week's operational report. Operating hours and downtime are as follows..."

[1430] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1431] Step 1:

[1432] Receiving and analyzing user instructions

[1433] The user instructs the robot via voice or text input, "Generate this week's operational status report." The robot uses a speech recognition system to convert the speech into text and natural language processing technology to analyze the meaning of the instruction. Specifically, a voice input device captures the user's voice, and speech recognition software converts it into text data. Next, a natural language processing engine analyzes the instruction and recognizes the specific task (in this case, generating an operational status report).

[1434] Input: User's voice instructions

[1435] Output: Analyzed text instructions

[1436] Step 2:

[1437] Transmission of instructions to the server

[1438] The terminal sends the analysis results to the server. This communication is secure using the HTTPS protocol. Specifically, the robot sends the analyzed text data to the server via the network communication module. The server receives this instruction and proceeds to the next data collection process.

[1439] Input: Parsed text instructions

[1440] Output: Instruction data to the server

[1441] Step 3:

[1442] Data collection

[1443] The server collects the necessary data from sensors and databases within the factory. PLCs and IoT devices are used to collect data from sensors. The server sends data requests back to each device and receives operational status data in return.

[1444] Input: Instruction data to the server

[1445] Output: Collected raw data

[1446] Step 4:

[1447] Data preprocessing

[1448] The server preprocesses the collected data. Preprocessing includes data cleaning (imputing missing values ​​and removing outliers), normalization, and feature selection. Specifically, it uses Python's Pandas and NumPy to generate dataframes and perform cleanup and standardization operations.

[1449] Input: Collected raw data

[1450] Output: Preprocessed data

[1451] Step 5:

[1452] Report generation

[1453] The server uses pre-processed data to execute specified tasks using a generative AI (e.g., GPT-4) and generate reports. The generative AI analyzes the input data and automatically creates operational status reports. Specifically, the generative AI model takes in pre-processed data and performs statistics on uptime and downtime analysis.

[1454] Input: Preprocessed data

[1455] Output: Generated operational status report

[1456] Step 6:

[1457] Presentation of results

[1458] The server sends the generated report to the robot, which then presents it to the user. An interactive GUI is used for the display, allowing the user to visually check the operational status. Specifically, visualization tools such as Tableau and D3.js are used.

[1459] Input: Generated operational status report

[1460] Output: An interactive report displayed to the user.

[1461] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1462] The present invention will now describe embodiments for carrying out the present invention. The present invention is a system that combines user emotion recognition technology, enabling users to easily perform advanced data analysis tasks and obtain results quickly and accurately.

[1463] System Overview

[1464] When a user enters a specific instruction into the device, the instruction is analyzed and sent to the server. The server collects the necessary data based on the instruction and performs preprocessing. A generative AI uses the preprocessed data to execute the specified task and presents the results to the user through the device. In addition, an emotion engine recognizes the user's emotions and reflects them in the generated results and how they are presented.

[1465] Program processing

[1466] 1. User input

[1467] The user inputs a command into the terminal, such as "Generate this week's sales report." This input is done via voice recognition or keyboard input.

[1468] 2. Instruction Analysis

[1469] The device analyzes the user's instructions and extracts task details from the text. Natural language processing technology is used for the analysis.

[1470] The terminal sends the analysis results (task details) to the server.

[1471] 3. Transmission of commands and activation of generative AI

[1472] The terminal sends the analysis results to the server. Network protocols are used for communication.

[1473] The server analyzes the received instructions and determines what type of report needs to be generated.

[1474] The server starts the generative AI and loads the necessary settings.

[1475] 4. Activation of the Emotional Engine

[1476] The device collects the user's voice and facial expressions through its camera and microphone and transmits them to the emotion engine.

[1477] The emotion engine analyzes the user's emotions and sends the results to the server.

[1478] 5. Data Collection

[1479] The server automatically collects the necessary data, including data retrieval from customer databases, sales databases, and marketing databases.

[1480] The server communicates with each database via APIs and data connections to collect necessary information.

[1481] 6. Data preprocessing

[1482] The server preprocesses the collected data, performing tasks such as imputing missing values, removing outliers, and standardizing data formats.

[1483] The server prepares the pre-processed data as a dataset for analysis.

[1484] 7. Task execution and result generation

[1485] The generative AI begins analysis using pre-processed data. For example, it calculates metrics such as sales revenue, conversion rate, and customer feedback.

[1486] Generative AI creates reports from generated data, and text generation technology generates explanatory text that includes insights.

[1487] 8. Submission and presentation of results

[1488] The server sends the generated report to the terminal. This communication uses the HTTPS protocol to ensure data security.

[1489] The device analyzes the received reports and adjusts the presentation method based on the user's perceived emotions. The format and emphasis of the presentation will differ depending on whether the user is relaxed or stressed.

[1490] Specific example

[1491] Sales report generation and sentiment feedback

[1492] 1. The user enters "Generate this week's sales report" into the terminal.

[1493] 2. The terminal analyzes the instructions and sends the information to the server.

[1494] 3. The server activates the generative AI and collects and preprocesses the necessary data.

[1495] 4. The emotion engine analyzes the user's emotions and sends the results to the server. For example, it might determine that the user is feeling stressed.

[1496] 5. Generative AI analyzes sales data and generates reports. These reports include sales trends and feedback from key customers.

[1497] 6. The server sends the generated effet and sentiment information to the terminal, which then displays it to the user.

[1498] Thus, this system automatically performs tasks from data collection and preprocessing to analysis, result generation, and presentation based on user instructions. Furthermore, it can improve the user experience by recognizing user emotions and adjusting the way results are presented accordingly.

[1499] The following describes the processing flow.

[1500] Step 1:

[1501] The user enters the instruction "Generate this week's sales report" into the terminal.

[1502] The device receives these instructions via voice recognition or text input.

[1503] Step 2:

[1504] The terminal analyzes the user's instructions. Natural language processing technology is used to extract task details (sales report generation) from the text.

[1505] The terminal sends the analysis results (task details) to the server.

[1506] Step 3:

[1507] The device collects the user's voice and facial expression data through its camera and microphone.

[1508] The device sends the collected data to the emotion engine.

[1509] Step 4:

[1510] The emotion engine analyzes voice and facial expression data to determine the user's emotional state (e.g., relaxed, stressed).

[1511] The emotion engine sends the analysis results (emotional information) to the server.

[1512] Step 5:

[1513] The server analyzes the instructions and emotional information received from the terminal.

[1514] The server starts the generative AI and loads the necessary settings.

[1515] Step 6:

[1516] The server automatically collects the necessary data based on instructions. This includes retrieving data from customer databases, sales databases, marketing databases, and so on.

[1517] The server communicates with each database via APIs and data connections to collect necessary information.

[1518] Step 7:

[1519] The server performs preprocessing on the collected data. Specifically, this includes data cleaning (imputing missing values ​​and removing outliers), normalization, and selecting necessary attributes.

[1520] The server prepares the pre-processed data into a format that can be used by generative AI for analysis.

[1521] Step 8:

[1522] A generative AI performs tasks using pre-processed data. Here, it calculates metrics such as sales revenue, conversion rate, and customer feedback.

[1523] Generative AI prepares data to generate explanatory text and graphs based on the analysis results.

[1524] Step 9:

[1525] The generative AI uses text generation technology to create a report based on the analysis results. The report includes insights, sales trend graphs, and statistical information.

[1526] The generative AI returns the final generated report to the server.

[1527] Step 10:

[1528] The server sends the generated report and sentiment information to the terminal. The communication uses the HTTPS protocol to ensure data security.

[1529] The server records transmission logs for later use in troubleshooting and feedback.

[1530] Step 11:

[1531] Based on the reports received by the device, the presentation method is adjusted according to the user's perceived emotions. For example, if the user is feeling stressed, the results are displayed in a concise and easy-to-understand format.

[1532] The terminal displays the generated report on the screen, allowing the user to review it.

[1533] Following the steps described above, the present invention automatically performs data collection, preprocessing, analysis, result generation, and presentation based on user instructions. Furthermore, it improves the user experience by recognizing the user's emotions and adjusting the result presentation method accordingly.

[1534] (Example 2)

[1535] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1536] Conventional data analysis systems often required numerous manual operations from the user's input of specific instructions to obtaining results, resulting in lengthy result generation times. Furthermore, they struggled to respond flexibly to the user's emotional state, sometimes leading to a diminished user experience. To address these challenges, it is necessary to provide a system that allows users to easily, quickly, and accurately obtain data analysis results while also presenting appropriate information based on their emotional state.

[1537] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1538] In this invention, the server includes means for receiving instructions from a user, means for interpreting and analyzing specific tasks based on the instructions, means for collecting and pre-processing necessary data, means for executing a specified task using generative artificial intelligence and generating results, means for recognizing the user's emotions, means for reflecting the user's emotions in the generated results and how they are presented, and means for presenting the generated results to the user. This makes it possible to automate everything from receiving instructions from the user to data collection, analysis, result generation, and flexible information presentation based on emotions, thereby improving the user experience.

[1539] A "user" is a person or entity that operates a system and provides instructions or inputs.

[1540] "Instructions" refer to the specific operations or commands that a user performs on the system.

[1541] A "task" is a specific task or process that a system should perform based on instructions.

[1542] "Analysis" refers to data processing and analysis that interprets user instructions and translates them into specific tasks.

[1543] "Data" is a collection of information that a system needs to perform a task.

[1544] "Data collection" is the process of obtaining necessary data from various data sources.

[1545] "Preprocessing" refers to the process of preparing collected data for analysis or task execution.

[1546] "Generative artificial intelligence" refers to artificial intelligence technology used to perform specified tasks and generate results.

[1547] "Results" refer to the output or outcome obtained when a generative artificial intelligence performs a task.

[1548] "Emotions" are internal reactions that indicate a user's psychological state or feedback.

[1549] "Recognition" is the process of analyzing the user's voice and facial expressions to determine their emotions.

[1550] "Presentation" refers to the act of showing the generated results to the user visually or audibly.

[1551] The system of this invention performs advanced data analysis tasks based on user instructions and provides the results quickly and accurately. Furthermore, it can recognize the user's emotions and adjust the way the results are presented accordingly. This system is comprised of the following key hardware and software components.

[1552] User input

[1553] When users input instructions into the terminal, they use voice recognition technology (e.g., a voice recognition API) or keyboard input. The input is a prompt message such as "Generate this week's sales report."

[1554] Instruction parsing

[1555] The device uses speech recognition technology (speech recognition API) to convert voice input into text in order to analyze user instructions. Next, it uses natural language processing technology (natural language processing API) to analyze the input text and extract the task details. The analysis results are stored in a database and sent from the device to the server.

[1556] Transmission of commands and activation of generative AI

[1557] The HTTPS protocol is used when the terminal sends the analyzed instructions to the server. Based on the received analysis results, the server determines what type of report to generate. The server then activates the generative artificial intelligence (generative artificial intelligence API) and loads the necessary settings.

[1558] Emotional engine activation

[1559] The device collects the user's voice and facial expression data through its camera and microphone and sends it to an emotion engine (emotion recognition API). The emotion engine analyzes the user's emotions and sends the results to a server.

[1560] Data collection

[1561] When the server automatically collects the necessary data, it retrieves data from external data sources such as customer databases, sales databases, and marketing databases through APIs and data connections.

[1562] Data preprocessing

[1563] The server preprocesses the collected data. This includes imputing missing values, removing outliers, and standardizing data formats, preparing the dataset for analysis.

[1564] Task execution and result generation

[1565] Generative artificial intelligence begins analysis using pre-processed data. For example, it calculates metrics such as sales revenue, conversion rate, and customer feedback. A report is created from the generated data, and explanatory text containing insights is generated using text generation technology.

[1566] Sending and presenting results

[1567] The server sends the generated report to the terminal using the HTTPS protocol. The terminal analyzes the received report and adjusts the presentation method based on the user's emotion recognition results. For example, if the user is stressed, important information is highlighted and displayed clearly. Conversely, if the user is relaxed, detailed information is also presented.

[1568] Specific example

[1569] Sales report generation and sentiment feedback

[1570] The user enters a command into the terminal, such as "Generate this week's sales report." The terminal analyzes this command and sends it to the server. The server collects the necessary data and preprocesses it. The emotion engine analyzes the user's emotions and sends the results to the server. For example, if it determines that the user is feeling stressed, it appropriately highlights important information in the report. Generative artificial intelligence analyzes the sales data and generates the report. The server then sends the generated report to the terminal, which displays it to the user.

[1571] This allows users to obtain data analysis results quickly and accurately, and enables the presentation of information tailored to the user's emotional state.

[1572] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1573] Step 1:

[1574] The user inputs the command "Generate this week's sales report" into the device. Speech recognition technology (such as a speech recognition API) is used to convert the voice input into text. The input data is output in text format as "Generate this week's sales report".

[1575] Step 2:

[1576] The terminal analyzes the input text using natural language processing technology (such as a natural language processing API). Specifically, it extracts the keywords "generate," "sales report," and "this week," and recognizes them as tasks. The analysis results are output as JSON data containing the task details, such as "Generate Sales Report."

[1577] Step 3:

[1578] The terminal sends the analysis results to the server using the HTTPS protocol. The transmitted data is in JSON format and contains task details. The server checks the received instructions and determines which type of task to execute. Based on this determination, the instruction "Generate Sales Report" is issued.

[1579] Step 4:

[1580] The device collects user voice and facial expression data through its camera and microphone and sends it to an emotion engine (emotion recognition API). For example, it analyzes stress levels from the user's face. The emotion engine analyzes the emotional state and outputs the result to the server as data such as "the user is feeling stressed."

[1581] Step 5:

[1582] The server collects the necessary data based on the "Sales Report Generation" task. For example, it retrieves data from external data sources such as customer databases, sales databases, and marketing databases via APIs. The collected data includes customer information and sales data. This data is aggregated on the server in an integrated format.

[1583] Step 6:

[1584] The server preprocesses the collected data. Specifically, it imputes missing values, removes outliers, and standardizes the data format. The preprocessed dataset is output in a format suitable for analysis.

[1585] Step 7:

[1586] Generative artificial intelligence performs tasks using pre-processed data. Specifically, it calculates metrics such as sales trends, conversion rates, and customer feedback. Based on the analysis results, the generative AI creates a report and generates explanatory text containing insights using text generation technology. The generated report is output in text and graph formats.

[1587] Step 8:

[1588] The server sends the generated report to the terminal using the HTTPS protocol. The transmitted data includes not only the report information but also the user's emotional state obtained from the emotion engine.

[1589] Step 9:

[1590] The device analyzes the received report and adjusts the presentation method based on the user's emotional state. For example, if the user is feeling stressed, important parts of the report will be highlighted. The presentation method ensures that the information is presented in a format that is easiest for the user to understand.

[1591] Example prompt statements

[1592] "Generate this week's sales report."

[1593] "A compilation of the latest customer feedback"

[1594] "Tell me your sales forecast for next week."

[1595] The above outlines the detailed processing steps of the system. This process allows users to obtain data analysis results quickly and accurately, and further facilitates the presentation of information tailored to the user's emotional state.

[1596] (Application Example 2)

[1597] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1598] Traditional in-store customer service struggled to recognize customer emotions in real time and suggest appropriate products and services based on those emotions. Furthermore, the lack of systems to provide feedback and recommendations tailored to customer emotional states prevented improvements in the customer experience. This resulted in problems such as decreased customer satisfaction and reduced purchasing intent.

[1599] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving instructions from the user, means for interpreting and analyzing specific tasks based on the instructions, and means for recognizing the user's emotions and adjusting the method of presenting the results generated based on those emotions. This makes it possible to recognize customer emotions in real time in a physical store and propose appropriate products and services based on those emotions.

[1600] "Means for receiving instructions from the user" refers to an interface for receiving voice or text instructions entered by the user through a device.

[1601] "Means for interpreting and analyzing specific tasks based on instructions" refers to a process that uses natural language processing technology to analyze the user's instructions and determine which task to execute.

[1602] "Means for collecting and pre-processing necessary data" refers to data processing functions that collect necessary information from diverse data sources and perform pre-processing such as data cleansing and formatting standardization.

[1603] "A means of executing a specified task and generating results using generative AI" refers to an algorithm that uses a generative AI model to perform a specified task from pre-processed data and generate results.

[1604] "Means for recognizing user emotions and adjusting the presentation method of results generated based on those emotions" refers to technology that analyzes user emotions from facial expressions and voice, and presents the results in the most optimal format based on the analysis results.

[1605] "Means of presenting the generated results to the user" refers to UI / UX functions for displaying the generated results on the user's device.

[1606] A description of embodiments for carrying out the present invention will be provided.

[1607] System Configuration

[1608] This system can recognize user emotions and provide personalized product and service recommendations based on those emotions. The system consists of the following components:

[1609] 1. Devices: These include user devices such as smart glasses and smartphones. They are used to collect user input and emotional data.

[1610] 2. Server: The cloud server performs instruction analysis, data collection, preprocessing, activation of generative AI, result generation, and result presentation.

[1611] 3. Emotion Recognition Engine: This engine analyzes data collected by the device through the camera and microphone to recognize the user's emotions. For example, Microsoft Azure's Emotion API is used for this engine.

[1612] 4. Generative AI: AI modules that perform tasks using pre-processed data and generate results. For example, OpenAI's GPT model is used.

[1613] Data collection and analysis

[1614] The device collects the user's voice and facial expression data in real time. This data is sent to an emotion recognition engine to recognize the user's emotions. The emotion-recognized data is sent to a server and analyzed along with the user's instructions.

[1615] The analysis uses natural language processing technology (e.g., Google Cloud Natural Language API) to convert the instructions into tasks. For example, if a user enters "I'm looking for new summer clothes," this instruction is parsed into the task "Recommend new summer items."

[1616] Data collection and preprocessing

[1617] The server collects the necessary data from the database based on the analyzed task. This data may include customer purchase history, inventory information, and promotional information. The collected data is preprocessed (data cleansing, formatting, etc.) and prepared in a format suitable for generative AI.

[1618] Task execution using generative AI

[1619] A generative AI analyzes pre-processed data to generate optimal product and service recommendations based on the user's emotions and preferences. The generated results are then sent back to the terminal via the server.

[1620] Presentation of results

[1621] The device displays results generated based on the user's emotions. It adjusts the display method and emphasis depending on whether the user is relaxed or stressed. For example, it highlights relaxing items for tense users and recommends stimulating items for excited users.

[1622] Specific example

[1623] For example, if a customer voice-inputs "I'm looking for new summer clothes" into smart glasses, the emotion recognition engine detects that the customer is relaxed. The system then generates a list of the latest summer items and displays them in a relaxed and easy-to-view format.

[1624] Example of a prompt

[1625] "Please generate a list of summer products that customers can easily access."

[1626] As described above, the present invention can provide a more personalized user experience by recognizing the user's emotions and making recommendations based on them. Furthermore, by combining generative AI with an emotion recognition engine, it becomes possible to respond appropriately to the diverse needs of users.

[1627] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1628] Step 1:

[1629] The user inputs instructions into the device. The user inputs instructions via voice or text using smart glasses or a smartphone. These instructions are sent to the device as input data. For example, the instruction "I want new summer clothes" is entered into the device.

[1630] Step 2:

[1631] The device collects the user's emotions. The device uses its camera and microphone to capture the user's facial expressions and tone of voice in real time and sends this to an emotion recognition engine. The emotion recognition engine (e.g., Microsoft Azure's Emotion API) analyzes the collected data and recognizes the user's emotions (e.g., relaxed, excited, stressed). This recognition result is returned to the device as emotion data.

[1632] Step 3:

[1633] The device analyzes the user's instructions and extracts specific tasks. The device uses natural language processing technology (e.g., Google Cloud Natural Language API) to analyze instructions entered via voice or text. The analyzed instructions are sent to the server as text data. For example, the instruction "I'm looking for new summer clothes" is converted into the task "Recommend new summer items."

[1634] Step 4:

[1635] The server collects and preprocesses the necessary data. The server accesses databases (e.g., customer purchase history, inventory information, promotional information) and collects the required data based on the task. The collected data undergoes preprocessing such as imputing missing values, removing outliers, and standardizing data formats. The preprocessed data is then sent to the generative AI.

[1636] Step 5:

[1637] The server uses generative AI to perform the specified task. The generative AI (e.g., OpenAI GPT model) generates a list of optimal products and services based on pre-processed data, taking into account the user's emotions and instructions. This generation process uses the prompt "Generate a list of summer products that are easily accessible to the customer." The generated results are returned to the server as recommendation data.

[1638] Step 6:

[1639] The server sends recommendation data to the device. The server sends the generated results to the device using the HTTPS protocol to ensure data security. The final presented data is then sent to the device.

[1640] Step 7:

[1641] The device presents results based on the user's emotions. Based on the user's emotional data, the device adjusts the presentation method and emphasis. For example, if the user is relaxed, the results are displayed in a calm tone; if they are stressed, the results are displayed more intuitively and concisely. The user then reviews the results and makes a selection of products or services.

[1642] Through these steps, personalized product and service recommendations that take user emotions into consideration are achieved.

[1643] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1644] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1645] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1646] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1647] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1648] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1649] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1650] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1651] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1652] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1653] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1654] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1655] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1656] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1657] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1658] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1659] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1660] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1661] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1662] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1663] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1664] The following is further disclosed regarding the embodiments described above.

[1665] (Claim 1)

[1666] Means of receiving instructions from users,

[1667] A means of interpreting and analyzing a specific task based on instructions,

[1668] A means for collecting and pre-processing the necessary data,

[1669] A means of using generative AI to execute a specified task and generate results,

[1670] A means of presenting the generated results to the user,

[1671] ...a system that includes

[1672] (Claim 2)

[1673] The system according to claim 1, having a function to generate and present a sales report to the user based on the user's instructions.

[1674] (Claim 3)

[1675] The system according to claim 1, having a function for automating data collection and preprocessing.

[1676] "Example 1"

[1677] (Claim 1)

[1678] Means of receiving instructions from users,

[1679] A means of interpreting a task based on instructions and analyzing it using natural language processing techniques,

[1680] A means of sending the analyzed commands to a server and activating a generative AI,

[1681] A preprocessing means that collects necessary data from a database, cleans and normalizes the data, and performs feature selection,

[1682] A means of using generative AI to execute a specified task based on pre-processed data and generate results,

[1683] A means of presenting the generated results to the user in an interactive format on the terminal,

[1684] ...

[1685] A system that includes this.

[1686] (Claim 2)

[1687] The system according to claim 1, having a function to generate and present a sales report to the user based on the user's instructions.

[1688] (Claim 3)

[1689] The system according to claim 1, having a function for automating data collection and preprocessing.

[1690] "Application Example 1"

[1691] (Claim 1)

[1692] Means of receiving instructions from users,

[1693] A means of interpreting and analyzing a specific task based on instructions,

[1694] A means for collecting and pre-processing the necessary data,

[1695] A means of using generative AI to execute a specified task and generate results,

[1696] A means of presenting the generated results to the user,

[1697] A means of creating an operational status report from the generated data,

[1698] A means including an interactive interface for displaying operational status reports,

[1699] ...a system that includes

[1700] (Claim 2)

[1701] The system according to claim 1, which has a function to generate sales reports and operational status reports based on user instructions and present them to the user.

[1702] (Claim 3)

[1703] The system according to claim 1, which has a function to automate data collection and preprocessing, and further has a function to display the collected data to the user in an interactive manner.

[1704] "Example 2 of combining an emotion engine"

[1705] (Claim 1)

[1706] Means of receiving instructions from users,

[1707] A means of interpreting and analyzing a specific task based on instructions,

[1708] A means for collecting and pre-processing the necessary data,

[1709] A means of using generative artificial intelligence to perform a specified task and generate results,

[1710] Means of recognizing user emotions,

[1711] Means of reflecting user emotions in the generated results and how they are presented,

[1712] A means of presenting the generated results to the user,

[1713] ...

[1714] A system that includes this.

[1715] (Claim 2)

[1716] The system according to claim 1, having a function to generate and present a sales report to the user based on the user's instructions.

[1717] (Claim 3)

[1718] The system according to claim 1, having a function for automating data collection and preprocessing.

[1719] "Application example 2 when combining with an emotional engine"

[1720] (Claim 1)

[1721] Means of receiving instructions from users,

[1722] A means of interpreting and analyzing a specific task based on instructions,

[1723] A means for collecting and pre-processing the necessary data,

[1724] A means of using generative AI to execute a specified task and generate results,

[1725] A means for recognizing the user's emotions and adjusting the method of presenting the results generated based on those emotions,

[1726] A means of presenting the generated results to the user,

[1727] A system that includes this.

[1728] (Claim 2)

[1729] The system according to claim 1, having a function to generate and present a sales report to the user based on the user's instructions.

[1730] (Claim 3)

[1731] The system according to claim 1, having a function for automating data collection and preprocessing. [Explanation of symbols]

[1732] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means of receiving instructions from users, A means of interpreting and analyzing a specific task based on instructions, A means for collecting and pre-processing the necessary data, A means of using generative AI to execute a specified task and generate results, A means of presenting the generated results to the user, A system that includes this.

2. The system according to claim 1, which has a function to generate a sales report based on user instructions and present it to the user.

3. The system according to claim 1, having a function for automating data collection and preprocessing.

Images