System

Abstract

A system is provided.SOLUTION: A system for controlling a data processing system by voice operation, comprising: means for inputting original data into the system; means for converting voice instructions obtained from voice input into text data; means for analyzing the text data to identify a plurality of tasks for data extraction, data processing, data analysis, and result development; means for extracting information based on conditions specified from the original data; means for processing the extracted information; means for analyzing the processed information; means for converting the analysis result into a specified format; and means for presenting the analysis result to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current data processing systems extract data by manual input, which has the problem that efficiency varies greatly depending on the skill level of the user, and each process of data extraction, processing, analysis, and deployment requires a lot of time. There is also a need for a method to streamline these processes and minimize the impact of differences in user skill. [Means for solving the problem]

[0005] The present invention relates to a system for controlling a data processing system through voice operation. This system includes means for inputting original data into the system, means for converting voice instructions obtained from the voice input into text data, means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result development, means for extracting information from the original data based on specified conditions, means for processing the extracted information, means for analyzing the processed information, means for converting the analysis results into a specified format, and means for presenting the analysis results to a user. This allows a user to efficiently perform complex data processing using only simple voice instructions.

[0006] A "data processing system" is a system that has a set of functions for inputting original data, analyzing voice instructions, extracting data, processing, analyzing, and developing the results.

[0007] "Original data" refers to the initial data that a user inputs into the system, such as data obtained from a CSV file or various databases.

[0008] "Voice input" refers to the speech a user conveys to the system through a microphone, which contains instructions that the system needs to process.

[0009] "Text data" is data obtained by analyzing voice input and converting it into text information.

[0010] A "natural language processing engine" is a technology that analyzes voice instructions and text data and identifies tasks for data extraction, processing, analysis, and deployment.

[0011] "Data extraction" is the process of extracting necessary information from original data based on specified conditions.

[0012] "Data processing" is the process of converting extracted data into a form that is easier to analyze, and specifically refers to filtering and data normalization.

[0013] "Data analysis" is the process of analyzing processed data and finding useful information and patterns from that data.

[0014] "Results development" is the process of converting the results of data analysis into a user-friendly format (e.g., graphs or reports) and presenting them to the user.

[0015] An "intermediate data store" is a storage device or storage area for temporarily storing data at each stage of data extraction, processing, and analysis. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system that controls a data processing system by voice operation, and allows a user to input original data into the system and give voice instructions to fully automatically extract, process, analyze, and develop the results of the data. This system is implemented by the following program processing.

[0038] System configuration

[0039] 1. Data Entry

[0040] The user uploads raw data (e.g., CSV file) such as sales data to the system.

[0041] The terminal receives the original data from the user and stores it in a temporary data store.

[0042] 2. Acquiring voice instructions

[0043] The user issues a voice command into the microphone, such as "Analyze sales data monthly."

[0044] The terminal captures audio from a microphone, converts the audio data into text format in real time, and sends it to a server.

[0045] 3. Analysis of voice instructions

[0046] The server passes the text data of the voice instructions to a natural language processing engine, which analyzes the content of the voice instructions. This analysis identifies multiple tasks: data extraction, data processing, data analysis, and result presentation.

[0047] Execution of each task

[0048] Data Extraction

[0049] The server retrieves the raw data from the temporary data store and extracts the required information based on voice instructions, such as sales data for a specified month or period.

[0050] Data Processing

[0051] The server processes the extracted data, specifically by using techniques such as filtering and normalization to improve the quality of the data.

[0052] Data analysis

[0053] The server analyzes the processed data, for example, aggregating extracted sales data monthly to identify trends and patterns.

[0054] Expanding the results

[0055] The server converts the analysis results into the specified format (e.g., graphs or reports).

[0056] The terminal presents the results sent from the server to the user, for example displaying bar graphs or reports showing sales trends.

[0057] Specific examples

[0058] 1. Inputting the original data

[0059] A user uploads a CSV file of sales data into the system.

[0060] The device will save this file and accept microphone operations.

[0061] 2. Enter voice commands

[0062] The user issues a voice command such as "Analyze sales data monthly."

[0063] The device converts this speech into text and sends it to the server.

[0064] 3. Analysis of voice instructions

[0065] The server analyzes the voice instructions and identifies tasks for data extraction, processing, analysis, and deployment.

[0066] 4. Execute the task

[0067] The server performs a series of data extraction, data processing, and data analysis, and generates the results as a graph.

[0068] The device displays the generated graphs to the user, providing a visual representation of monthly sales trends and patterns.

[0069] This system allows users to perform complex data processing with simple voice commands, enabling efficient data analysis regardless of the user's skill level. Furthermore, the automation of each task significantly reduces work time.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The user uploads a CSV file of sales data to the system.

[0073] The device verifies the uploaded file and saves it in a temporary data store.

[0074] Step 2:

[0075] The user issues a voice command into the microphone saying, "Analyze sales data monthly."

[0076] The device collects audio from a microphone and converts the audio data into text format.

[0077] Step 3:

[0078] The terminal transmits the converted text data to the server.

[0079] The server analyzes the received text data using a natural language processing (NLP) engine and identifies tasks, such as data extraction, data processing, data analysis, and results presentation.

[0080] Step 4:

[0081] The server retrieves the original data file from the temporary data store.

[0082] The server extracts specified information, such as monthly sales data, from the original data based on the analyzed voice instructions.

[0083] Step 5:

[0084] The server obtains the extracted data for data processing.

[0085] As part of the data processing, the server performs operations such as filtering and normalization to improve the quality of the data.

[0086] Step 6:

[0087] The server obtains the processed data for data analysis.

[0088] The server uses the obtained data to perform a specified analysis (e.g., monthly sales trend analysis) and identify trends and patterns in the data.

[0089] Step 7:

[0090] The server converts the analysis results into the specified format (e.g., graphs or reports).

[0091] The server transmits the converted result to the terminal.

[0092] Step 8:

[0093] The terminal displays the received analysis results to the user.

[0094] Users can view graphs and reports displayed on their devices to understand sales trends and patterns.

[0095] This series of steps allows users, regardless of their skill level, to efficiently perform complex data processing using voice instructions.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] Conventional data processing systems require users to manually perform numerous steps to extract, process, analyze, and develop results, often resulting in complex operations and requiring specialized knowledge. Furthermore, there are few fully automated voice-operated data processing systems, placing a significant burden on users. Furthermore, there is a lack of technology that can accurately analyze voice commands and efficiently execute each task automatically.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes means for converting voice instructions obtained from voice input into text data, means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result development, and means for processing the extracted information, thereby enabling the user to fully automatically extract, process, analyze, and develop results using only voice operations.

[0101] "Voice operation" is an operation method in which the user controls various functions of the system using voice.

[0102] A "data processing system" is a system for inputting, extracting, processing, analyzing, and expanding data.

[0103] "Original data" refers to the initial data entered into the system, including formats such as CSV files.

[0104] "Voice input" is a method in which a user supplies voice to the system using a microphone or the like.

[0105] "Voice instructions" are instructions given to the system by voice input.

[0106] "Text data" is data in the form of a character string converted from voice input.

[0107] "Analysis" is the act of understanding input data (here, text data) and grasping its meaning.

[0108] "Data extraction" is the process of extracting only the necessary information from the original data based on specific conditions.

[0109] "Data processing" is the process of filtering and normalizing extracted data to improve its quality and usefulness.

[0110] "Data analysis" is the process of performing calculations and aggregations on processed data to find useful information and patterns.

[0111] "Developing results" refers to summarizing the analysis results in an easy-to-read format and presenting them to the user.

[0112] "Speech recognition software" is software for converting voice data into text data.

[0113] A "natural language processing engine" is a software technology for analyzing text data and understanding the meaning of human language.

[0114] A "task" is a discrete operation performed by the system, such as data extraction, data processing, data analysis, and results presentation.

[0115] The present invention relates to a method and system for controlling a data processing system by voice operation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS ... will be described in detail below with reference to preferred embodiments thereof.

[0116] System configuration

[0117] Hardware used

[0118] 1. Terminal: A device operated by a user (such as a personal computer, smartphone, or tablet)

[0119] 2. Server: A computer for processing data

[0120] 3. Microphone: An input device for obtaining voice instructions

[0121] Software used

[0122] 1. Speech recognition software: Used to convert voice data into text data (e.g., Google Cloud Speech-to-Text, IBM Watson Speech to Text)

[0123] 2. Natural language processing engines: used to analyze text data (e.g., Google Cloud Natural Language, Microsoft Azure Text Analytics)

[0124] 3. Data processing tools: Programming languages ​​and libraries (e.g., Python, Pandas, NumPy) for processing the extracted data.

[0125] 4. Data analysis tools: Programming languages ​​and libraries for analyzing the processed data (e.g., Python, Scikit-learn, Matplotlib, Seaborn)

[0126] System Operation

[0127] User Operation

[0128] The user uploads the original data (e.g., a CSV file of sales data) from their terminal to the system. To do this, the user selects the file using the browser's file selection dialog and presses the upload button. Next, the user gives voice instructions into the microphone. For example, they can input instructions in voice format such as "Analyze the sales data monthly."

[0129] Device Features

[0130] The device receives files uploaded by users and temporarily stores them in local storage. The device also captures voice data from the microphone and converts it into text format in real time using voice recognition software. This converted text data is then sent to the server for further processing.

[0131] Server Processing

[0132] The server passes the text data sent from the speech recognition software to the natural language processing engine for analysis. Based on the analysis results, the server identifies the following tasks: data extraction, data processing, data analysis, and result distribution. The server executes these tasks sequentially to generate the final analysis results.

[0133] For example, in the data extraction stage, the server uses Python's Pandas library to read the original data CSV file and extract the specified monthly sales data. In the data processing stage, it fills in missing values ​​with zeros and filters specific columns. In the data analysis stage, it calculates the total sales for each month and analyzes the increase or decrease. In the final results development stage, it creates a graph using Matplotlib and saves it as an image file.

[0134] Presentation of results

[0135] The generated analysis results (e.g., graph images and reports) are sent from the server to the terminal. The terminal visually presents the results to the user. Specifically, graph images can be displayed on the browser.

[0136] Specific examples

[0137] Next, a specific example will be given.

[0138] 1. Upload the original data

[0139] A user uploads a file called sales_data.csv.

[0140] The device will save this file in the / tmp / uploads / directory.

[0141] 2. Enter voice commands

[0142] The user issues a voice command such as "Analyze sales data monthly."

[0143] The device converts the speech into text and sends it to the server.

[0144] 3. Analysis of voice instructions

[0145] The server analyzes the voice instructions and identifies tasks for data extraction, processing, analysis, and deployment.

[0146] 4. Task execution and results presentation

[0147] The server performs a series of data extraction, data processing, and data analysis, and generates the results as a graph.

[0148] The device displays the generated graphs to the user, providing a visual representation of monthly sales trends and patterns.

[0149] Prompt Sentence Examples

[0150] "Analyze sales data monthly."

[0151] "Compare sales trends this month and last month."

[0152] "What are the top 5 product sales figures?"

[0153] This allows users to easily perform advanced data analysis using only voice commands, enabling them to gain useful insights from large amounts of data and complete processes efficiently, even without specific skills.

[0154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0155] Step 1: Data entry

[0156] The user uploads a CSV file of sales data (e.g., sales_data.csv) to the system.

[0157] The terminal receives the CSV file uploaded by the user and temporarily stores it in local storage (e.g., / tmp / uploads / ).

[0158] Input: A CSV file selected by the user in the browser.

[0159] Output: CSV file saved in temporary directory.

[0160] Specific operation: The device reads the file selected from the browser's file selection dialog using the JavaScript FileReader object and saves it in a temporary storage directory.

[0161] Step 2: Getting voice instructions

[0162] The user issues a voice command into the microphone saying, "Analyze sales data monthly."

[0163] The device receives voice data from the microphone and converts it into text using real-time voice recognition software (e.g., Google Cloud Speech-to-Text), which then sends the text data to a server.

[0164] Input: User's voice command.

[0165] Output: Spoken instructions converted to text (e.g., "Analyze sales data monthly").

[0166] Specific operation: The device acquires microphone data using the navigator.mediaDevices.getUserMedia API, sends the audio data to the speech recognition API, and sends the converted text data to the server via WebSocket.

[0167] Step 3: Analyzing voice instructions

[0168] The server passes the text data obtained by speech recognition to a natural language processing engine (e.g., Google Cloud Natural Language) and analyzes the voice instructions.

[0169] Input: Text-based voice instructions.

[0170] Output: Parsed instructions (e.g. "Monthly analysis of sales data").

[0171] Specific operations: The server sends an API request to the natural language processing engine, receives the analysis results of the text data, and identifies tasks such as data extraction, data processing, data analysis, and result presentation.

[0172] Step 4: Data extraction

[0173] The server reads the original data from the temporarily saved CSV file and extracts the necessary information based on voice instructions.

[0174] Input: Temporarily saved CSV file, parsed instructions.

[0175] Output: The extracted data (e.g. sales data for a specified month).

[0176] Specific operation: The server uses Python's Pandas library to read the CSV file and extract sales data for the specified period (e.g., df[df['month'] == '2023-01']).

[0177] Step 5: Data processing

[0178] The server processes the extracted data, for example, by handling missing values, filtering, normalizing, etc.

[0179] Input: The extracted data.

[0180] Output: Processed data (e.g., sales data with missing values ​​filled with zeros and filtered).

[0181] Specific operation: The server uses the Pandas library to fill missing values ​​with zeros (e.g., df.fillna(0, inplace=True)), or extracts only specific columns to create a new data frame (e.g., df[['Sales', 'Month']]).

[0182] Step 6: Data analysis

[0183] The server then performs analytical processing on the processed data, such as aggregating monthly sales figures to identify trends and patterns.

[0184] Input: Processed data.

[0185] Output: Analysis results (e.g. monthly sales totals, sales increase / decrease).

[0186] Specific operation: The server uses Pandas and Scikit-learn to calculate the total sales for each month (e.g., df.groupby('Month')['Sales'].sum()) and calculates the increase or decrease in sales (e.g., df.pct_change()).

[0187] Step 7: Deploying the results

[0188] The server converts the analysis results into the specified format (e.g., graphs or reports). It uses Matplotlib to generate graphs and saves them as image files.

[0189] The terminal presents the results (e.g., graph images) sent from the server to the user.

[0190] Input: Analysis results.

[0191] Output: Visualized results (e.g., graph images, PDF reports).

[0192] Specific operation: The server creates a graph using the Matplotlib library (e.g., df.plot(kind='bar')) and saves it as an image (e.g., plt.savefig(' / tmp / results / sales_trend.png')). The terminal saves the generated image file as an HTML Present it to the user using tags.

[0193] (Application example 1)

[0194] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0195] In conventional data processing systems, the process of manually extracting, processing, and analyzing data is complex and requires a lot of time and effort. It is also difficult to properly manage inventory and issue shipping instructions in real time at logistics centers, resulting in a decrease in the efficiency of logistics operations. To solve these problems, the present invention aims to provide a system that automates logistics management by efficiently processing data through voice operation.

[0196] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0197] In this invention, the server includes means for inputting original data into the system to control the data processing system by voice operation, means for converting voice instructions obtained from the voice input into text data, and means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result deployment, thereby enabling the user to perform inventory management and shipping instructions based on voice instructions and receive feedback in real time.

[0198] "Voice operation" refers to an operation in which a user gives instructions to a system using a voice input device such as a microphone.

[0199] A "data processing system" is a computer system that performs a series of processes, from inputting original data to receiving and analyzing voice instructions, executing tasks, and presenting results.

[0200] "Original data" refers to the initial data (e.g., CSV file) provided by the user for processing by the system.

[0201] A "voice instruction" is a specific command or request that the user issues into the microphone (e.g., "Analyze sales data monthly").

[0202] "Text data" refers to data in the form of a character string obtained by converting voice input.

[0203] A "task" is a specific unit of processing performed by the system, such as data extraction, data processing, data analysis, and result deployment.

[0204] "Data extraction" is the process of extracting necessary information from original data based on specified conditions.

[0205] "Data processing" refers to the process of organizing extracted information using methods such as filtering and normalization to improve its quality.

[0206] "Data analysis" is the process of finding trends and patterns based on processed information.

[0207] "Result development" refers to the process of converting the analysis results into a specified format (e.g., graph, report, etc.) and presenting them to the user.

[0208] "Feedback" refers to providing the system with real-time responses and results to user-generated voice commands.

[0209] "Logistics management means" refers to functions that allow inventory management and shipping instructions to be performed based on voice commands, and provide feedback in real time.

[0210] The system of the present invention is designed to efficiently perform data processing and logistics management through voice operations. This system is realized by the following configuration and processing.

[0211] System configuration

[0212] 1. Device:

[0213] Hardware: Smartphones and tablets with microphones

[0214] Software: Speech recognition library (e.g., SpeechRecognition)

[0215] 2. Server:

[0216] Hardware: Cloud-based server infrastructure

[0217] Software: Natural language processing engine, data processing program (e.g., Pandas, Matplotlib)

[0218] 3. Data Store:

[0219] A database where raw data (e.g., inventory data, shipping data) is stored

[0220] Program processing

[0221] The user inputs voice instructions using a microphone into the terminal, such as "Tell me the inventory status" or "Ship 50 units of product A" that are related to logistics management.

[0222] The device receives this voice instruction and converts it into text data using a speech recognition library, which is then sent to the server as a request.

[0223] The server passes the received text data to a natural language processing engine for analysis. This analysis identifies multiple tasks, such as data extraction, data processing, and data analysis. For example, from an instruction such as "Ship 50 units of product A," updating inventory data and generating shipping instructions are identified as tasks.

[0224] The server extracts the necessary data from the database based on the specified task, then performs processing such as filtering and normalization on the extracted data, and analyzes the processed data to generate the results.

[0225] The generated results are converted into a specified format (e.g., inventory list, shipping report, graph, etc.) and sent to the terminal.

[0226] The terminal displays the results sent from the server to the user, allowing the user to receive feedback on inventory status and shipping instructions in real time.

[0227] Examples of specific examples and prompts

[0228] Examples:

[0229] The user issues a voice command to their smartphone saying, "Ship 50 units of product A."

[0230] The system converts this speech into text and analyzes it on the server.

[0231] The server updates inventory data and generates shipping instructions.

[0232] The terminal provides real-time feedback to the user, such as "Shipping instructions: 50 units of product A have been shipped."

[0233] Example prompt sentence:

[0234] User: "Ship 50 units of product A."

[0235] Example of system response: "Shipping instructions: 50 units of product A have been shipped."

[0236] As described above, the system of the present invention enables efficient data processing and logistics management through voice operation, and provides real-time feedback.

[0237] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0238] Step 1:

[0239] The user inputs voice instructions using a microphone, which include specific commands or requests (e.g., "Ship 50 units of product A").

[0240] Input: User's voice data

[0241] Output: Audio file captured from microphone

[0242] Step 2:

[0243] The device uses a speech recognition library to convert the captured voice data into text data in real time. This conversion process is performed using a speech recognition engine (e.g., SpeechRecognition).

[0244] Input: Audio file

[0245] Output: Text-based voice instructions (e.g., "Ship 50 units of product A")

[0246] Step 3:

[0247] The terminal sends the converted text data to the server as a request in the form of an HTTP request.

[0248] Input: Text-based voice instructions

[0249] Output: HTTP request

[0250] Step 4:

[0251] The server passes the received text data to a natural language processing engine for analysis. The analysis identifies tasks such as data extraction, data processing, and data analysis. Specifically, from the command "Ship 50 units of product A," it identifies the tasks of updating inventory data and generating shipping instructions.

[0252] Input: Text-based voice instructions

[0253] Output: Identified tasks (e.g., updating inventory data, generating shipping instructions)

[0254] Step 5:

[0255] The server extracts the required data from the database based on the specified task, using SQL queries.

[0256] Input: Task Instructions

[0257] Output: Extracted data (e.g. inventory data)

[0258] Step 6:

[0259] The server processes the extracted data, such as filtering and normalization, using a data frame manipulation library (e.g., Pandas).

[0260] Input: Extracted data

[0261] Output: Processed data

[0262] Step 7:

[0263] The server analyzes the processed data and generates results, where data analysis tools (e.g., Matplotlib) are used.

[0264] Input: Processed data

[0265] Output: Analysis results (e.g. shipping instructions)

[0266] Step 8:

[0267] The server converts the generated results into the specified format (e.g., report, graph, etc.) using a data visualization library (e.g., Matplotlib).

[0268] Input: Analysis results

[0269] Output: Results in the specified format (e.g. graph, report)

[0270] Step 9:

[0271] The server transmits the generated results to the terminal.

[0272] Input: Result in the specified format

[0273] Output: HTTP response

[0274] Step 10:

[0275] The terminal then presents the received results to the user, who can receive real-time feedback on inventory information and shipping instructions.

[0276] Input: HTTP response (result in the specified format)

[0277] Output: Feedback displayed on the user screen (e.g., "Shipping instructions: 50 units of product A have been shipped.")

[0278] This process enables efficient data processing and logistics management through voice control, providing real-time feedback.

[0279] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0280] The present invention combines a voice-operated data processing system with an emotion engine, and is a system that realizes more intuitive and effective data processing by recognizing the user's emotional state and automatically adjusting data processing and providing feedback based on that state.

[0281] System configuration

[0282] 1. Data Entry

[0283] The user uploads raw data (e.g., CSV file) such as sales data to the system.

[0284] The terminal receives the original data from the user and stores it in a temporary data store.

[0285] 2. Acquiring voice instructions

[0286] The user issues a voice command into the microphone, such as "Analyze sales data monthly."

[0287] The device collects audio from a microphone and converts the audio data into text format.

[0288] 3. Voice command analysis and emotion recognition

[0289] The device uses an emotion engine to analyze the user's emotions based on the converted text data as well as the voice data.

[0290] The server passes the received text data and the emotional state information obtained from the emotion engine to the natural language processing (NLP) engine, and identifies tasks, including data extraction, data processing, data analysis, and result deployment.

[0291] Execution of each task

[0292] Data Extraction

[0293] The server retrieves the original data file from the temporary data store and extracts the required information based on voice instructions, such as sales data for a specified month or period.

[0294] Data Processing

[0295] The server retrieves the extracted data for processing, specifically, performing operations such as filtering and normalization to improve the data quality.

[0296] Data analysis

[0297] The server retrieves the processed data for data analysis, for example, aggregating the extracted sales data monthly to identify trends and patterns.

[0298] Expanding the results

[0299] The server converts the analysis results into the specified format (e.g., graphs or reports).

[0300] The terminal presents the results sent from the server to the user.

[0301] Applications of Emotion Recognition

[0302] Once the emotion engine has analyzed the user's emotions, it can adjust the system's behavior based on the resulting emotional state as follows:

[0303] If the emotion engine recognizes that the user is feeling stressed, the server will provide the results of the data processing in a more intuitive and easy-to-understand format (for example, more concise graphs or reports explained in simpler language).

[0304] If the emotion engine recognizes that the user is happy, the device will display more detailed data analysis results and in-depth analysis results.

[0305] The server responds to the voice prompts by providing appropriate feedback to the user and suggesting the next action (e.g., "Would you like to perform a more detailed analysis?").

[0306] Specific examples

[0307] 1. The user issues a voice command to the system saying, "Analyze sales data monthly," and uploads the original data.

[0308] 2. The device receives the voice instructions and the original data, converts the voice into text, and sends it to the server.

[0309] 3. The device uses an emotion engine to analyze the user's emotions and recognize, for example, that the user is feeling anxious.

[0310] 4. Based on the voice instructions and emotional data received by the server, data is extracted, processed, and analyzed, and a report is generated that highlights only the most important points for impatient users.

[0311] 5. The terminal displays the generated report to the user, who can then view the report in an easy-to-understand format.

[0312] The system is able to take into account the user's emotional state and provide more responsive and user-friendly data processing and feedback.

[0313] The processing flow will be explained below.

[0314] Step 1:

[0315] The user uploads a CSV file of sales data to the system.

[0316] The device verifies the uploaded file and saves it in a temporary data store.

[0317] Step 2:

[0318] The user issues a voice command into the microphone saying, "Analyze sales data monthly."

[0319] The device collects audio from a microphone and converts the audio data into text format.

[0320] Step 3:

[0321] The terminal inputs the voice data to the emotion engine together with the voice instruction converted into text data.

[0322] The emotion engine analyzes the audio data to identify the user's emotional state (e.g., impatience, joy, stress, etc.).

[0323] The terminal transmits the voice instruction in text format and the analysis results of the emotion engine to the server.

[0324] Step 4:

[0325] The server analyzes the received text data and emotional state using a natural language processing (NLP) engine.

[0326] The server identifies tasks based on the voice instructions, including "data extraction," "data processing," "data analysis," and "result deployment."

[0327] Step 5:

[0328] The server retrieves the original data file from the temporary data store.

[0329] The server extracts monthly sales data from the original data based on the analyzed voice instructions.

[0330] Step 6:

[0331] The server processes the extracted data, specifically by filtering, normalizing, and other operations to improve the quality of the data.

[0332] Step 7:

[0333] The server analyzes the processed data, for example by aggregating extracted sales data monthly to identify trends and patterns.

[0334] The server adjusts the analysis method and the result display format based on the user's emotional state.

[0335] Step 8:

[0336] The server converts the analysis results into the specified format (e.g., graphs or reports).

[0337] The server transmits the converted result to the terminal.

[0338] Step 9:

[0339] The device then displays the received analysis results to the user. For example, if the device detects that the user is in a hurry, it may present the results in a more concise and visually understandable format.

[0340] The device provides additional feedback and action suggestions (e.g., "Would you like to conduct a more detailed analysis?") depending on the user's emotions.

[0341] This series of steps allows users to experience responsive data processing and feedback that utilizes emotion recognition.

[0342] Example 2

[0343] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0344] Conventional voice-operated data processing systems have a limited scope for automating data processing based on voice instructions, and they are unable to provide feedback according to the user's emotional state. As a result, the system's response may be inappropriate when the user is stressed or when more detailed information is required.

[0345] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to input original data into the system, a means for converting voice instructions obtained from voice input into text data, a means for analyzing the converted text data and voice parameters and recognizing the user's emotional state, and a means for analyzing the text data and emotional data and identifying multiple tasks for data extraction, data processing, data analysis, and result development. This enables highly responsive data processing and feedback according to the user's emotional state.

[0346] A "user" is an entity that operates the system and inputs voice instructions and data.

[0347] "Original data" refers to the underlying information, such as sales data, and refers to the data files uploaded to the system.

[0348] "Voice input" refers to voice instructions given by the user through a microphone.

[0349] "Text data" refers to speech input converted into text format.

[0350] "Voice parameters" refer to features and information extracted from voice data, and are data that can be used for emotion analysis.

[0351] "Emotional state" is information indicating the user's emotions, such as impatience, joy, or stress.

[0352] "Data extraction" refers to the process of extracting necessary information from original data based on specified conditions.

[0353] "Data processing" refers to the process of filtering, normalizing, and other processes performed on extracted data to improve its quality.

[0354] "Data analysis" refers to the analytical process of using processed data to identify trends and patterns.

[0355] "Result development" refers to the process of converting the analysis results into a specified format (e.g., graphs or reports) and presenting them to the user.

[0356] "Feedback" refers to information or suggestions provided to users based on the results of data processing and analysis.

[0357] The present invention combines a voice-operated data processing system with an emotion engine, and is a system that realizes more intuitive and effective data processing by recognizing the user's emotional state and automatically adjusting data processing and providing feedback based on that state.

[0358] First, the user uploads raw data, such as sales data, to the system. The raw data is typically in CSV file format, and the device stores it in a temporary data store. The user uses a microphone to provide voice input. Voice instructions can be simple commands such as "Analyze sales data monthly." The device collects voice from the microphone and converts the voice data into text format using voice recognition software (e.g., a voice recognition API). Along with this converted text data, voice parameters are also obtained.

[0359] Next, the device analyzes the user's emotional state using a sentiment analysis engine (e.g., a sentiment analysis API). The analyzed emotional information is sent to a server, where a natural language processing (NLP) engine (e.g., an NLP model) analyzes the received text data and emotional data and identifies tasks. The identified tasks include "data extraction," "data processing," "data analysis," and "result deployment."

[0360] The server retrieves raw data from the temporary data store and extracts the required information based on voice commands. For example, sales data for a specified month or period may be extracted. The extracted data undergoes data processing operations such as filtering and normalization, and is then analyzed. For example, the analysis may aggregate sales data monthly to identify trends and patterns. The processed and analyzed data is then converted into a specified format (for example, a graph or report).

[0361] The server sends the calculated analysis results to the device, which then provides more appropriate feedback based on the user's emotional state as recognized by the emotion engine. For example, the device might display the results of data processing in an intuitive and easy-to-understand format to a stressed user, or display more detailed data and in-depth analysis results to a happy user. The server also provides appropriate feedback in response to voice commands and suggests the next course of action.

[0362] As a concrete example, consider a scenario in which a user issues a voice command to the system to "analyze monthly sales data" and uploads the original data. The device receives the voice command and the original data, converts the voice into text, and sends it to the server. The device then uses an emotion engine to analyze the user's emotions and recognizes, for example, that the user is feeling anxious. The server extracts, processes, and analyzes data based on the received voice command and emotional data, and generates a report that highlights only the most important points for anxious users. The device then displays the generated report to the user, who can view it in an easy-to-understand format.

[0363] Examples of prompt sentences include the following:

[0364] "The user issues a voice command to the system saying, 'Analyze monthly sales data,' and uploads the original data. The device receives the voice command and the original data, converts the voice into text, and sends it to the server. The device uses an emotion engine to analyze the user's emotions and recognizes, for example, that the user is feeling anxious. The server extracts, processes, and analyzes the data based on the received voice command and emotional data, and generates a report that highlights only the most important points for anxious users."

[0365] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0366] Step 1: User uploads data

[0367] Users upload raw data, such as sales data, to the system interface by dragging and dropping it. The input is a CSV-formatted data file, which the terminal saves in a temporary data store. The output is a CSV file saved in the temporary data store. The specific operation is to upload the file using a browser.

[0368] Step 2: The device receives voice instructions

[0369] The user speaks into the microphone, saying, "Analyze sales data monthly." The input is the user's voice command, and the device picks up this voice through the built-in microphone. The output is voice data (audio file). Specifically, the device uses the microphone to capture the voice.

[0370] Step 3: Your device converts your voice commands to text

[0371] The device uses voice recognition software (e.g., voice recognition API) to convert the voice data into text format. The input is voice data, and the output is text data such as "Analyze sales data monthly." Specifically, the device calls the voice recognition API and converts the voice data into text.

[0372] Step 4: The device performs sentiment analysis

[0373] The device passes the text data and voice parameters to an emotion analysis engine (e.g., emotion analysis API) to analyze the user's emotional state. The input is text data and voice parameters, and the output is the user's emotional state (e.g., impatience, joy). Specifically, the device calls the emotion analysis API and performs emotion analysis.

[0374] Step 5: Server identifies the task

[0375] The server passes text and sentiment data to a natural language processing (NLP) engine (e.g., an NLP model) to identify a task. The input is text and sentiment data, and the output is the identified task (e.g., data extraction, data processing, data analysis, or result deployment). Specifically, the server uses an NLP model to analyze the text and sentiment and identify the task.

[0376] Step 6: The server performs the data extraction

[0377] The server retrieves the raw data (CSV files) from a temporary data store and extracts the required information based on voice instructions. The input is the raw data and voice instructions, and the output is the extracted information (e.g., sales data for a specified month). Specifically, it uses a database query to extract data that matches the criteria.

[0378] Step 7: The server performs the data processing

[0379] The server processes the extracted data by filtering and normalizing it. The input is the extracted information, and the output is the processed information (e.g., clean data). Specifically, the program complements missing values ​​in the data and removes outliers.

[0380] Step 8: The server performs the data analysis

[0381] The server uses the processed data to perform analysis to identify trends and patterns. The input is the processed information, and the output is the analytical results (e.g., monthly sales trends). Specifically, it runs analytical algorithms and generates statistical information on the data.

[0382] Step 9: The server extracts the results

[0383] The server converts the analysis results into the specified format (e.g., graphs or reports) and sends them to the terminal. The input is the analysis results, and the output is the converted results (e.g., graphs, reports). Specifically, the results are visualized using a data visualization tool.

[0384] Step 10: Your device will display the results

[0385] The terminal displays the results sent from the server to the user. The input is the converted result, and the output is a screen display that is visually presented to the user. Specifically, it displays graphs and reports on the screen so that the user can check them.

[0386] Step 11: The server and device adjust according to the emotion.

[0387] The server adjusts the data feedback based on the emotional state recognized by the emotion engine, and the device displays the results in a format appropriate for the user. The input is the emotional state and the resulting data, and the output is adjusted feedback (e.g., concise display, detailed data). Specifically, the display content is dynamically changed according to the user's emotional state.

[0388] (Application example 2)

[0389] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0390] Today, users use voice commands to perform various data processing tasks. However, conventional systems do not take into account the user's emotional state, resulting in a lack of an intuitive and personal operating experience. As a result, users can experience inconvenience and stress when browsing and processing data. By solving this problem, we aim to improve user satisfaction and the effectiveness of data processing.

[0391] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting original data to the system to control the data processing system by voice operation, means for converting voice instructions obtained from the voice input into text data, means for analyzing the user's emotional state from the text data and voice data, means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result development, means for extracting information from the original data based on specified conditions, means for processing the extracted information, means for analyzing the processed information, means for converting information based on the analysis results and the user's emotional state into a specified format, and means for presenting appropriate feedback to the user based on the analysis results and the user's emotional state. This enables personalized data processing and feedback that takes the user's emotional state into consideration.

[0392] "Voice operation" is a technology that controls a system and executes instructions through user voice input.

[0393] A "data processing system" is a system that extracts, processes, and analyzes various types of data based on input data.

[0394] "Raw data" refers to the original, unprocessed data that is input into the system.

[0395] "Voice instructions" refer to instructions or commands given by the user through voice.

[0396] "Text data" is data in character format converted based on voice instructions.

[0397] An "emotional state" is a psychological state inferred from a user's voice or other input.

[0398] A "task" is a specific process or operation that a system performs.

[0399] "Data extraction" refers to extracting necessary information from original data based on specified conditions.

[0400] "Data processing" is the act of formatting or converting extracted data to make it easier to use.

[0401] "Data analysis" is the process of using processed data to find meaningful patterns and trends.

[0402] "Result presentation" refers to presenting the results of data analysis in a user-friendly format.

[0403] "Feedback" refers to the reaction or information that a system provides to a user.

[0404] A "natural language processing engine" is a technology for analyzing text data and understanding its meaning and context.

[0405] An "emotion engine" is a technology for analyzing a user's emotional state from their voice and data.

[0406] The present invention is a system that combines a voice-operated data processing system with an emotion engine, which recognizes the user's emotional state and provides automatic adjustments and feedback to data processing based on that state. The system of the present invention is implemented using the following hardware and software.

[0407] Hardware and software used

[0408] Hardware

[0409] Smartphones (e.g. iPhone, Android devices)

[0410] microphone

[0411] server

[0412] software

[0413] Google Cloud Speech-to-Text API

[0414] IBM Watson Tone Analyzer

[0415] Google Natural Language API

[0416] Food delivery backend server

[0417] System configuration

[0418] 1. Acquiring voice instructions

[0419] The user speaks into the smartphone's microphone, saying something like "I want to order a pizza." This voice input is captured by the smartphone. The captured voice data is converted into text data using the Google Cloud Speech-to-Text API.

[0420] 2. Emotion analysis

[0421] The converted text data and the original voice data are then analyzed using IBM Watson Tone Analyzer to determine the user's emotional state, for example, whether the user is relaxed or anxious.

[0422] 3. Analysis of voice instructions

[0423] The parsed text data is then passed through the Google Natural Language API to identify the tasks the system should perform (e.g., display a menu, process an order), allowing it to optimally respond based on the user's emotional state and voice commands.

[0424] 4. Food delivery order fulfillment

[0425] The food delivery backend server retrieves menu information from a database based on the user's emotional state and voice instructions, and sends it to the smartphone. For example, if the user is tired, a simple menu is displayed, and if the user is relaxed, a detailed menu is displayed.

[0426] 5. Personalized feedback

[0427] Feedback that takes into account the user's emotional state is provided via smartphone, such as suggestions like "Today's recommendation is Margherita pizza."

[0428] Specific examples

[0429] A user speaks to their smartphone and says, "I'd like to order a pizza." The smartphone captures the voice data and converts it to text using the Google Cloud Speech-to-Text API. The system then uses IBM Watson Tone Analyzer to analyze the user's emotional state. For example, if the system detects that the user is tired, it might suggest, "How about a Margherita pizza?"

[0430] Prompt Sentence Examples

[0431] "If a user says, 'I want to order a pizza':

[0432] "User wants to order pizza. Emotional state is relaxed."

[0433] "Yes, this is the menu we recommend for users in a relaxed state."

[0434] The system allows users to have a personalized ordering experience that is tailored to their emotional state, resulting in increased user satisfaction and a more intuitive experience.

[0435] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0436] Step 1:

[0437] The user speaks into the smartphone's microphone. For example, they say, "I'd like to order a pizza." The input here is the user's voice, and the output is the voice data captured by the smartphone. The smartphone sends this voice data to the Google Cloud Speech-to-Text API, which converts it into text.

[0438] Step 2:

[0439] The smartphone uses the Google Cloud Speech-to-Text API to convert the acquired voice data into text data. The converted text is in the form of "I would like to order a pizza." The input of this process is voice data, and the output is text data.

[0440] Step 3:

[0441] The smartphone uses IBM Watson Tone Analyzer to analyze the user's emotional state from the text data and the original voice data. For example, it determines whether the user is relaxed or tired. The input is text data and voice data, and the output is data indicating the user's emotional state.

[0442] Step 4:

[0443] The smartphone sends text data and emotion data to the Google Natural Language API for analysis. The analyzed data is used to identify the tasks the system should perform (e.g., display a menu or process an order). The inputs for this step are text data and emotion data, and the output is the identified tasks.

[0444] Step 5:

[0445] The server accesses the food delivery backend system and retrieves menu information from the database based on the user's emotional state and voice instructions. For example, if the user is tired, a concise menu is displayed. The input is the specified task and the user's emotional state, and the output is the adjusted menu information.

[0446] Step 6:

[0447] The server sends the adjusted menu information to the smartphone, which then displays this information to the user, for example, "Today's special is Margherita pizza." The input is the adjusted menu information, and the output is the user's visual feedback.

[0448] Step 7:

[0449] The user makes a selection from the displayed menu and confirms the order by voice, for example, "I'd like to order a Margherita pizza." The smartphone converts this speech to text, again using the Google Cloud Speech-to-Text API. The input is the user's voice, and the output is text data.

[0450] Step 8:

[0451] The smartphone sends a final confirmation text to the food delivery backend system to confirm the order. The server receives the order and starts processing it. The input is the order confirmation text, and the output is the food delivery order data.

[0452] This process allows users to have a personalized ordering experience that is tailored to their emotional state, resulting in increased user satisfaction and a more intuitive experience.

[0453] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0454] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0455] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0456] [Second embodiment]

[0457] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0458] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0459] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0460] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0461] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0462] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0463] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0464] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0465] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0466] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0467] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0468] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0469] The present invention is a system that controls a data processing system by voice operation, and allows a user to input original data into the system and give voice instructions to fully automatically extract, process, analyze, and develop the results of the data. This system is implemented by the following program processing.

[0470] System configuration

[0471] 1. Data Entry

[0472] The user uploads raw data (e.g., CSV file) such as sales data to the system.

[0473] The terminal receives the original data from the user and stores it in a temporary data store.

[0474] 2. Acquiring voice instructions

[0475] The user issues a voice command into the microphone, such as "Analyze sales data monthly."

[0476] The terminal captures audio from a microphone, converts the audio data into text format in real time, and sends it to a server.

[0477] 3. Analysis of voice instructions

[0478] The server passes the text data of the voice instructions to a natural language processing engine, which analyzes the content of the voice instructions. This analysis identifies multiple tasks: data extraction, data processing, data analysis, and result presentation.

[0479] Execution of each task

[0480] Data Extraction

[0481] The server retrieves the raw data from the temporary data store and extracts the required information based on voice instructions, such as sales data for a specified month or period.

[0482] Data Processing

[0483] The server processes the extracted data, specifically by using techniques such as filtering and normalization to improve the quality of the data.

[0484] Data analysis

[0485] The server analyzes the processed data, for example, aggregating extracted sales data monthly to identify trends and patterns.

[0486] Expanding the results

[0487] The server converts the analysis results into the specified format (e.g., graphs or reports).

[0488] The terminal presents the results sent from the server to the user, for example displaying bar graphs or reports showing sales trends.

[0489] Specific examples

[0490] 1. Inputting the original data

[0491] A user uploads a CSV file of sales data into the system.

[0492] The device will save this file and accept microphone operations.

[0493] 2. Enter voice commands

[0494] The user issues a voice command such as "Analyze sales data monthly."

[0495] The device converts this speech into text and sends it to the server.

[0496] 3. Analysis of voice instructions

[0497] The server analyzes the voice instructions and identifies tasks for data extraction, processing, analysis, and deployment.

[0498] 4. Execute the task

[0499] The server performs a series of data extraction, data processing, and data analysis, and generates the results as a graph.

[0500] The device displays the generated graphs to the user, providing a visual representation of monthly sales trends and patterns.

[0501] This system allows users to perform complex data processing with simple voice commands, enabling efficient data analysis regardless of the user's skill level. Furthermore, the automation of each task significantly reduces work time.

[0502] The processing flow will be explained below.

[0503] Step 1:

[0504] The user uploads a CSV file of sales data to the system.

[0505] The device verifies the uploaded file and saves it in a temporary data store.

[0506] Step 2:

[0507] The user issues a voice command into the microphone saying, "Analyze sales data monthly."

[0508] The device collects audio from a microphone and converts the audio data into text format.

[0509] Step 3:

[0510] The terminal transmits the converted text data to the server.

[0511] The server analyzes the received text data using a natural language processing (NLP) engine and identifies tasks, such as data extraction, data processing, data analysis, and results presentation.

[0512] Step 4:

[0513] The server retrieves the original data file from the temporary data store.

[0514] The server extracts specified information, such as monthly sales data, from the original data based on the analyzed voice instructions.

[0515] Step 5:

[0516] The server obtains the extracted data for data processing.

[0517] As part of the data processing, the server performs operations such as filtering and normalization to improve the quality of the data.

[0518] Step 6:

[0519] The server obtains the processed data for data analysis.

[0520] The server uses the obtained data to perform a specified analysis (e.g., monthly sales trend analysis) and identify trends and patterns in the data.

[0521] Step 7:

[0522] The server converts the analysis results into the specified format (e.g., graphs or reports).

[0523] The server transmits the converted result to the terminal.

[0524] Step 8:

[0525] The terminal displays the received analysis results to the user.

[0526] Users can view graphs and reports displayed on their devices to understand sales trends and patterns.

[0527] This series of steps allows users, regardless of their skill level, to efficiently perform complex data processing using voice instructions.

[0528] Example 1

[0529] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0530] Conventional data processing systems require users to manually perform numerous steps to extract, process, analyze, and develop results, often resulting in complex operations and requiring specialized knowledge. Furthermore, there are few fully automated voice-operated data processing systems, placing a significant burden on users. Furthermore, there is a lack of technology that can accurately analyze voice commands and efficiently execute each task automatically.

[0531] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0532] In this invention, the server includes means for converting voice instructions obtained from voice input into text data, means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result development, and means for processing the extracted information, thereby enabling the user to fully automatically extract, process, analyze, and develop results using only voice operations.

[0533] "Voice operation" is an operation method in which the user controls various functions of the system using voice.

[0534] A "data processing system" is a system for inputting, extracting, processing, analyzing, and expanding data.

[0535] "Original data" refers to the initial data entered into the system, including formats such as CSV files.

[0536] "Voice input" is a method in which a user supplies voice to the system using a microphone or the like.

[0537] "Voice instructions" are instructions given to the system by voice input.

[0538] "Text data" is data in the form of a character string converted from voice input.

[0539] "Analysis" is the act of understanding input data (here, text data) and grasping its meaning.

[0540] "Data extraction" is the process of extracting only the necessary information from the original data based on specific conditions.

[0541] "Data processing" is the process of filtering and normalizing extracted data to improve its quality and usefulness.

[0542] "Data analysis" is the process of performing calculations and aggregations on processed data to find useful information and patterns.

[0543] "Developing results" refers to summarizing the analysis results in an easy-to-read format and presenting them to the user.

[0544] "Speech recognition software" is software for converting voice data into text data.

[0545] A "natural language processing engine" is a software technology for analyzing text data and understanding the meaning of human language.

[0546] A "task" is a discrete operation performed by the system, such as data extraction, data processing, data analysis, and results presentation.

[0547] The present invention relates to a method and system for controlling a data processing system by voice operation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS ... will be described in detail below with reference to preferred embodiments thereof.

[0548] System configuration

[0549] Hardware used

[0550] 1. Terminal: A device operated by a user (such as a personal computer, smartphone, or tablet)

[0551] 2. Server: A computer for processing data

[0552] 3. Microphone: An input device for obtaining voice instructions

[0553] Software used

[0554] 1. Speech recognition software: Used to convert voice data into text data (e.g., Google Cloud Speech-to-Text, IBM Watson Speech to Text)

[0555] 2. Natural language processing engines: used to analyze text data (e.g., Google Cloud Natural Language, Microsoft Azure Text Analytics)

[0556] 3. Data processing tools: Programming languages ​​and libraries (e.g., Python, Pandas, NumPy) for processing the extracted data.

[0557] 4. Data analysis tools: Programming languages ​​and libraries for analyzing the processed data (e.g., Python, Scikit-learn, Matplotlib, Seaborn)

[0558] System Operation

[0559] User Operation

[0560] The user uploads the original data (e.g., a CSV file of sales data) from their terminal to the system. To do this, the user selects the file using the browser's file selection dialog and presses the upload button. Next, the user gives voice instructions into the microphone. For example, they can input instructions in voice format such as "Analyze the sales data monthly."

[0561] Device Features

[0562] The device receives files uploaded by users and temporarily stores them in local storage. The device also captures voice data from the microphone and converts it into text format in real time using voice recognition software. This converted text data is then sent to the server for further processing.

[0563] Server Processing

[0564] The server passes the text data sent from the speech recognition software to the natural language processing engine for analysis. Based on the analysis results, the server identifies the following tasks: data extraction, data processing, data analysis, and result distribution. The server executes these tasks sequentially to generate the final analysis results.

[0565] For example, in the data extraction stage, the server uses Python's Pandas library to read the original data CSV file and extract the specified monthly sales data. In the data processing stage, it fills in missing values ​​with zeros and filters specific columns. In the data analysis stage, it calculates the total sales for each month and analyzes the increase or decrease. In the final results development stage, it creates a graph using Matplotlib and saves it as an image file.

[0566] Presentation of results

[0567] The generated analysis results (e.g., graph images and reports) are sent from the server to the terminal. The terminal visually presents the results to the user. Specifically, graph images can be displayed on the browser.

[0568] Specific examples

[0569] Next, a specific example will be given.

[0570] 1. Upload the original data

[0571] A user uploads a file called sales_data.csv.

[0572] The device will save this file in the / tmp / uploads / directory.

[0573] 2. Enter voice commands

[0574] The user issues a voice command such as "Analyze sales data monthly."

[0575] The device converts the speech into text and sends it to the server.

[0576] 3. Analysis of voice instructions

[0577] The server analyzes the voice instructions and identifies tasks for data extraction, processing, analysis, and deployment.

[0578] 4. Task execution and results presentation

[0579] The server performs a series of data extraction, data processing, and data analysis, and generates the results as a graph.

[0580] The device displays the generated graphs to the user, providing a visual representation of monthly sales trends and patterns.

[0581] Prompt Sentence Examples

[0582] "Analyze sales data monthly."

[0583] "Compare sales trends this month and last month."

[0584] "What are the top 5 product sales figures?"

[0585] This allows users to easily perform advanced data analysis using only voice commands, enabling them to gain useful insights from large amounts of data and complete processes efficiently, even without specific skills.

[0586] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0587] Step 1: Data entry

[0588] The user uploads a CSV file of sales data (e.g., sales_data.csv) to the system.

[0589] The terminal receives the CSV file uploaded by the user and temporarily stores it in local storage (e.g., / tmp / uploads / ).

[0590] Input: A CSV file selected by the user in the browser.

[0591] Output: CSV file saved in temporary directory.

[0592] Specific operation: The device reads the file selected from the browser's file selection dialog using the JavaScript FileReader object and saves it in a temporary storage directory.

[0593] Step 2: Getting voice instructions

[0594] The user issues a voice command into the microphone saying, "Analyze sales data monthly."

[0595] The device receives voice data from the microphone and converts it into text using real-time voice recognition software (e.g., Google Cloud Speech-to-Text), which then sends the text data to a server.

[0596] Input: User's voice command.

[0597] Output: Spoken instructions converted to text (e.g., "Analyze sales data monthly").

[0598] Specific operation: The device acquires microphone data using the navigator.mediaDevices.getUserMedia API, sends the audio data to the speech recognition API, and sends the converted text data to the server via WebSocket.

[0599] Step 3: Analyzing voice instructions

[0600] The server passes the text data obtained by speech recognition to a natural language processing engine (e.g., Google Cloud Natural Language) and analyzes the voice instructions.

[0601] Input: Text-based voice instructions.

[0602] Output: Parsed instructions (e.g. "Monthly analysis of sales data").

[0603] Specific operations: The server sends an API request to the natural language processing engine, receives the analysis results of the text data, and identifies tasks such as data extraction, data processing, data analysis, and result presentation.

[0604] Step 4: Data extraction

[0605] The server reads the original data from the temporarily saved CSV file and extracts the necessary information based on voice instructions.

[0606] Input: Temporarily saved CSV file, parsed instructions.

[0607] Output: The extracted data (e.g. sales data for a specified month).

[0608] Specific operation: The server uses Python's Pandas library to read the CSV file and extract sales data for the specified period (e.g., df[df['month'] == '2023-01']).

[0609] Step 5: Data processing

[0610] The server processes the extracted data, for example, by handling missing values, filtering, normalizing, etc.

[0611] Input: The extracted data.

[0612] Output: Processed data (e.g., sales data with missing values ​​filled with zeros and filtered).

[0613] Specific operation: The server uses the Pandas library to fill missing values ​​with zeros (e.g., df.fillna(0, inplace=True)), or extracts only specific columns to create a new data frame (e.g., df[['Sales', 'Month']]).

[0614] Step 6: Data analysis

[0615] The server then performs analytical processing on the processed data, such as aggregating monthly sales figures to identify trends and patterns.

[0616] Input: Processed data.

[0617] Output: Analysis results (e.g. monthly sales totals, sales increase / decrease).

[0618] Specific operation: The server uses Pandas and Scikit-learn to calculate the total sales for each month (e.g., df.groupby('Month')['Sales'].sum()) and calculates the increase or decrease in sales (e.g., df.pct_change()).

[0619] Step 7: Deploying the results

[0620] The server converts the analysis results into the specified format (e.g., graphs or reports). It uses Matplotlib to generate graphs and saves them as image files.

[0621] The terminal presents the results (e.g., graph images) sent from the server to the user.

[0622] Input: Analysis results.

[0623] Output: Visualized results (e.g., graph images, PDF reports).

[0624] Specific operation: The server creates a graph using the Matplotlib library (e.g., df.plot(kind='bar')) and saves it as an image (e.g., plt.savefig(' / tmp / results / sales_trend.png')). The terminal saves the generated image file as an HTML Present it to the user using tags.

[0625] (Application example 1)

[0626] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0627] In conventional data processing systems, the process of manually extracting, processing, and analyzing data is complex and requires a lot of time and effort. It is also difficult to properly manage inventory and issue shipping instructions in real time at logistics centers, resulting in a decrease in the efficiency of logistics operations. To solve these problems, the present invention aims to provide a system that automates logistics management by efficiently processing data through voice operation.

[0628] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0629] In this invention, the server includes means for inputting original data into the system to control the data processing system by voice operation, means for converting voice instructions obtained from the voice input into text data, and means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result deployment, thereby enabling the user to perform inventory management and shipping instructions based on voice instructions and receive feedback in real time.

[0630] "Voice operation" refers to an operation in which a user gives instructions to a system using a voice input device such as a microphone.

[0631] A "data processing system" is a computer system that performs a series of processes, from inputting original data to receiving and analyzing voice instructions, executing tasks, and presenting results.

[0632] "Original data" refers to the initial data (e.g., CSV file) provided by the user for processing by the system.

[0633] A "voice instruction" is a specific command or request that the user issues into the microphone (e.g., "Analyze sales data monthly").

[0634] "Text data" refers to data in the form of a character string obtained by converting voice input.

[0635] A "task" is a specific unit of processing performed by the system, such as data extraction, data processing, data analysis, and result deployment.

[0636] "Data extraction" is the process of extracting necessary information from original data based on specified conditions.

[0637] "Data processing" refers to the process of organizing extracted information using methods such as filtering and normalization to improve its quality.

[0638] "Data analysis" is the process of finding trends and patterns based on processed information.

[0639] "Result development" refers to the process of converting the analysis results into a specified format (e.g., graph, report, etc.) and presenting them to the user.

[0640] "Feedback" refers to providing the system with real-time responses and results to user-generated voice commands.

[0641] "Logistics management means" refers to functions that allow inventory management and shipping instructions to be performed based on voice commands, and provide feedback in real time.

[0642] The system of the present invention is designed to efficiently perform data processing and logistics management through voice operations. This system is realized by the following configuration and processing.

[0643] System configuration

[0644] 1. Device:

[0645] Hardware: Smartphones and tablets with microphones

[0646] Software: Speech recognition library (e.g., SpeechRecognition)

[0647] 2. Server:

[0648] Hardware: Cloud-based server infrastructure

[0649] Software: Natural language processing engine, data processing program (e.g., Pandas, Matplotlib)

[0650] 3. Data Store:

[0651] A database where raw data (e.g., inventory data, shipping data) is stored

[0652] Program processing

[0653] The user inputs voice instructions using a microphone into the terminal, such as "Tell me the inventory status" or "Ship 50 units of product A" that are related to logistics management.

[0654] The device receives this voice instruction and converts it into text data using a speech recognition library, which is then sent to the server as a request.

[0655] The server passes the received text data to a natural language processing engine for analysis. This analysis identifies multiple tasks, such as data extraction, data processing, and data analysis. For example, from an instruction such as "Ship 50 units of product A," updating inventory data and generating shipping instructions are identified as tasks.

[0656] The server extracts the necessary data from the database based on the specified task, then performs processing such as filtering and normalization on the extracted data, and analyzes the processed data to generate the results.

[0657] The generated results are converted into a specified format (e.g., inventory list, shipping report, graph, etc.) and sent to the terminal.

[0658] The terminal displays the results sent from the server to the user, allowing the user to receive feedback on inventory status and shipping instructions in real time.

[0659] Examples of specific examples and prompts

[0660] Examples:

[0661] The user issues a voice command to their smartphone saying, "Ship 50 units of product A."

[0662] The system converts this speech into text and analyzes it on the server.

[0663] The server updates inventory data and generates shipping instructions.

[0664] The terminal provides real-time feedback to the user, such as "Shipping instructions: 50 units of product A have been shipped."

[0665] Example prompt sentence:

[0666] User: "Ship 50 units of product A."

[0667] Example of system response: "Shipping instructions: 50 units of product A have been shipped."

[0668] As described above, the system of the present invention enables efficient data processing and logistics management through voice operation, and provides real-time feedback.

[0669] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0670] Step 1:

[0671] The user inputs voice instructions using a microphone, which include specific commands or requests (e.g., "Ship 50 units of product A").

[0672] Input: User's voice data

[0673] Output: Audio file captured from microphone

[0674] Step 2:

[0675] The device uses a speech recognition library to convert the captured voice data into text data in real time. This conversion process is performed using a speech recognition engine (e.g., SpeechRecognition).

[0676] Input: Audio file

[0677] Output: Text-based voice instructions (e.g., "Ship 50 units of product A")

[0678] Step 3:

[0679] The terminal sends the converted text data to the server as a request in the form of an HTTP request.

[0680] Input: Text-based voice instructions

[0681] Output: HTTP request

[0682] Step 4:

[0683] The server passes the received text data to a natural language processing engine for analysis. The analysis identifies tasks such as data extraction, data processing, and data analysis. Specifically, from the command "Ship 50 units of product A," it identifies the tasks of updating inventory data and generating shipping instructions.

[0684] Input: Text-based voice instructions

[0685] Output: Identified tasks (e.g., updating inventory data, generating shipping instructions)

[0686] Step 5:

[0687] The server extracts the required data from the database based on the specified task, using SQL queries.

[0688] Input: Task Instructions

[0689] Output: Extracted data (e.g. inventory data)

[0690] Step 6:

[0691] The server processes the extracted data, such as filtering and normalization, using a data frame manipulation library (e.g., Pandas).

[0692] Input: Extracted data

[0693] Output: Processed data

[0694] Step 7:

[0695] The server analyzes the processed data and generates results, where data analysis tools (e.g., Matplotlib) are used.

[0696] Input: Processed data

[0697] Output: Analysis results (e.g. shipping instructions)

[0698] Step 8:

[0699] The server converts the generated results into the specified format (e.g., report, graph, etc.) using a data visualization library (e.g., Matplotlib).

[0700] Input: Analysis results

[0701] Output: Results in the specified format (e.g. graph, report)

[0702] Step 9:

[0703] The server transmits the generated results to the terminal.

[0704] Input: Result in the specified format

[0705] Output: HTTP response

[0706] Step 10:

[0707] The terminal then presents the received results to the user, who can receive real-time feedback on inventory information and shipping instructions.

[0708] Input: HTTP response (result in the specified format)

[0709] Output: Feedback displayed on the user screen (e.g., "Shipping instructions: 50 units of product A have been shipped.")

[0710] This process enables efficient data processing and logistics management through voice control, providing real-time feedback.

[0711] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0712] The present invention combines a voice-operated data processing system with an emotion engine, and is a system that realizes more intuitive and effective data processing by recognizing the user's emotional state and automatically adjusting data processing and providing feedback based on that state.

[0713] System configuration

[0714] 1. Data Entry

[0715] The user uploads raw data (e.g., CSV file) such as sales data to the system.

[0716] The terminal receives the original data from the user and stores it in a temporary data store.

[0717] 2. Acquiring voice instructions

[0718] The user issues a voice command into the microphone, such as "Analyze sales data monthly."

[0719] The device collects audio from a microphone and converts the audio data into text format.

[0720] 3. Voice command analysis and emotion recognition

[0721] The device uses an emotion engine to analyze the user's emotions based on the converted text data as well as the voice data.

[0722] The server passes the received text data and the emotional state information obtained from the emotion engine to the natural language processing (NLP) engine, and identifies tasks, including data extraction, data processing, data analysis, and result deployment.

[0723] Execution of each task

[0724] Data Extraction

[0725] The server retrieves the original data file from the temporary data store and extracts the required information based on voice instructions, such as sales data for a specified month or period.

[0726] Data Processing

[0727] The server retrieves the extracted data for processing, specifically, performing operations such as filtering and normalization to improve the data quality.

[0728] Data analysis

[0729] The server retrieves the processed data for data analysis, for example, aggregating the extracted sales data monthly to identify trends and patterns.

[0730] Expanding the results

[0731] The server converts the analysis results into the specified format (e.g., graphs or reports).

[0732] The terminal presents the results sent from the server to the user.

[0733] Applications of Emotion Recognition

[0734] Once the emotion engine has analyzed the user's emotions, it can adjust the system's behavior based on the resulting emotional state as follows:

[0735] If the emotion engine recognizes that the user is feeling stressed, the server will provide the results of the data processing in a more intuitive and easy-to-understand format (for example, more concise graphs or reports explained in simpler language).

[0736] If the emotion engine recognizes that the user is happy, the device will display more detailed data analysis results and in-depth analysis results.

[0737] The server responds to the voice prompts by providing appropriate feedback to the user and suggesting the next action (e.g., "Would you like to perform a more detailed analysis?").

[0738] Specific examples

[0739] 1. The user issues a voice command to the system saying, "Analyze sales data monthly," and uploads the original data.

[0740] 2. The device receives the voice instructions and the original data, converts the voice into text, and sends it to the server.

[0741] 3. The device uses an emotion engine to analyze the user's emotions and recognize, for example, that the user is feeling anxious.

[0742] 4. Based on the voice instructions and emotional data received by the server, data is extracted, processed, and analyzed, and a report is generated that highlights only the most important points for impatient users.

[0743] 5. The terminal displays the generated report to the user, who can then view the report in an easy-to-understand format.

[0744] The system is able to take into account the user's emotional state and provide more responsive and user-friendly data processing and feedback.

[0745] The processing flow will be explained below.

[0746] Step 1:

[0747] The user uploads a CSV file of sales data to the system.

[0748] The device verifies the uploaded file and saves it in a temporary data store.

[0749] Step 2:

[0750] The user issues a voice command into the microphone saying, "Analyze sales data monthly."

[0751] The device collects audio from a microphone and converts the audio data into text format.

[0752] Step 3:

[0753] The terminal inputs the voice data to the emotion engine together with the voice instruction converted into text data.

[0754] The emotion engine analyzes the audio data to identify the user's emotional state (e.g., impatience, joy, stress, etc.).

[0755] The terminal transmits the voice instruction in text format and the analysis results of the emotion engine to the server.

[0756] Step 4:

[0757] The server analyzes the received text data and emotional state using a natural language processing (NLP) engine.

[0758] The server identifies tasks based on the voice instructions, including "data extraction," "data processing," "data analysis," and "result deployment."

[0759] Step 5:

[0760] The server retrieves the original data file from the temporary data store.

[0761] The server extracts monthly sales data from the original data based on the analyzed voice instructions.

[0762] Step 6:

[0763] The server processes the extracted data, specifically by filtering, normalizing, and other operations to improve the quality of the data.

[0764] Step 7:

[0765] The server analyzes the processed data, for example by aggregating extracted sales data monthly to identify trends and patterns.

[0766] The server adjusts the analysis method and the result display format based on the user's emotional state.

[0767] Step 8:

[0768] The server converts the analysis results into the specified format (e.g., graphs or reports).

[0769] The server transmits the converted result to the terminal.

[0770] Step 9:

[0771] The device then displays the received analysis results to the user. For example, if the device detects that the user is in a hurry, it may present the results in a more concise and visually understandable format.

[0772] The device provides additional feedback and action suggestions (e.g., "Would you like to conduct a more detailed analysis?") depending on the user's emotions.

[0773] This series of steps allows users to experience responsive data processing and feedback that utilizes emotion recognition.

[0774] Example 2

[0775] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0776] Conventional voice-operated data processing systems have a limited scope for automating data processing based on voice instructions, and they are unable to provide feedback according to the user's emotional state. As a result, the system's response may be inappropriate when the user is stressed or when more detailed information is required.

[0777] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to input original data into the system, a means for converting voice instructions obtained from voice input into text data, a means for analyzing the converted text data and voice parameters and recognizing the user's emotional state, and a means for analyzing the text data and emotional data and identifying multiple tasks for data extraction, data processing, data analysis, and result development. This enables highly responsive data processing and feedback according to the user's emotional state.

[0778] A "user" is an entity that operates the system and inputs voice instructions and data.

[0779] "Original data" refers to the underlying information, such as sales data, and refers to the data files uploaded to the system.

[0780] "Voice input" refers to voice instructions given by the user through a microphone.

[0781] "Text data" refers to speech input converted into text format.

[0782] "Voice parameters" refer to features and information extracted from voice data, and are data that can be used for emotion analysis.

[0783] "Emotional state" is information indicating the user's emotions, such as impatience, joy, or stress.

[0784] "Data extraction" refers to the process of extracting necessary information from original data based on specified conditions.

[0785] "Data processing" refers to the process of filtering, normalizing, and other processes performed on extracted data to improve its quality.

[0786] "Data analysis" refers to the analytical process of using processed data to identify trends and patterns.

[0787] "Result development" refers to the process of converting the analysis results into a specified format (e.g., graphs or reports) and presenting them to the user.

[0788] "Feedback" refers to information or suggestions provided to users based on the results of data processing and analysis.

[0789] The present invention combines a voice-operated data processing system with an emotion engine, and is a system that realizes more intuitive and effective data processing by recognizing the user's emotional state and automatically adjusting data processing and providing feedback based on that state.

[0790] First, the user uploads raw data, such as sales data, to the system. The raw data is typically in CSV file format, and the device stores it in a temporary data store. The user uses a microphone to provide voice input. Voice instructions can be simple commands such as "Analyze sales data monthly." The device collects voice from the microphone and converts the voice data into text format using voice recognition software (e.g., a voice recognition API). Along with this converted text data, voice parameters are also obtained.

[0791] Next, the device analyzes the user's emotional state using a sentiment analysis engine (e.g., a sentiment analysis API). The analyzed emotional information is sent to a server, where a natural language processing (NLP) engine (e.g., an NLP model) analyzes the received text data and emotional data and identifies tasks. The identified tasks include "data extraction," "data processing," "data analysis," and "result deployment."

[0792] The server retrieves raw data from the temporary data store and extracts the required information based on voice commands. For example, sales data for a specified month or period may be extracted. The extracted data undergoes data processing operations such as filtering and normalization, and is then analyzed. For example, the analysis may aggregate sales data monthly to identify trends and patterns. The processed and analyzed data is then converted into a specified format (for example, a graph or report).

[0793] The server sends the calculated analysis results to the device, which then provides more appropriate feedback based on the user's emotional state as recognized by the emotion engine. For example, the device might display the results of data processing in an intuitive and easy-to-understand format to a stressed user, or display more detailed data and in-depth analysis results to a happy user. The server also provides appropriate feedback in response to voice commands and suggests the next course of action.

[0794] As a concrete example, consider a scenario in which a user issues a voice command to the system to "analyze monthly sales data" and uploads the original data. The device receives the voice command and the original data, converts the voice into text, and sends it to the server. The device then uses an emotion engine to analyze the user's emotions and recognizes, for example, that the user is feeling anxious. The server extracts, processes, and analyzes data based on the received voice command and emotional data, and generates a report that highlights only the most important points for anxious users. The device then displays the generated report to the user, who can view it in an easy-to-understand format.

[0795] Examples of prompt sentences include the following:

[0796] "The user issues a voice command to the system saying, 'Analyze monthly sales data,' and uploads the original data. The device receives the voice command and the original data, converts the voice into text, and sends it to the server. The device uses an emotion engine to analyze the user's emotions and recognizes, for example, that the user is feeling anxious. The server extracts, processes, and analyzes the data based on the received voice command and emotional data, and generates a report that highlights only the most important points for anxious users."

[0797] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0798] Step 1: User uploads data

[0799] Users upload raw data, such as sales data, to the system interface by dragging and dropping it. The input is a CSV-formatted data file, which the terminal saves in a temporary data store. The output is a CSV file saved in the temporary data store. The specific operation is to upload the file using a browser.

[0800] Step 2: The device receives voice instructions

[0801] The user speaks into the microphone, saying, "Analyze sales data monthly." The input is the user's voice command, and the device picks up this voice through the built-in microphone. The output is voice data (audio file). Specifically, the device uses the microphone to capture the voice.

[0802] Step 3: Your device converts your voice commands to text

[0803] The device uses voice recognition software (e.g., voice recognition API) to convert the voice data into text format. The input is voice data, and the output is text data such as "Analyze sales data monthly." Specifically, the device calls the voice recognition API and converts the voice data into text.

[0804] Step 4: The device performs sentiment analysis

[0805] The device passes the text data and voice parameters to an emotion analysis engine (e.g., emotion analysis API) to analyze the user's emotional state. The input is text data and voice parameters, and the output is the user's emotional state (e.g., impatience, joy). Specifically, the device calls the emotion analysis API and performs emotion analysis.

[0806] Step 5: Server identifies the task

[0807] The server passes text and sentiment data to a natural language processing (NLP) engine (e.g., an NLP model) to identify a task. The input is text and sentiment data, and the output is the identified task (e.g., data extraction, data processing, data analysis, or result deployment). Specifically, the server uses an NLP model to analyze the text and sentiment and identify the task.

[0808] Step 6: The server performs the data extraction

[0809] The server retrieves the raw data (CSV files) from a temporary data store and extracts the required information based on voice instructions. The input is the raw data and voice instructions, and the output is the extracted information (e.g., sales data for a specified month). Specifically, it uses a database query to extract data that matches the criteria.

[0810] Step 7: The server performs the data processing

[0811] The server processes the extracted data by filtering and normalizing it. The input is the extracted information, and the output is the processed information (e.g., clean data). Specifically, the program complements missing values ​​in the data and removes outliers.

[0812] Step 8: The server performs the data analysis

[0813] The server uses the processed data to perform analysis to identify trends and patterns. The input is the processed information, and the output is the analytical results (e.g., monthly sales trends). Specifically, it runs analytical algorithms and generates statistical information on the data.

[0814] Step 9: The server extracts the results

[0815] The server converts the analysis results into the specified format (e.g., graphs or reports) and sends them to the terminal. The input is the analysis results, and the output is the converted results (e.g., graphs, reports). Specifically, the results are visualized using a data visualization tool.

[0816] Step 10: Your device will display the results

[0817] The terminal displays the results sent from the server to the user. The input is the converted result, and the output is a screen display that is visually presented to the user. Specifically, it displays graphs and reports on the screen so that the user can check them.

[0818] Step 11: The server and device adjust according to the emotion.

[0819] The server adjusts the data feedback based on the emotional state recognized by the emotion engine, and the device displays the results in a format appropriate for the user. The input is the emotional state and the resulting data, and the output is adjusted feedback (e.g., concise display, detailed data). Specifically, the display content is dynamically changed according to the user's emotional state.

[0820] (Application example 2)

[0821] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0822] Today, users use voice commands to perform various data processing tasks. However, conventional systems do not take into account the user's emotional state, resulting in a lack of an intuitive and personal operating experience. As a result, users can experience inconvenience and stress when browsing and processing data. By solving this problem, we aim to improve user satisfaction and the effectiveness of data processing.

[0823] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting original data to the system to control the data processing system by voice operation, means for converting voice instructions obtained from the voice input into text data, means for analyzing the user's emotional state from the text data and voice data, means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result development, means for extracting information from the original data based on specified conditions, means for processing the extracted information, means for analyzing the processed information, means for converting information based on the analysis results and the user's emotional state into a specified format, and means for presenting appropriate feedback to the user based on the analysis results and the user's emotional state. This enables personalized data processing and feedback that takes the user's emotional state into consideration.

[0824] "Voice operation" is a technology that controls a system and executes instructions through user voice input.

[0825] A "data processing system" is a system that extracts, processes, and analyzes various types of data based on input data.

[0826] "Raw data" refers to the original, unprocessed data that is input into the system.

[0827] "Voice instructions" refer to instructions or commands given by the user through voice.

[0828] "Text data" is data in character format converted based on voice instructions.

[0829] An "emotional state" is a psychological state inferred from a user's voice or other input.

[0830] A "task" is a specific process or operation that a system performs.

[0831] "Data extraction" refers to extracting necessary information from original data based on specified conditions.

[0832] "Data processing" is the act of formatting or converting extracted data to make it easier to use.

[0833] "Data analysis" is the process of using processed data to find meaningful patterns and trends.

[0834] "Result presentation" refers to presenting the results of data analysis in a user-friendly format.

[0835] "Feedback" refers to the reaction or information that a system provides to a user.

[0836] A "natural language processing engine" is a technology for analyzing text data and understanding its meaning and context.

[0837] An "emotion engine" is a technology for analyzing a user's emotional state from their voice and data.

[0838] The present invention is a system that combines a voice-operated data processing system with an emotion engine, which recognizes the user's emotional state and provides automatic adjustments and feedback to data processing based on that state. The system of the present invention is implemented using the following hardware and software.

[0839] Hardware and software used

[0840] Hardware

[0841] Smartphones (e.g. iPhone, Android devices)

[0842] microphone

[0843] server

[0844] software

[0845] Google Cloud Speech-to-Text API

[0846] IBM Watson Tone Analyzer

[0847] Google Natural Language API

[0848] Food delivery backend server

[0849] System configuration

[0850] 1. Acquiring voice instructions

[0851] The user speaks into the smartphone's microphone, saying something like "I want to order a pizza." This voice input is captured by the smartphone. The captured voice data is converted into text data using the Google Cloud Speech-to-Text API.

[0852] 2. Emotion analysis

[0853] The converted text data and the original voice data are then analyzed using IBM Watson Tone Analyzer to determine the user's emotional state, for example, whether the user is relaxed or anxious.

[0854] 3. Analysis of voice instructions

[0855] The parsed text data is then passed through the Google Natural Language API to identify the tasks the system should perform (e.g., display a menu, process an order), allowing it to optimally respond based on the user's emotional state and voice commands.

[0856] 4. Food delivery order fulfillment

[0857] The food delivery backend server retrieves menu information from a database based on the user's emotional state and voice instructions, and sends it to the smartphone. For example, if the user is tired, a simple menu is displayed, and if the user is relaxed, a detailed menu is displayed.

[0858] 5. Personalized feedback

[0859] Feedback that takes into account the user's emotional state is provided via smartphone, such as suggestions like "Today's recommendation is Margherita pizza."

[0860] Specific examples

[0861] A user speaks to their smartphone and says, "I'd like to order a pizza." The smartphone captures the voice data and converts it to text using the Google Cloud Speech-to-Text API. The system then uses IBM Watson Tone Analyzer to analyze the user's emotional state. For example, if the system detects that the user is tired, it might suggest, "How about a Margherita pizza?"

[0862] Prompt Sentence Examples

[0863] "If a user says, 'I want to order a pizza':

[0864] "User wants to order pizza. Emotional state is relaxed."

[0865] "Yes, this is the menu we recommend for users in a relaxed state."

[0866] The system allows users to have a personalized ordering experience that is tailored to their emotional state, resulting in increased user satisfaction and a more intuitive experience.

[0867] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0868] Step 1:

[0869] The user speaks into the smartphone's microphone. For example, they say, "I'd like to order a pizza." The input here is the user's voice, and the output is the voice data captured by the smartphone. The smartphone sends this voice data to the Google Cloud Speech-to-Text API, which converts it into text.

[0870] Step 2:

[0871] The smartphone uses the Google Cloud Speech-to-Text API to convert the acquired voice data into text data. The converted text is in the form of "I would like to order a pizza." The input of this process is voice data, and the output is text data.

[0872] Step 3:

[0873] The smartphone uses IBM Watson Tone Analyzer to analyze the user's emotional state from the text data and the original voice data. For example, it determines whether the user is relaxed or tired. The input is text data and voice data, and the output is data indicating the user's emotional state.

[0874] Step 4:

[0875] The smartphone sends text data and emotion data to the Google Natural Language API for analysis. The analyzed data is used to identify the tasks the system should perform (e.g., display a menu or process an order). The inputs for this step are text data and emotion data, and the output is the identified tasks.

[0876] Step 5:

[0877] The server accesses the food delivery backend system and retrieves menu information from the database based on the user's emotional state and voice instructions. For example, if the user is tired, a concise menu is displayed. The input is the specified task and the user's emotional state, and the output is the adjusted menu information.

[0878] Step 6:

[0879] The server sends the adjusted menu information to the smartphone, which then displays this information to the user, for example, "Today's special is Margherita pizza." The input is the adjusted menu information, and the output is the user's visual feedback.

[0880] Step 7:

[0881] The user makes a selection from the displayed menu and confirms the order by voice, for example, "I'd like to order a Margherita pizza." The smartphone converts this speech to text, again using the Google Cloud Speech-to-Text API. The input is the user's voice, and the output is text data.

[0882] Step 8:

[0883] The smartphone sends a final confirmation text to the food delivery backend system to confirm the order. The server receives the order and starts processing it. The input is the order confirmation text, and the output is the food delivery order data.

[0884] This process allows users to have a personalized ordering experience that is tailored to their emotional state, resulting in increased user satisfaction and a more intuitive experience.

[0885] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0886] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0887] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0888] [Third embodiment]

[0889] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0890] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0891] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0892] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0893] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0894] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0895] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0896] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0897] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0898] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0899] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0900] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0901] The present invention is a system that controls a data processing system by voice operation, and allows a user to input original data into the system and give voice instructions to fully automatically extract, process, analyze, and develop the results of the data. This system is implemented by the following program processing.

[0902] System configuration

[0903] 1. Data Entry

[0904] The user uploads raw data (e.g., CSV file) such as sales data to the system.

[0905] The terminal receives the original data from the user and stores it in a temporary data store.

[0906] 2. Acquiring voice instructions

[0907] The user issues a voice command into the microphone, such as "Analyze sales data monthly."

[0908] The terminal captures audio from a microphone, converts the audio data into text format in real time, and sends it to a server.

[0909] 3. Analysis of voice instructions

[0910] The server passes the text data of the voice instructions to a natural language processing engine, which analyzes the content of the voice instructions. This analysis identifies multiple tasks: data extraction, data processing, data analysis, and result presentation.

[0911] Execution of each task

[0912] Data Extraction

[0913] The server retrieves the raw data from the temporary data store and extracts the required information based on voice instructions, such as sales data for a specified month or period.

[0914] Data Processing

[0915] The server processes the extracted data, specifically by using techniques such as filtering and normalization to improve the quality of the data.

[0916] Data analysis

[0917] The server analyzes the processed data, for example, aggregating extracted sales data monthly to identify trends and patterns.

[0918] Expanding the results

[0919] The server converts the analysis results into the specified format (e.g., graphs or reports).

[0920] The terminal presents the results sent from the server to the user, for example displaying bar graphs or reports showing sales trends.

[0921] Specific examples

[0922] 1. Inputting the original data

[0923] A user uploads a CSV file of sales data into the system.

[0924] The device will save this file and accept microphone operations.

[0925] 2. Enter voice commands

[0926] The user issues a voice command such as "Analyze sales data monthly."

[0927] The device converts this speech into text and sends it to the server.

[0928] 3. Analysis of voice instructions

[0929] The server analyzes the voice instructions and identifies tasks for data extraction, processing, analysis, and deployment.

[0930] 4. Execute the task

[0931] The server performs a series of data extraction, data processing, and data analysis, and generates the results as a graph.

[0932] The device displays the generated graphs to the user, providing a visual representation of monthly sales trends and patterns.

[0933] This system allows users to perform complex data processing with simple voice commands, enabling efficient data analysis regardless of the user's skill level. Furthermore, the automation of each task significantly reduces work time.

[0934] The processing flow will be explained below.

[0935] Step 1:

[0936] The user uploads a CSV file of sales data to the system.

[0937] The device verifies the uploaded file and saves it in a temporary data store.

[0938] Step 2:

[0939] The user issues a voice command into the microphone saying, "Analyze sales data monthly."

[0940] The device collects audio from a microphone and converts the audio data into text format.

[0941] Step 3:

[0942] The terminal transmits the converted text data to the server.

[0943] The server analyzes the received text data using a natural language processing (NLP) engine and identifies tasks, such as data extraction, data processing, data analysis, and results presentation.

[0944] Step 4:

[0945] The server retrieves the original data file from the temporary data store.

[0946] The server extracts specified information, such as monthly sales data, from the original data based on the analyzed voice instructions.

[0947] Step 5:

[0948] The server obtains the extracted data for data processing.

[0949] As part of the data processing, the server performs operations such as filtering and normalization to improve the quality of the data.

[0950] Step 6:

[0951] The server obtains the processed data for data analysis.

[0952] The server uses the obtained data to perform a specified analysis (e.g., monthly sales trend analysis) and identify trends and patterns in the data.

[0953] Step 7:

[0954] The server converts the analysis results into the specified format (e.g., graphs or reports).

[0955] The server transmits the converted result to the terminal.

[0956] Step 8:

[0957] The terminal displays the received analysis results to the user.

[0958] Users can view graphs and reports displayed on their devices to understand sales trends and patterns.

[0959] This series of steps allows users, regardless of their skill level, to efficiently perform complex data processing using voice instructions.

[0960] Example 1

[0961] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0962] Conventional data processing systems require users to manually perform numerous steps to extract, process, analyze, and develop results, often resulting in complex operations and requiring specialized knowledge. Furthermore, there are few fully automated voice-operated data processing systems, placing a significant burden on users. Furthermore, there is a lack of technology that can accurately analyze voice commands and efficiently execute each task automatically.

[0963] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0964] In this invention, the server includes means for converting voice instructions obtained from voice input into text data, means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result development, and means for processing the extracted information, thereby enabling the user to fully automatically extract, process, analyze, and develop results using only voice operations.

[0965] "Voice operation" is an operation method in which the user controls various functions of the system using voice.

[0966] A "data processing system" is a system for inputting, extracting, processing, analyzing, and expanding data.

[0967] "Original data" refers to the initial data entered into the system, including formats such as CSV files.

[0968] "Voice input" is a method in which a user supplies voice to the system using a microphone or the like.

[0969] "Voice instructions" are instructions given to the system by voice input.

[0970] "Text data" is data in the form of a character string converted from voice input.

[0971] "Analysis" is the act of understanding input data (here, text data) and grasping its meaning.

[0972] "Data extraction" is the process of extracting only the necessary information from the original data based on specific conditions.

[0973] "Data processing" is the process of filtering and normalizing extracted data to improve its quality and usefulness.

[0974] "Data analysis" is the process of performing calculations and aggregations on processed data to find useful information and patterns.

[0975] "Developing results" refers to summarizing the analysis results in an easy-to-read format and presenting them to the user.

[0976] "Speech recognition software" is software for converting voice data into text data.

[0977] A "natural language processing engine" is a software technology for analyzing text data and understanding the meaning of human language.

[0978] A "task" is a discrete operation performed by the system, such as data extraction, data processing, data analysis, and results presentation.

[0979] The present invention relates to a method and system for controlling a data processing system by voice operation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS ... will be described in detail below with reference to preferred embodiments thereof.

[0980] System configuration

[0981] Hardware used

[0982] 1. Terminal: A device operated by a user (such as a personal computer, smartphone, or tablet)

[0983] 2. Server: A computer for processing data

[0984] 3. Microphone: An input device for obtaining voice instructions

[0985] Software used

[0986] 1. Speech recognition software: Used to convert voice data into text data (e.g., Google Cloud Speech-to-Text, IBM Watson Speech to Text)

[0987] 2. Natural language processing engines: used to analyze text data (e.g., Google Cloud Natural Language, Microsoft Azure Text Analytics)

[0988] 3. Data processing tools: Programming languages ​​and libraries (e.g., Python, Pandas, NumPy) for processing the extracted data.

[0989] 4. Data analysis tools: Programming languages ​​and libraries for analyzing the processed data (e.g., Python, Scikit-learn, Matplotlib, Seaborn)

[0990] System Operation

[0991] User Operation

[0992] The user uploads the original data (e.g., a CSV file of sales data) from their terminal to the system. To do this, the user selects the file using the browser's file selection dialog and presses the upload button. Next, the user gives voice instructions into the microphone. For example, they can input instructions in voice format such as "Analyze the sales data monthly."

[0993] Device Features

[0994] The device receives files uploaded by users and temporarily stores them in local storage. The device also captures voice data from the microphone and converts it into text format in real time using voice recognition software. This converted text data is then sent to the server for further processing.

[0995] Server Processing

[0996] The server passes the text data sent from the speech recognition software to the natural language processing engine for analysis. Based on the analysis results, the server identifies the following tasks: data extraction, data processing, data analysis, and result distribution. The server executes these tasks sequentially to generate the final analysis results.

[0997] For example, in the data extraction stage, the server uses Python's Pandas library to read the original data CSV file and extract the specified monthly sales data. In the data processing stage, it fills in missing values ​​with zeros and filters specific columns. In the data analysis stage, it calculates the total sales for each month and analyzes the increase or decrease. In the final results development stage, it creates a graph using Matplotlib and saves it as an image file.

[0998] Presentation of results

[0999] The generated analysis results (e.g., graph images and reports) are sent from the server to the terminal. The terminal visually presents the results to the user. Specifically, graph images can be displayed on the browser.

[1000] Specific examples

[1001] Next, a specific example will be given.

[1002] 1. Upload the original data

[1003] A user uploads a file called sales_data.csv.

[1004] The device will save this file in the / tmp / uploads / directory.

[1005] 2. Enter voice commands

[1006] The user issues a voice command such as "Analyze sales data monthly."

[1007] The device converts the speech into text and sends it to the server.

[1008] 3. Analysis of voice instructions

[1009] The server analyzes the voice instructions and identifies tasks for data extraction, processing, analysis, and deployment.

[1010] 4. Task execution and results presentation

[1011] The server performs a series of data extraction, data processing, and data analysis, and generates the results as a graph.

[1012] The device displays the generated graphs to the user, providing a visual representation of monthly sales trends and patterns.

[1013] Prompt Sentence Examples

[1014] "Analyze sales data monthly."

[1015] "Compare sales trends this month and last month."

[1016] "What are the top 5 product sales figures?"

[1017] This allows users to easily perform advanced data analysis using only voice commands, enabling them to gain useful insights from large amounts of data and complete processes efficiently, even without specific skills.

[1018] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1019] Step 1: Data entry

[1020] The user uploads a CSV file of sales data (e.g., sales_data.csv) to the system.

[1021] The terminal receives the CSV file uploaded by the user and temporarily stores it in local storage (e.g., / tmp / uploads / ).

[1022] Input: A CSV file selected by the user in the browser.

[1023] Output: CSV file saved in temporary directory.

[1024] Specific operation: The device reads the file selected from the browser's file selection dialog using the JavaScript FileReader object and saves it in a temporary storage directory.

[1025] Step 2: Getting voice instructions

[1026] The user issues a voice command into the microphone saying, "Analyze sales data monthly."

[1027] The device receives voice data from the microphone and converts it into text using real-time voice recognition software (e.g., Google Cloud Speech-to-Text), which then sends the text data to a server.

[1028] Input: User's voice command.

[1029] Output: Spoken instructions converted to text (e.g., "Analyze sales data monthly").

[1030] Specific operation: The device acquires microphone data using the navigator.mediaDevices.getUserMedia API, sends the audio data to the speech recognition API, and sends the converted text data to the server via WebSocket.

[1031] Step 3: Analyzing voice instructions

[1032] The server passes the text data obtained by speech recognition to a natural language processing engine (e.g., Google Cloud Natural Language) and analyzes the voice instructions.

[1033] Input: Text-based voice instructions.

[1034] Output: Parsed instructions (e.g. "Monthly analysis of sales data").

[1035] Specific operations: The server sends an API request to the natural language processing engine, receives the analysis results of the text data, and identifies tasks such as data extraction, data processing, data analysis, and result presentation.

[1036] Step 4: Data extraction

[1037] The server reads the original data from the temporarily saved CSV file and extracts the necessary information based on voice instructions.

[1038] Input: Temporarily saved CSV file, parsed instructions.

[1039] Output: The extracted data (e.g. sales data for a specified month).

[1040] Specific operation: The server uses Python's Pandas library to read the CSV file and extract sales data for the specified period (e.g., df[df['month'] == '2023-01']).

[1041] Step 5: Data processing

[1042] The server processes the extracted data, for example, by handling missing values, filtering, normalizing, etc.

[1043] Input: The extracted data.

[1044] Output: Processed data (e.g., sales data with missing values ​​filled with zeros and filtered).

[1045] Specific operation: The server uses the Pandas library to fill missing values ​​with zeros (e.g., df.fillna(0, inplace=True)), or extracts only specific columns to create a new data frame (e.g., df[['Sales', 'Month']]).

[1046] Step 6: Data analysis

[1047] The server then performs analytical processing on the processed data, such as aggregating monthly sales figures to identify trends and patterns.

[1048] Input: Processed data.

[1049] Output: Analysis results (e.g. monthly sales totals, sales increase / decrease).

[1050] Specific operation: The server uses Pandas and Scikit-learn to calculate the total sales for each month (e.g., df.groupby('Month')['Sales'].sum()) and calculates the increase or decrease in sales (e.g., df.pct_change()).

[1051] Step 7: Deploying the results

[1052] The server converts the analysis results into the specified format (e.g., graphs or reports). It uses Matplotlib to generate graphs and saves them as image files.

[1053] The terminal presents the results (e.g., graph images) sent from the server to the user.

[1054] Input: Analysis results.

[1055] Output: Visualized results (e.g., graph images, PDF reports).

[1056] Specific operation: The server creates a graph using the Matplotlib library (e.g., df.plot(kind='bar')) and saves it as an image (e.g., plt.savefig(' / tmp / results / sales_trend.png')). The terminal saves the generated image file as an HTML Present it to the user using tags.

[1057] (Application example 1)

[1058] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1059] In conventional data processing systems, the process of manually extracting, processing, and analyzing data is complex and requires a lot of time and effort. It is also difficult to properly manage inventory and issue shipping instructions in real time at logistics centers, resulting in a decrease in the efficiency of logistics operations. To solve these problems, the present invention aims to provide a system that automates logistics management by efficiently processing data through voice operation.

[1060] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1061] In this invention, the server includes means for inputting original data into the system to control the data processing system by voice operation, means for converting voice instructions obtained from the voice input into text data, and means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result deployment, thereby enabling the user to perform inventory management and shipping instructions based on voice instructions and receive feedback in real time.

[1062] "Voice operation" refers to an operation in which a user gives instructions to a system using a voice input device such as a microphone.

[1063] A "data processing system" is a computer system that performs a series of processes, from inputting original data to receiving and analyzing voice instructions, executing tasks, and presenting results.

[1064] "Original data" refers to the initial data (e.g., CSV file) provided by the user for processing by the system.

[1065] A "voice instruction" is a specific command or request that the user issues into the microphone (e.g., "Analyze sales data monthly").

[1066] "Text data" refers to data in the form of a character string obtained by converting voice input.

[1067] A "task" is a specific unit of processing performed by the system, such as data extraction, data processing, data analysis, and result deployment.

[1068] "Data extraction" is the process of extracting necessary information from original data based on specified conditions.

[1069] "Data processing" refers to the process of organizing extracted information using methods such as filtering and normalization to improve its quality.

[1070] "Data analysis" is the process of finding trends and patterns based on processed information.

[1071] "Result development" refers to the process of converting the analysis results into a specified format (e.g., graph, report, etc.) and presenting them to the user.

[1072] "Feedback" refers to providing the system with real-time responses and results to user-generated voice commands.

[1073] "Logistics management means" refers to functions that allow inventory management and shipping instructions to be performed based on voice commands, and provide feedback in real time.

[1074] The system of the present invention is designed to efficiently perform data processing and logistics management through voice operations. This system is realized by the following configuration and processing.

[1075] System configuration

[1076] 1. Device:

[1077] Hardware: Smartphones and tablets with microphones

[1078] Software: Speech recognition library (e.g., SpeechRecognition)

[1079] 2. Server:

[1080] Hardware: Cloud-based server infrastructure

[1081] Software: Natural language processing engine, data processing program (e.g., Pandas, Matplotlib)

[1082] 3. Data Store:

[1083] A database where raw data (e.g., inventory data, shipping data) is stored

[1084] Program processing

[1085] The user inputs voice instructions using a microphone into the terminal, such as "Tell me the inventory status" or "Ship 50 units of product A" that are related to logistics management.

[1086] The device receives this voice instruction and converts it into text data using a speech recognition library, which is then sent to the server as a request.

[1087] The server passes the received text data to a natural language processing engine for analysis. This analysis identifies multiple tasks, such as data extraction, data processing, and data analysis. For example, from an instruction such as "Ship 50 units of product A," updating inventory data and generating shipping instructions are identified as tasks.

[1088] The server extracts the necessary data from the database based on the specified task, then performs processing such as filtering and normalization on the extracted data, and analyzes the processed data to generate the results.

[1089] The generated results are converted into a specified format (e.g., inventory list, shipping report, graph, etc.) and sent to the terminal.

[1090] The terminal displays the results sent from the server to the user, allowing the user to receive feedback on inventory status and shipping instructions in real time.

[1091] Examples of specific examples and prompts

[1092] Examples:

[1093] The user issues a voice command to their smartphone saying, "Ship 50 units of product A."

[1094] The system converts this speech into text and analyzes it on the server.

[1095] The server updates inventory data and generates shipping instructions.

[1096] The terminal provides real-time feedback to the user, such as "Shipping instructions: 50 units of product A have been shipped."

[1097] Example prompt sentence:

[1098] User: "Ship 50 units of product A."

[1099] Example of system response: "Shipping instructions: 50 units of product A have been shipped."

[1100] As described above, the system of the present invention enables efficient data processing and logistics management through voice operation, and provides real-time feedback.

[1101] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1102] Step 1:

[1103] The user inputs voice instructions using a microphone, which include specific commands or requests (e.g., "Ship 50 units of product A").

[1104] Input: User's voice data

[1105] Output: Audio file captured from microphone

[1106] Step 2:

[1107] The device uses a speech recognition library to convert the captured voice data into text data in real time. This conversion process is performed using a speech recognition engine (e.g., SpeechRecognition).

[1108] Input: Audio file

[1109] Output: Text-based voice instructions (e.g., "Ship 50 units of product A")

[1110] Step 3:

[1111] The terminal sends the converted text data to the server as a request in the form of an HTTP request.

[1112] Input: Text-based voice instructions

[1113] Output: HTTP request

[1114] Step 4:

[1115] The server passes the received text data to a natural language processing engine for analysis. The analysis identifies tasks such as data extraction, data processing, and data analysis. Specifically, from the command "Ship 50 units of product A," it identifies the tasks of updating inventory data and generating shipping instructions.

[1116] Input: Text-based voice instructions

[1117] Output: Identified tasks (e.g., updating inventory data, generating shipping instructions)

[1118] Step 5:

[1119] The server extracts the required data from the database based on the specified task, using SQL queries.

[1120] Input: Task Instructions

[1121] Output: Extracted data (e.g. inventory data)

[1122] Step 6:

[1123] The server processes the extracted data, such as filtering and normalization, using a data frame manipulation library (e.g., Pandas).

[1124] Input: Extracted data

[1125] Output: Processed data

[1126] Step 7:

[1127] The server analyzes the processed data and generates results, where data analysis tools (e.g., Matplotlib) are used.

[1128] Input: Processed data

[1129] Output: Analysis results (e.g. shipping instructions)

[1130] Step 8:

[1131] The server converts the generated results into the specified format (e.g., report, graph, etc.) using a data visualization library (e.g., Matplotlib).

[1132] Input: Analysis results

[1133] Output: Results in the specified format (e.g. graph, report)

[1134] Step 9:

[1135] The server transmits the generated results to the terminal.

[1136] Input: Result in the specified format

[1137] Output: HTTP response

[1138] Step 10:

[1139] The terminal then presents the received results to the user, who can receive real-time feedback on inventory information and shipping instructions.

[1140] Input: HTTP response (result in the specified format)

[1141] Output: Feedback displayed on the user screen (e.g., "Shipping instructions: 50 units of product A have been shipped.")

[1142] This process enables efficient data processing and logistics management through voice control, providing real-time feedback.

[1143] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1144] The present invention combines a voice-operated data processing system with an emotion engine, and is a system that realizes more intuitive and effective data processing by recognizing the user's emotional state and automatically adjusting data processing and providing feedback based on that state.

[1145] System configuration

[1146] 1. Data Entry

[1147] The user uploads raw data (e.g., CSV file) such as sales data to the system.

[1148] The terminal receives the original data from the user and stores it in a temporary data store.

[1149] 2. Acquiring voice instructions

[1150] The user issues a voice command into the microphone, such as "Analyze sales data monthly."

[1151] The device collects audio from a microphone and converts the audio data into text format.

[1152] 3. Voice command analysis and emotion recognition

[1153] The device uses an emotion engine to analyze the user's emotions based on the converted text data as well as the voice data.

[1154] The server passes the received text data and the emotional state information obtained from the emotion engine to the natural language processing (NLP) engine, and identifies tasks, including data extraction, data processing, data analysis, and result deployment.

[1155] Execution of each task

[1156] Data Extraction

[1157] The server retrieves the original data file from the temporary data store and extracts the required information based on voice instructions, such as sales data for a specified month or period.

[1158] Data Processing

[1159] The server retrieves the extracted data for processing, specifically, performing operations such as filtering and normalization to improve the data quality.

[1160] Data analysis

[1161] The server retrieves the processed data for data analysis, for example, aggregating the extracted sales data monthly to identify trends and patterns.

[1162] Expanding the results

[1163] The server converts the analysis results into the specified format (e.g., graphs or reports).

[1164] The terminal presents the results sent from the server to the user.

[1165] Applications of Emotion Recognition

[1166] Once the emotion engine has analyzed the user's emotions, it can adjust the system's behavior based on the resulting emotional state as follows:

[1167] If the emotion engine recognizes that the user is feeling stressed, the server will provide the results of the data processing in a more intuitive and easy-to-understand format (for example, more concise graphs or reports explained in simpler language).

[1168] If the emotion engine recognizes that the user is happy, the device will display more detailed data analysis results and in-depth analysis results.

[1169] The server responds to the voice prompts by providing appropriate feedback to the user and suggesting the next action (e.g., "Would you like to perform a more detailed analysis?").

[1170] Specific examples

[1171] 1. The user issues a voice command to the system saying, "Analyze sales data monthly," and uploads the original data.

[1172] 2. The device receives the voice instructions and the original data, converts the voice into text, and sends it to the server.

[1173] 3. The device uses an emotion engine to analyze the user's emotions and recognize, for example, that the user is feeling anxious.

[1174] 4. Based on the voice instructions and emotional data received by the server, data is extracted, processed, and analyzed, and a report is generated that highlights only the most important points for impatient users.

[1175] 5. The terminal displays the generated report to the user, who can then view the report in an easy-to-understand format.

[1176] The system is able to take into account the user's emotional state and provide more responsive and user-friendly data processing and feedback.

[1177] The processing flow will be explained below.

[1178] Step 1:

[1179] The user uploads a CSV file of sales data to the system.

[1180] The device verifies the uploaded file and saves it in a temporary data store.

[1181] Step 2:

[1182] The user issues a voice command into the microphone saying, "Analyze sales data monthly."

[1183] The device collects audio from a microphone and converts the audio data into text format.

[1184] Step 3:

[1185] The terminal inputs the voice data to the emotion engine together with the voice instruction converted into text data.

[1186] The emotion engine analyzes the audio data to identify the user's emotional state (e.g., impatience, joy, stress, etc.).

[1187] The terminal transmits the voice instruction in text format and the analysis results of the emotion engine to the server.

[1188] Step 4:

[1189] The server analyzes the received text data and emotional state using a natural language processing (NLP) engine.

[1190] The server identifies tasks based on the voice instructions, including "data extraction," "data processing," "data analysis," and "result deployment."

[1191] Step 5:

[1192] The server retrieves the original data file from the temporary data store.

[1193] The server extracts monthly sales data from the original data based on the analyzed voice instructions.

[1194] Step 6:

[1195] The server processes the extracted data, specifically by filtering, normalizing, and other operations to improve the quality of the data.

[1196] Step 7:

[1197] The server analyzes the processed data, for example by aggregating extracted sales data monthly to identify trends and patterns.

[1198] The server adjusts the analysis method and the result display format based on the user's emotional state.

[1199] Step 8:

[1200] The server converts the analysis results into the specified format (e.g., graphs or reports).

[1201] The server transmits the converted result to the terminal.

[1202] Step 9:

[1203] The device then displays the received analysis results to the user. For example, if the device detects that the user is in a hurry, it may present the results in a more concise and visually understandable format.

[1204] The device provides additional feedback and action suggestions (e.g., "Would you like to conduct a more detailed analysis?") depending on the user's emotions.

[1205] This series of steps allows users to experience responsive data processing and feedback that utilizes emotion recognition.

[1206] Example 2

[1207] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1208] Conventional voice-operated data processing systems have a limited scope for automating data processing based on voice instructions, and they are unable to provide feedback according to the user's emotional state. As a result, the system's response may be inappropriate when the user is stressed or when more detailed information is required.

[1209] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to input original data into the system, a means for converting voice instructions obtained from voice input into text data, a means for analyzing the converted text data and voice parameters and recognizing the user's emotional state, and a means for analyzing the text data and emotional data and identifying multiple tasks for data extraction, data processing, data analysis, and result development. This enables highly responsive data processing and feedback according to the user's emotional state.

[1210] A "user" is an entity that operates the system and inputs voice instructions and data.

[1211] "Original data" refers to the underlying information, such as sales data, and refers to the data files uploaded to the system.

[1212] "Voice input" refers to voice instructions given by the user through a microphone.

[1213] "Text data" refers to speech input converted into text format.

[1214] "Voice parameters" refer to features and information extracted from voice data, and are data that can be used for emotion analysis.

[1215] "Emotional state" is information indicating the user's emotions, such as impatience, joy, or stress.

[1216] "Data extraction" refers to the process of extracting necessary information from original data based on specified conditions.

[1217] "Data processing" refers to the process of filtering, normalizing, and other processes performed on extracted data to improve its quality.

[1218] "Data analysis" refers to the analytical process of using processed data to identify trends and patterns.

[1219] "Result development" refers to the process of converting the analysis results into a specified format (e.g., graphs or reports) and presenting them to the user.

[1220] "Feedback" refers to information or suggestions provided to users based on the results of data processing and analysis.

[1221] The present invention combines a voice-operated data processing system with an emotion engine, and is a system that realizes more intuitive and effective data processing by recognizing the user's emotional state and automatically adjusting data processing and providing feedback based on that state.

[1222] First, the user uploads raw data, such as sales data, to the system. The raw data is typically in CSV file format, and the device stores it in a temporary data store. The user uses a microphone to provide voice input. Voice instructions can be simple commands such as "Analyze sales data monthly." The device collects voice from the microphone and converts the voice data into text format using voice recognition software (e.g., a voice recognition API). Along with this converted text data, voice parameters are also obtained.

[1223] Next, the device analyzes the user's emotional state using a sentiment analysis engine (e.g., a sentiment analysis API). The analyzed emotional information is sent to a server, where a natural language processing (NLP) engine (e.g., an NLP model) analyzes the received text data and emotional data and identifies tasks. The identified tasks include "data extraction," "data processing," "data analysis," and "result deployment."

[1224] The server retrieves raw data from the temporary data store and extracts the required information based on voice commands. For example, sales data for a specified month or period may be extracted. The extracted data undergoes data processing operations such as filtering and normalization, and is then analyzed. For example, the analysis may aggregate sales data monthly to identify trends and patterns. The processed and analyzed data is then converted into a specified format (for example, a graph or report).

[1225] The server sends the calculated analysis results to the device, which then provides more appropriate feedback based on the user's emotional state as recognized by the emotion engine. For example, the device might display the results of data processing in an intuitive and easy-to-understand format to a stressed user, or display more detailed data and in-depth analysis results to a happy user. The server also provides appropriate feedback in response to voice commands and suggests the next course of action.

[1226] As a concrete example, consider a scenario in which a user issues a voice command to the system to "analyze monthly sales data" and uploads the original data. The device receives the voice command and the original data, converts the voice into text, and sends it to the server. The device then uses an emotion engine to analyze the user's emotions and recognizes, for example, that the user is feeling anxious. The server extracts, processes, and analyzes data based on the received voice command and emotional data, and generates a report that highlights only the most important points for anxious users. The device then displays the generated report to the user, who can view it in an easy-to-understand format.

[1227] Examples of prompt sentences include the following:

[1228] "The user issues a voice command to the system saying, 'Analyze monthly sales data,' and uploads the original data. The device receives the voice command and the original data, converts the voice into text, and sends it to the server. The device uses an emotion engine to analyze the user's emotions and recognizes, for example, that the user is feeling anxious. The server extracts, processes, and analyzes the data based on the received voice command and emotional data, and generates a report that highlights only the most important points for anxious users."

[1229] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1230] Step 1: User uploads data

[1231] Users upload raw data, such as sales data, to the system interface by dragging and dropping it. The input is a CSV-formatted data file, which the terminal saves in a temporary data store. The output is a CSV file saved in the temporary data store. The specific operation is to upload the file using a browser.

[1232] Step 2: The device receives voice instructions

[1233] The user speaks into the microphone, saying, "Analyze sales data monthly." The input is the user's voice command, and the device picks up this voice through the built-in microphone. The output is voice data (audio file). Specifically, the device uses the microphone to capture the voice.

[1234] Step 3: Your device converts your voice commands to text

[1235] The device uses voice recognition software (e.g., voice recognition API) to convert the voice data into text format. The input is voice data, and the output is text data such as "Analyze sales data monthly." Specifically, the device calls the voice recognition API and converts the voice data into text.

[1236] Step 4: The device performs sentiment analysis

[1237] The device passes the text data and voice parameters to an emotion analysis engine (e.g., emotion analysis API) to analyze the user's emotional state. The input is text data and voice parameters, and the output is the user's emotional state (e.g., impatience, joy). Specifically, the device calls the emotion analysis API and performs emotion analysis.

[1238] Step 5: Server identifies the task

[1239] The server passes text and sentiment data to a natural language processing (NLP) engine (e.g., an NLP model) to identify a task. The input is text and sentiment data, and the output is the identified task (e.g., data extraction, data processing, data analysis, or result deployment). Specifically, the server uses an NLP model to analyze the text and sentiment and identify the task.

[1240] Step 6: The server performs the data extraction

[1241] The server retrieves the raw data (CSV files) from a temporary data store and extracts the required information based on voice instructions. The input is the raw data and voice instructions, and the output is the extracted information (e.g., sales data for a specified month). Specifically, it uses a database query to extract data that matches the criteria.

[1242] Step 7: The server performs the data processing

[1243] The server processes the extracted data by filtering and normalizing it. The input is the extracted information, and the output is the processed information (e.g., clean data). Specifically, the program complements missing values ​​in the data and removes outliers.

[1244] Step 8: The server performs the data analysis

[1245] The server uses the processed data to perform analysis to identify trends and patterns. The input is the processed information, and the output is the analytical results (e.g., monthly sales trends). Specifically, it runs analytical algorithms and generates statistical information on the data.

[1246] Step 9: The server extracts the results

[1247] The server converts the analysis results into the specified format (e.g., graphs or reports) and sends them to the terminal. The input is the analysis results, and the output is the converted results (e.g., graphs, reports). Specifically, the results are visualized using a data visualization tool.

[1248] Step 10: Your device will display the results

[1249] The terminal displays the results sent from the server to the user. The input is the converted result, and the output is a screen display that is visually presented to the user. Specifically, it displays graphs and reports on the screen so that the user can check them.

[1250] Step 11: The server and device adjust according to the emotion.

[1251] The server adjusts the data feedback based on the emotional state recognized by the emotion engine, and the device displays the results in a format appropriate for the user. The input is the emotional state and the resulting data, and the output is adjusted feedback (e.g., concise display, detailed data). Specifically, the display content is dynamically changed according to the user's emotional state.

[1252] (Application example 2)

[1253] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1254] Today, users use voice commands to perform various data processing tasks. However, conventional systems do not take into account the user's emotional state, resulting in a lack of an intuitive and personal operating experience. As a result, users can experience inconvenience and stress when browsing and processing data. By solving this problem, we aim to improve user satisfaction and the effectiveness of data processing.

[1255] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting original data to the system to control the data processing system by voice operation, means for converting voice instructions obtained from the voice input into text data, means for analyzing the user's emotional state from the text data and voice data, means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result development, means for extracting information from the original data based on specified conditions, means for processing the extracted information, means for analyzing the processed information, means for converting information based on the analysis results and the user's emotional state into a specified format, and means for presenting appropriate feedback to the user based on the analysis results and the user's emotional state. This enables personalized data processing and feedback that takes the user's emotional state into consideration.

[1256] "Voice operation" is a technology that controls a system and executes instructions through user voice input.

[1257] A "data processing system" is a system that extracts, processes, and analyzes various types of data based on input data.

[1258] "Raw data" refers to the original, unprocessed data that is input into the system.

[1259] "Voice instructions" refer to instructions or commands given by the user through voice.

[1260] "Text data" is data in character format converted based on voice instructions.

[1261] An "emotional state" is a psychological state inferred from a user's voice or other input.

[1262] A "task" is a specific process or operation that a system performs.

[1263] "Data extraction" refers to extracting necessary information from original data based on specified conditions.

[1264] "Data processing" is the act of formatting or converting extracted data to make it easier to use.

[1265] "Data analysis" is the process of using processed data to find meaningful patterns and trends.

[1266] "Result presentation" refers to presenting the results of data analysis in a user-friendly format.

[1267] "Feedback" refers to the reaction or information that a system provides to a user.

[1268] A "natural language processing engine" is a technology for analyzing text data and understanding its meaning and context.

[1269] An "emotion engine" is a technology for analyzing a user's emotional state from their voice and data.

[1270] The present invention is a system that combines a voice-operated data processing system with an emotion engine, which recognizes the user's emotional state and provides automatic adjustments and feedback to data processing based on that state. The system of the present invention is implemented using the following hardware and software.

[1271] Hardware and software used

[1272] Hardware

[1273] Smartphones (e.g. iPhone, Android devices)

[1274] microphone

[1275] server

[1276] software

[1277] Google Cloud Speech-to-Text API

[1278] IBM Watson Tone Analyzer

[1279] Google Natural Language API

[1280] Food delivery backend server

[1281] System configuration

[1282] 1. Acquiring voice instructions

[1283] The user speaks into the smartphone's microphone, saying something like "I want to order a pizza." This voice input is captured by the smartphone. The captured voice data is converted into text data using the Google Cloud Speech-to-Text API.

[1284] 2. Emotion analysis

[1285] The converted text data and the original voice data are then analyzed using IBM Watson Tone Analyzer to determine the user's emotional state, for example, whether the user is relaxed or anxious.

[1286] 3. Analysis of voice instructions

[1287] The parsed text data is then passed through the Google Natural Language API to identify the tasks the system should perform (e.g., display a menu, process an order), allowing it to optimally respond based on the user's emotional state and voice commands.

[1288] 4. Food delivery order fulfillment

[1289] The food delivery backend server retrieves menu information from a database based on the user's emotional state and voice instructions, and sends it to the smartphone. For example, if the user is tired, a simple menu is displayed, and if the user is relaxed, a detailed menu is displayed.

[1290] 5. Personalized feedback

[1291] Feedback that takes into account the user's emotional state is provided via smartphone, such as suggestions like "Today's recommendation is Margherita pizza."

[1292] Specific examples

[1293] A user speaks to their smartphone and says, "I'd like to order a pizza." The smartphone captures the voice data and converts it to text using the Google Cloud Speech-to-Text API. The system then uses IBM Watson Tone Analyzer to analyze the user's emotional state. For example, if the system detects that the user is tired, it might suggest, "How about a Margherita pizza?"

[1294] Prompt Sentence Examples

[1295] "If a user says, 'I want to order a pizza':

[1296] "User wants to order pizza. Emotional state is relaxed."

[1297] "Yes, this is the menu we recommend for users in a relaxed state."

[1298] The system allows users to have a personalized ordering experience that is tailored to their emotional state, resulting in increased user satisfaction and a more intuitive experience.

[1299] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1300] Step 1:

[1301] The user speaks into the smartphone's microphone. For example, they say, "I'd like to order a pizza." The input here is the user's voice, and the output is the voice data captured by the smartphone. The smartphone sends this voice data to the Google Cloud Speech-to-Text API, which converts it into text.

[1302] Step 2:

[1303] The smartphone uses the Google Cloud Speech-to-Text API to convert the acquired voice data into text data. The converted text is in the form of "I would like to order a pizza." The input of this process is voice data, and the output is text data.

[1304] Step 3:

[1305] The smartphone uses IBM Watson Tone Analyzer to analyze the user's emotional state from the text data and the original voice data. For example, it determines whether the user is relaxed or tired. The input is text data and voice data, and the output is data indicating the user's emotional state.

[1306] Step 4:

[1307] The smartphone sends text data and emotion data to the Google Natural Language API for analysis. The analyzed data is used to identify the tasks the system should perform (e.g., display a menu or process an order). The inputs for this step are text data and emotion data, and the output is the identified tasks.

[1308] Step 5:

[1309] The server accesses the food delivery backend system and retrieves menu information from the database based on the user's emotional state and voice instructions. For example, if the user is tired, a concise menu is displayed. The input is the specified task and the user's emotional state, and the output is the adjusted menu information.

[1310] Step 6:

[1311] The server sends the adjusted menu information to the smartphone, which then displays this information to the user, for example, "Today's special is Margherita pizza." The input is the adjusted menu information, and the output is the user's visual feedback.

[1312] Step 7:

[1313] The user makes a selection from the displayed menu and confirms the order by voice, for example, "I'd like to order a Margherita pizza." The smartphone converts this speech to text, again using the Google Cloud Speech-to-Text API. The input is the user's voice, and the output is text data.

[1314] Step 8:

[1315] The smartphone sends a final confirmation text to the food delivery backend system to confirm the order. The server receives the order and starts processing it. The input is the order confirmation text, and the output is the food delivery order data.

[1316] This process allows users to have a personalized ordering experience that is tailored to their emotional state, resulting in increased user satisfaction and a more intuitive experience.

[1317] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1318] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1319] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1320] [Fourth embodiment]

[1321] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1322] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1323] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1324] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1325] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1326] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1327] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1328] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1329] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1330] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1331] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1332] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1333] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1334] The present invention is a system that controls a data processing system by voice operation, and allows a user to input original data into the system and give voice instructions to fully automatically extract, process, analyze, and develop the results of the data. This system is implemented by the following program processing.

[1335] System configuration

[1336] 1. Data Entry

[1337] The user uploads raw data (e.g., CSV file) such as sales data to the system.

[1338] The terminal receives the original data from the user and stores it in a temporary data store.

[1339] 2. Acquiring voice instructions

[1340] The user issues a voice command into the microphone, such as "Analyze sales data monthly."

[1341] The terminal captures audio from a microphone, converts the audio data into text format in real time, and sends it to a server.

[1342] 3. Analysis of voice instructions

[1343] The server passes the text data of the voice instructions to a natural language processing engine, which analyzes the content of the voice instructions. This analysis identifies multiple tasks: data extraction, data processing, data analysis, and result presentation.

[1344] Execution of each task

[1345] Data Extraction

[1346] The server retrieves the raw data from the temporary data store and extracts the required information based on voice instructions, such as sales data for a specified month or period.

[1347] Data Processing

[1348] The server processes the extracted data, specifically by using techniques such as filtering and normalization to improve the quality of the data.

[1349] Data analysis

[1350] The server analyzes the processed data, for example, aggregating extracted sales data monthly to identify trends and patterns.

[1351] Expanding the results

[1352] The server converts the analysis results into the specified format (e.g., graphs or reports).

[1353] The terminal presents the results sent from the server to the user, for example displaying bar graphs or reports showing sales trends.

[1354] Specific examples

[1355] 1. Inputting the original data

[1356] A user uploads a CSV file of sales data into the system.

[1357] The device will save this file and accept microphone operations.

[1358] 2. Enter voice commands

[1359] The user issues a voice command such as "Analyze sales data monthly."

[1360] The device converts this speech into text and sends it to the server.

[1361] 3. Analysis of voice instructions

[1362] The server analyzes the voice instructions and identifies tasks for data extraction, processing, analysis, and deployment.

[1363] 4. Execute the task

[1364] The server performs a series of data extraction, data processing, and data analysis, and generates the results as a graph.

[1365] The device displays the generated graphs to the user, providing a visual representation of monthly sales trends and patterns.

[1366] This system allows users to perform complex data processing with simple voice commands, enabling efficient data analysis regardless of the user's skill level. Furthermore, the automation of each task significantly reduces work time.

[1367] The processing flow will be explained below.

[1368] Step 1:

[1369] The user uploads a CSV file of sales data to the system.

[1370] The device verifies the uploaded file and saves it in a temporary data store.

[1371] Step 2:

[1372] The user issues a voice command into the microphone saying, "Analyze sales data monthly."

[1373] The device collects audio from a microphone and converts the audio data into text format.

[1374] Step 3:

[1375] The terminal transmits the converted text data to the server.

[1376] The server analyzes the received text data using a natural language processing (NLP) engine and identifies tasks, such as data extraction, data processing, data analysis, and results presentation.

[1377] Step 4:

[1378] The server retrieves the original data file from the temporary data store.

[1379] The server extracts specified information, such as monthly sales data, from the original data based on the analyzed voice instructions.

[1380] Step 5:

[1381] The server obtains the extracted data for data processing.

[1382] As part of the data processing, the server performs operations such as filtering and normalization to improve the quality of the data.

[1383] Step 6:

[1384] The server obtains the processed data for data analysis.

[1385] The server uses the obtained data to perform a specified analysis (e.g., monthly sales trend analysis) and identify trends and patterns in the data.

[1386] Step 7:

[1387] The server converts the analysis results into the specified format (e.g., graphs or reports).

[1388] The server transmits the converted result to the terminal.

[1389] Step 8:

[1390] The terminal displays the received analysis results to the user.

[1391] Users can view graphs and reports displayed on their devices to understand sales trends and patterns.

[1392] This series of steps allows users, regardless of their skill level, to efficiently perform complex data processing using voice instructions.

[1393] Example 1

[1394] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1395] Conventional data processing systems require users to manually perform numerous steps to extract, process, analyze, and develop results, often resulting in complex operations and requiring specialized knowledge. Furthermore, there are few fully automated voice-operated data processing systems, placing a significant burden on users. Furthermore, there is a lack of technology that can accurately analyze voice commands and efficiently execute each task automatically.

[1396] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1397] In this invention, the server includes means for converting voice instructions obtained from voice input into text data, means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result development, and means for processing the extracted information, thereby enabling the user to fully automatically extract, process, analyze, and develop results using only voice operations.

[1398] "Voice operation" is an operation method in which the user controls various functions of the system using voice.

[1399] A "data processing system" is a system for inputting, extracting, processing, analyzing, and expanding data.

[1400] "Original data" refers to the initial data entered into the system, including formats such as CSV files.

[1401] "Voice input" is a method in which a user supplies voice to the system using a microphone or the like.

[1402] "Voice instructions" are instructions given to the system by voice input.

[1403] "Text data" is data in the form of a character string converted from voice input.

[1404] "Analysis" is the act of understanding input data (here, text data) and grasping its meaning.

[1405] "Data extraction" is the process of extracting only the necessary information from the original data based on specific conditions.

[1406] "Data processing" is the process of filtering and normalizing extracted data to improve its quality and usefulness.

[1407] "Data analysis" is the process of performing calculations and aggregations on processed data to find useful information and patterns.

[1408] "Developing results" refers to summarizing the analysis results in an easy-to-read format and presenting them to the user.

[1409] "Speech recognition software" is software for converting voice data into text data.

[1410] A "natural language processing engine" is a software technology for analyzing text data and understanding the meaning of human language.

[1411] A "task" is a discrete operation performed by the system, such as data extraction, data processing, data analysis, and results presentation.

[1412] The present invention relates to a method and system for controlling a data processing system by voice operation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS ... will be described in detail below with reference to preferred embodiments thereof.

[1413] System configuration

[1414] Hardware used

[1415] 1. Terminal: A device operated by a user (such as a personal computer, smartphone, or tablet)

[1416] 2. Server: A computer for processing data

[1417] 3. Microphone: An input device for obtaining voice instructions

[1418] Software used

[1419] 1. Speech recognition software: Used to convert voice data into text data (e.g., Google Cloud Speech-to-Text, IBM Watson Speech to Text)

[1420] 2. Natural language processing engines: used to analyze text data (e.g., Google Cloud Natural Language, Microsoft Azure Text Analytics)

[1421] 3. Data processing tools: Programming languages ​​and libraries (e.g., Python, Pandas, NumPy) for processing the extracted data.

[1422] 4. Data analysis tools: Programming languages ​​and libraries for analyzing the processed data (e.g., Python, Scikit-learn, Matplotlib, Seaborn)

[1423] System Operation

[1424] User Operation

[1425] The user uploads the original data (e.g., a CSV file of sales data) from their terminal to the system. To do this, the user selects the file using the browser's file selection dialog and presses the upload button. Next, the user gives voice instructions into the microphone. For example, they can input instructions in voice format such as "Analyze the sales data monthly."

[1426] Device Features

[1427] The device receives files uploaded by users and temporarily stores them in local storage. The device also captures voice data from the microphone and converts it into text format in real time using voice recognition software. This converted text data is then sent to the server for further processing.

[1428] Server Processing

[1429] The server passes the text data sent from the speech recognition software to the natural language processing engine for analysis. Based on the analysis results, the server identifies the following tasks: data extraction, data processing, data analysis, and result distribution. The server executes these tasks sequentially to generate the final analysis results.

[1430] For example, in the data extraction stage, the server uses Python's Pandas library to read the original data CSV file and extract the specified monthly sales data. In the data processing stage, it fills in missing values ​​with zeros and filters specific columns. In the data analysis stage, it calculates the total sales for each month and analyzes the increase or decrease. In the final results development stage, it creates a graph using Matplotlib and saves it as an image file.

[1431] Presentation of results

[1432] The generated analysis results (e.g., graph images and reports) are sent from the server to the terminal. The terminal visually presents the results to the user. Specifically, graph images can be displayed on the browser.

[1433] Specific examples

[1434] Next, a specific example will be given.

[1435] 1. Upload the original data

[1436] A user uploads a file called sales_data.csv.

[1437] The device will save this file in the / tmp / uploads / directory.

[1438] 2. Enter voice commands

[1439] The user issues a voice command such as "Analyze sales data monthly."

[1440] The device converts the speech into text and sends it to the server.

[1441] 3. Analysis of voice instructions

[1442] The server analyzes the voice instructions and identifies tasks for data extraction, processing, analysis, and deployment.

[1443] 4. Task execution and results presentation

[1444] The server performs a series of data extraction, data processing, and data analysis, and generates the results as a graph.

[1445] The device displays the generated graphs to the user, providing a visual representation of monthly sales trends and patterns.

[1446] Prompt Sentence Examples

[1447] "Analyze sales data monthly."

[1448] "Compare sales trends this month and last month."

[1449] "What are the top 5 product sales figures?"

[1450] This allows users to easily perform advanced data analysis using only voice commands, enabling them to gain useful insights from large amounts of data and complete processes efficiently, even without specific skills.

[1451] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1452] Step 1: Data entry

[1453] The user uploads a CSV file of sales data (e.g., sales_data.csv) to the system.

[1454] The terminal receives the CSV file uploaded by the user and temporarily stores it in local storage (e.g., / tmp / uploads / ).

[1455] Input: A CSV file selected by the user in the browser.

[1456] Output: CSV file saved in temporary directory.

[1457] Specific operation: The device reads the file selected from the browser's file selection dialog using the JavaScript FileReader object and saves it in a temporary storage directory.

[1458] Step 2: Getting voice instructions

[1459] The user issues a voice command into the microphone saying, "Analyze sales data monthly."

[1460] The device receives voice data from the microphone and converts it into text using real-time voice recognition software (e.g., Google Cloud Speech-to-Text), which then sends the text data to a server.

[1461] Input: User's voice command.

[1462] Output: Spoken instructions converted to text (e.g., "Analyze sales data monthly").

[1463] Specific operation: The device acquires microphone data using the navigator.mediaDevices.getUserMedia API, sends the audio data to the speech recognition API, and sends the converted text data to the server via WebSocket.

[1464] Step 3: Analyzing voice instructions

[1465] The server passes the text data obtained by speech recognition to a natural language processing engine (e.g., Google Cloud Natural Language) and analyzes the voice instructions.

[1466] Input: Text-based voice instructions.

[1467] Output: Parsed instructions (e.g. "Monthly analysis of sales data").

[1468] Specific operations: The server sends an API request to the natural language processing engine, receives the analysis results of the text data, and identifies tasks such as data extraction, data processing, data analysis, and result presentation.

[1469] Step 4: Data extraction

[1470] The server reads the original data from the temporarily saved CSV file and extracts the necessary information based on voice instructions.

[1471] Input: Temporarily saved CSV file, parsed instructions.

[1472] Output: The extracted data (e.g. sales data for a specified month).

[1473] Specific operation: The server uses Python's Pandas library to read the CSV file and extract sales data for the specified period (e.g., df[df['month'] == '2023-01']).

[1474] Step 5: Data processing

[1475] The server processes the extracted data, for example, by handling missing values, filtering, normalizing, etc.

[1476] Input: The extracted data.

[1477] Output: Processed data (e.g., sales data with missing values ​​filled with zeros and filtered).

[1478] Specific operation: The server uses the Pandas library to fill missing values ​​with zeros (e.g., df.fillna(0, inplace=True)), or extracts only specific columns to create a new data frame (e.g., df[['Sales', 'Month']]).

[1479] Step 6: Data analysis

[1480] The server then performs analytical processing on the processed data, such as aggregating monthly sales figures to identify trends and patterns.

[1481] Input: Processed data.

[1482] Output: Analysis results (e.g. monthly sales totals, sales increase / decrease).

[1483] Specific operation: The server uses Pandas and Scikit-learn to calculate the total sales for each month (e.g., df.groupby('Month')['Sales'].sum()) and calculates the increase or decrease in sales (e.g., df.pct_change()).

[1484] Step 7: Deploying the results

[1485] The server converts the analysis results into the specified format (e.g., graphs or reports). It uses Matplotlib to generate graphs and saves them as image files.

[1486] The terminal presents the results (e.g., graph images) sent from the server to the user.

[1487] Input: Analysis results.

[1488] Output: Visualized results (e.g., graph images, PDF reports).

[1489] Specific operation: The server creates a graph using the Matplotlib library (e.g., df.plot(kind='bar')) and saves it as an image (e.g., plt.savefig(' / tmp / results / sales_trend.png')). The terminal saves the generated image file as an HTML Present it to the user using tags.

[1490] (Application example 1)

[1491] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1492] In conventional data processing systems, the process of manually extracting, processing, and analyzing data is complex and requires a lot of time and effort. It is also difficult to properly manage inventory and issue shipping instructions in real time at logistics centers, resulting in a decrease in the efficiency of logistics operations. To solve these problems, the present invention aims to provide a system that automates logistics management by efficiently processing data through voice operation.

[1493] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1494] In this invention, the server includes means for inputting original data into the system to control the data processing system by voice operation, means for converting voice instructions obtained from the voice input into text data, and means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result deployment, thereby enabling the user to perform inventory management and shipping instructions based on voice instructions and receive feedback in real time.

[1495] "Voice operation" refers to an operation in which a user gives instructions to a system using a voice input device such as a microphone.

[1496] A "data processing system" is a computer system that performs a series of processes, from inputting original data to receiving and analyzing voice instructions, executing tasks, and presenting results.

[1497] "Original data" refers to the initial data (e.g., CSV file) provided by the user for processing by the system.

[1498] A "voice instruction" is a specific command or request that the user issues into the microphone (e.g., "Analyze sales data monthly").

[1499] "Text data" refers to data in the form of a character string obtained by converting voice input.

[1500] A "task" is a specific unit of processing performed by the system, such as data extraction, data processing, data analysis, and result deployment.

[1501] "Data extraction" is the process of extracting necessary information from original data based on specified conditions.

[1502] "Data processing" refers to the process of organizing extracted information using methods such as filtering and normalization to improve its quality.

[1503] "Data analysis" is the process of finding trends and patterns based on processed information.

[1504] "Result development" refers to the process of converting the analysis results into a specified format (e.g., graph, report, etc.) and presenting them to the user.

[1505] "Feedback" refers to providing the system with real-time responses and results to user-generated voice commands.

[1506] "Logistics management means" refers to functions that allow inventory management and shipping instructions to be performed based on voice commands, and provide feedback in real time.

[1507] The system of the present invention is designed to efficiently perform data processing and logistics management through voice operations. This system is realized by the following configuration and processing.

[1508] System configuration

[1509] 1. Device:

[1510] Hardware: Smartphones and tablets with microphones

[1511] Software: Speech recognition library (e.g., SpeechRecognition)

[1512] 2. Server:

[1513] Hardware: Cloud-based server infrastructure

[1514] Software: Natural language processing engine, data processing program (e.g., Pandas, Matplotlib)

[1515] 3. Data Store:

[1516] A database where raw data (e.g., inventory data, shipping data) is stored

[1517] Program processing

[1518] The user inputs voice instructions using a microphone into the terminal, such as "Tell me the inventory status" or "Ship 50 units of product A" that are related to logistics management.

[1519] The device receives this voice instruction and converts it into text data using a speech recognition library, which is then sent to the server as a request.

[1520] The server passes the received text data to a natural language processing engine for analysis. This analysis identifies multiple tasks, such as data extraction, data processing, and data analysis. For example, from an instruction such as "Ship 50 units of product A," updating inventory data and generating shipping instructions are identified as tasks.

[1521] The server extracts the necessary data from the database based on the specified task, then performs processing such as filtering and normalization on the extracted data, and analyzes the processed data to generate the results.

[1522] The generated results are converted into a specified format (e.g., inventory list, shipping report, graph, etc.) and sent to the terminal.

[1523] The terminal displays the results sent from the server to the user, allowing the user to receive feedback on inventory status and shipping instructions in real time.

[1524] Examples of specific examples and prompts

[1525] Examples:

[1526] The user issues a voice command to their smartphone saying, "Ship 50 units of product A."

[1527] The system converts this speech into text and analyzes it on the server.

[1528] The server updates inventory data and generates shipping instructions.

[1529] The terminal provides real-time feedback to the user, such as "Shipping instructions: 50 units of product A have been shipped."

[1530] Example prompt sentence:

[1531] User: "Ship 50 units of product A."

[1532] Example of system response: "Shipping instructions: 50 units of product A have been shipped."

[1533] As described above, the system of the present invention enables efficient data processing and logistics management through voice operation, and provides real-time feedback.

[1534] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1535] Step 1:

[1536] The user inputs voice instructions using a microphone, which include specific commands or requests (e.g., "Ship 50 units of product A").

[1537] Input: User's voice data

[1538] Output: Audio file captured from microphone

[1539] Step 2:

[1540] The device uses a speech recognition library to convert the captured voice data into text data in real time. This conversion process is performed using a speech recognition engine (e.g., SpeechRecognition).

[1541] Input: Audio file

[1542] Output: Text-based voice instructions (e.g., "Ship 50 units of product A")

[1543] Step 3:

[1544] The terminal sends the converted text data to the server as a request in the form of an HTTP request.

[1545] Input: Text-based voice instructions

[1546] Output: HTTP request

[1547] Step 4:

[1548] The server passes the received text data to a natural language processing engine for analysis. The analysis identifies tasks such as data extraction, data processing, and data analysis. Specifically, from the command "Ship 50 units of product A," it identifies the tasks of updating inventory data and generating shipping instructions.

[1549] Input: Text-based voice instructions

[1550] Output: Identified tasks (e.g., updating inventory data, generating shipping instructions)

[1551] Step 5:

[1552] The server extracts the required data from the database based on the specified task, using SQL queries.

[1553] Input: Task Instructions

[1554] Output: Extracted data (e.g. inventory data)

[1555] Step 6:

[1556] The server processes the extracted data, such as filtering and normalization, using a data frame manipulation library (e.g., Pandas).

[1557] Input: Extracted data

[1558] Output: Processed data

[1559] Step 7:

[1560] The server analyzes the processed data and generates results, where data analysis tools (e.g., Matplotlib) are used.

[1561] Input: Processed data

[1562] Output: Analysis results (e.g. shipping instructions)

[1563] Step 8:

[1564] The server converts the generated results into the specified format (e.g., report, graph, etc.) using a data visualization library (e.g., Matplotlib).

[1565] Input: Analysis results

[1566] Output: Results in the specified format (e.g. graph, report)

[1567] Step 9:

[1568] The server transmits the generated results to the terminal.

[1569] Input: Result in the specified format

[1570] Output: HTTP response

[1571] Step 10:

[1572] The terminal then presents the received results to the user, who can receive real-time feedback on inventory information and shipping instructions.

[1573] Input: HTTP response (result in the specified format)

[1574] Output: Feedback displayed on the user screen (e.g., "Shipping instructions: 50 units of product A have been shipped.")

[1575] This process enables efficient data processing and logistics management through voice control, providing real-time feedback.

[1576] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1577] The present invention combines a voice-operated data processing system with an emotion engine, and is a system that realizes more intuitive and effective data processing by recognizing the user's emotional state and automatically adjusting data processing and providing feedback based on that state.

[1578] System configuration

[1579] 1. Data Entry

[1580] The user uploads raw data (e.g., CSV file) such as sales data to the system.

[1581] The terminal receives the original data from the user and stores it in a temporary data store.

[1582] 2. Acquiring voice instructions

[1583] The user issues a voice command into the microphone, such as "Analyze sales data monthly."

[1584] The device collects audio from a microphone and converts the audio data into text format.

[1585] 3. Voice command analysis and emotion recognition

[1586] The device uses an emotion engine to analyze the user's emotions based on the converted text data as well as the voice data.

[1587] The server passes the received text data and the emotional state information obtained from the emotion engine to the natural language processing (NLP) engine, and identifies tasks, including data extraction, data processing, data analysis, and result deployment.

[1588] Execution of each task

[1589] Data Extraction

[1590] The server retrieves the original data file from the temporary data store and extracts the required information based on voice instructions, such as sales data for a specified month or period.

[1591] Data Processing

[1592] The server retrieves the extracted data for processing, specifically, performing operations such as filtering and normalization to improve the data quality.

[1593] Data analysis

[1594] The server retrieves the processed data for data analysis, for example, aggregating the extracted sales data monthly to identify trends and patterns.

[1595] Expanding the results

[1596] The server converts the analysis results into the specified format (e.g., graphs or reports).

[1597] The terminal presents the results sent from the server to the user.

[1598] Applications of Emotion Recognition

[1599] Once the emotion engine has analyzed the user's emotions, it can adjust the system's behavior based on the resulting emotional state as follows:

[1600] If the emotion engine recognizes that the user is feeling stressed, the server will provide the results of the data processing in a more intuitive and easy-to-understand format (for example, more concise graphs or reports explained in simpler language).

[1601] If the emotion engine recognizes that the user is happy, the device will display more detailed data analysis results and in-depth analysis results.

[1602] The server responds to the voice prompts by providing appropriate feedback to the user and suggesting the next action (e.g., "Would you like to perform a more detailed analysis?").

[1603] Specific examples

[1604] 1. The user issues a voice command to the system saying, "Analyze sales data monthly," and uploads the original data.

[1605] 2. The device receives the voice instructions and the original data, converts the voice into text, and sends it to the server.

[1606] 3. The device uses an emotion engine to analyze the user's emotions and recognize, for example, that the user is feeling anxious.

[1607] 4. Based on the voice instructions and emotional data received by the server, data is extracted, processed, and analyzed, and a report is generated that highlights only the most important points for impatient users.

[1608] 5. The terminal displays the generated report to the user, who can then view the report in an easy-to-understand format.

[1609] The system is able to take into account the user's emotional state and provide more responsive and user-friendly data processing and feedback.

[1610] The processing flow will be explained below.

[1611] Step 1:

[1612] The user uploads a CSV file of sales data to the system.

[1613] The device verifies the uploaded file and saves it in a temporary data store.

[1614] Step 2:

[1615] The user issues a voice command into the microphone saying, "Analyze sales data monthly."

[1616] The device collects audio from a microphone and converts the audio data into text format.

[1617] Step 3:

[1618] The terminal inputs the voice data to the emotion engine together with the voice instruction converted into text data.

[1619] The emotion engine analyzes the audio data to identify the user's emotional state (e.g., impatience, joy, stress, etc.).

[1620] The terminal transmits the voice instruction in text format and the analysis results of the emotion engine to the server.

[1621] Step 4:

[1622] The server analyzes the received text data and emotional state using a natural language processing (NLP) engine.

[1623] The server identifies tasks based on the voice instructions, including "data extraction," "data processing," "data analysis," and "result deployment."

[1624] Step 5:

[1625] The server retrieves the original data file from the temporary data store.

[1626] The server extracts monthly sales data from the original data based on the analyzed voice instructions.

[1627] Step 6:

[1628] The server processes the extracted data, specifically by filtering, normalizing, and other operations to improve the quality of the data.

[1629] Step 7:

[1630] The server analyzes the processed data, for example by aggregating extracted sales data monthly to identify trends and patterns.

[1631] The server adjusts the analysis method and the result display format based on the user's emotional state.

[1632] Step 8:

[1633] The server converts the analysis results into the specified format (e.g., graphs or reports).

[1634] The server transmits the converted result to the terminal.

[1635] Step 9:

[1636] The device then displays the received analysis results to the user. For example, if the device detects that the user is in a hurry, it may present the results in a more concise and visually understandable format.

[1637] The device provides additional feedback and action suggestions (e.g., "Would you like to conduct a more detailed analysis?") depending on the user's emotions.

[1638] This series of steps allows users to experience responsive data processing and feedback that utilizes emotion recognition.

[1639] Example 2

[1640] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1641] Conventional voice-operated data processing systems have a limited scope for automating data processing based on voice instructions, and they are unable to provide feedback according to the user's emotional state. As a result, the system's response may be inappropriate when the user is stressed or when more detailed information is required.

[1642] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for a user to input original data into the system, a means for converting voice instructions obtained from voice input into text data, a means for analyzing the converted text data and voice parameters and recognizing the user's emotional state, and a means for analyzing the text data and emotional data and identifying multiple tasks for data extraction, data processing, data analysis, and result development. This enables highly responsive data processing and feedback according to the user's emotional state.

[1643] A "user" is an entity that operates the system and inputs voice instructions and data.

[1644] "Original data" refers to the underlying information, such as sales data, and refers to the data files uploaded to the system.

[1645] "Voice input" refers to voice instructions given by the user through a microphone.

[1646] "Text data" refers to speech input converted into text format.

[1647] "Voice parameters" refer to features and information extracted from voice data, and are data that can be used for emotion analysis.

[1648] "Emotional state" is information indicating the user's emotions, such as impatience, joy, or stress.

[1649] "Data extraction" refers to the process of extracting necessary information from original data based on specified conditions.

[1650] "Data processing" refers to the process of filtering, normalizing, and other processes performed on extracted data to improve its quality.

[1651] "Data analysis" refers to the analytical process of using processed data to identify trends and patterns.

[1652] "Result development" refers to the process of converting the analysis results into a specified format (e.g., graphs or reports) and presenting them to the user.

[1653] "Feedback" refers to information or suggestions provided to users based on the results of data processing and analysis.

[1654] The present invention combines a voice-operated data processing system with an emotion engine, and is a system that realizes more intuitive and effective data processing by recognizing the user's emotional state and automatically adjusting data processing and providing feedback based on that state.

[1655] First, the user uploads raw data, such as sales data, to the system. The raw data is typically in CSV file format, and the device stores it in a temporary data store. The user uses a microphone to provide voice input. Voice instructions can be simple commands such as "Analyze sales data monthly." The device collects voice from the microphone and converts the voice data into text format using voice recognition software (e.g., a voice recognition API). Along with this converted text data, voice parameters are also obtained.

[1656] Next, the device analyzes the user's emotional state using a sentiment analysis engine (e.g., a sentiment analysis API). The analyzed emotional information is sent to a server, where a natural language processing (NLP) engine (e.g., an NLP model) analyzes the received text data and emotional data and identifies tasks. The identified tasks include "data extraction," "data processing," "data analysis," and "result deployment."

[1657] The server retrieves raw data from the temporary data store and extracts the required information based on voice commands. For example, sales data for a specified month or period may be extracted. The extracted data undergoes data processing operations such as filtering and normalization, and is then analyzed. For example, the analysis may aggregate sales data monthly to identify trends and patterns. The processed and analyzed data is then converted into a specified format (for example, a graph or report).

[1658] The server sends the calculated analysis results to the device, which then provides more appropriate feedback based on the user's emotional state as recognized by the emotion engine. For example, the device might display the results of data processing in an intuitive and easy-to-understand format to a stressed user, or display more detailed data and in-depth analysis results to a happy user. The server also provides appropriate feedback in response to voice commands and suggests the next course of action.

[1659] As a concrete example, consider a scenario in which a user issues a voice command to the system to "analyze monthly sales data" and uploads the original data. The device receives the voice command and the original data, converts the voice into text, and sends it to the server. The device then uses an emotion engine to analyze the user's emotions and recognizes, for example, that the user is feeling anxious. The server extracts, processes, and analyzes data based on the received voice command and emotional data, and generates a report that highlights only the most important points for anxious users. The device then displays the generated report to the user, who can view it in an easy-to-understand format.

[1660] Examples of prompt sentences include the following:

[1661] "The user issues a voice command to the system saying, 'Analyze monthly sales data,' and uploads the original data. The device receives the voice command and the original data, converts the voice into text, and sends it to the server. The device uses an emotion engine to analyze the user's emotions and recognizes, for example, that the user is feeling anxious. The server extracts, processes, and analyzes the data based on the received voice command and emotional data, and generates a report that highlights only the most important points for anxious users."

[1662] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1663] Step 1: User uploads data

[1664] Users upload raw data, such as sales data, to the system interface by dragging and dropping it. The input is a CSV-formatted data file, which the terminal saves in a temporary data store. The output is a CSV file saved in the temporary data store. The specific operation is to upload the file using a browser.

[1665] Step 2: The device receives voice instructions

[1666] The user speaks into the microphone, saying, "Analyze sales data monthly." The input is the user's voice command, and the device picks up this voice through the built-in microphone. The output is voice data (audio file). Specifically, the device uses the microphone to capture the voice.

[1667] Step 3: Your device converts your voice commands to text

[1668] The device uses voice recognition software (e.g., voice recognition API) to convert the voice data into text format. The input is voice data, and the output is text data such as "Analyze sales data monthly." Specifically, the device calls the voice recognition API and converts the voice data into text.

[1669] Step 4: The device performs sentiment analysis

[1670] The device passes the text data and voice parameters to an emotion analysis engine (e.g., emotion analysis API) to analyze the user's emotional state. The input is text data and voice parameters, and the output is the user's emotional state (e.g., impatience, joy). Specifically, the device calls the emotion analysis API and performs emotion analysis.

[1671] Step 5: Server identifies the task

[1672] The server passes text and sentiment data to a natural language processing (NLP) engine (e.g., an NLP model) to identify a task. The input is text and sentiment data, and the output is the identified task (e.g., data extraction, data processing, data analysis, or result deployment). Specifically, the server uses an NLP model to analyze the text and sentiment and identify the task.

[1673] Step 6: The server performs the data extraction

[1674] The server retrieves the raw data (CSV files) from a temporary data store and extracts the required information based on voice instructions. The input is the raw data and voice instructions, and the output is the extracted information (e.g., sales data for a specified month). Specifically, it uses a database query to extract data that matches the criteria.

[1675] Step 7: The server performs the data processing

[1676] The server processes the extracted data by filtering and normalizing it. The input is the extracted information, and the output is the processed information (e.g., clean data). Specifically, the program complements missing values ​​in the data and removes outliers.

[1677] Step 8: The server performs the data analysis

[1678] The server uses the processed data to perform analysis to identify trends and patterns. The input is the processed information, and the output is the analytical results (e.g., monthly sales trends). Specifically, it runs analytical algorithms and generates statistical information on the data.

[1679] Step 9: The server extracts the results

[1680] The server converts the analysis results into the specified format (e.g., graphs or reports) and sends them to the terminal. The input is the analysis results, and the output is the converted results (e.g., graphs, reports). Specifically, the results are visualized using a data visualization tool.

[1681] Step 10: Your device will display the results

[1682] The terminal displays the results sent from the server to the user. The input is the converted result, and the output is a screen display that is visually presented to the user. Specifically, it displays graphs and reports on the screen so that the user can check them.

[1683] Step 11: The server and device adjust according to the emotion.

[1684] The server adjusts the data feedback based on the emotional state recognized by the emotion engine, and the device displays the results in a format appropriate for the user. The input is the emotional state and the resulting data, and the output is adjusted feedback (e.g., concise display, detailed data). Specifically, the display content is dynamically changed according to the user's emotional state.

[1685] (Application example 2)

[1686] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1687] Today, users use voice commands to perform various data processing tasks. However, conventional systems do not take into account the user's emotional state, resulting in a lack of an intuitive and personal operating experience. As a result, users can experience inconvenience and stress when browsing and processing data. By solving this problem, we aim to improve user satisfaction and the effectiveness of data processing.

[1688] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting original data to the system to control the data processing system by voice operation, means for converting voice instructions obtained from the voice input into text data, means for analyzing the user's emotional state from the text data and voice data, means for analyzing the text data and identifying multiple tasks for data extraction, data processing, data analysis, and result development, means for extracting information from the original data based on specified conditions, means for processing the extracted information, means for analyzing the processed information, means for converting information based on the analysis results and the user's emotional state into a specified format, and means for presenting appropriate feedback to the user based on the analysis results and the user's emotional state. This enables personalized data processing and feedback that takes the user's emotional state into consideration.

[1689] "Voice operation" is a technology that controls a system and executes instructions through user voice input.

[1690] A "data processing system" is a system that extracts, processes, and analyzes various types of data based on input data.

[1691] "Raw data" refers to the original, unprocessed data that is input into the system.

[1692] "Voice instructions" refer to instructions or commands given by the user through voice.

[1693] "Text data" is data in character format converted based on voice instructions.

[1694] An "emotional state" is a psychological state inferred from a user's voice or other input.

[1695] A "task" is a specific process or operation that a system performs.

[1696] "Data extraction" refers to extracting necessary information from original data based on specified conditions.

[1697] "Data processing" is the act of formatting or converting extracted data to make it easier to use.

[1698] "Data analysis" is the process of using processed data to find meaningful patterns and trends.

[1699] "Result presentation" refers to presenting the results of data analysis in a user-friendly format.

[1700] "Feedback" refers to the reaction or information that a system provides to a user.

[1701] A "natural language processing engine" is a technology for analyzing text data and understanding its meaning and context.

[1702] An "emotion engine" is a technology for analyzing a user's emotional state from their voice and data.

[1703] The present invention is a system that combines a voice-operated data processing system with an emotion engine, which recognizes the user's emotional state and provides automatic adjustments and feedback to data processing based on that state. The system of the present invention is implemented using the following hardware and software.

[1704] Hardware and software used

[1705] Hardware

[1706] Smartphones (e.g. iPhone, Android devices)

[1707] microphone

[1708] server

[1709] software

[1710] Google Cloud Speech-to-Text API

[1711] IBM Watson Tone Analyzer

[1712] Google Natural Language API

[1713] Food delivery backend server

[1714] System configuration

[1715] 1. Acquiring voice instructions

[1716] The user speaks into the smartphone's microphone, saying something like "I want to order a pizza." This voice input is captured by the smartphone. The captured voice data is converted into text data using the Google Cloud Speech-to-Text API.

[1717] 2. Emotion analysis

[1718] The converted text data and the original voice data are then analyzed using IBM Watson Tone Analyzer to determine the user's emotional state, for example, whether the user is relaxed or anxious.

[1719] 3. Analysis of voice instructions

[1720] The parsed text data is then passed through the Google Natural Language API to identify the tasks the system should perform (e.g., display a menu, process an order), allowing it to optimally respond based on the user's emotional state and voice commands.

[1721] 4. Food delivery order fulfillment

[1722] The food delivery backend server retrieves menu information from a database based on the user's emotional state and voice instructions, and sends it to the smartphone. For example, if the user is tired, a simple menu is displayed, and if the user is relaxed, a detailed menu is displayed.

[1723] 5. Personalized feedback

[1724] Feedback that takes into account the user's emotional state is provided via smartphone, such as suggestions like "Today's recommendation is Margherita pizza."

[1725] Specific examples

[1726] A user speaks to their smartphone and says, "I'd like to order a pizza." The smartphone captures the voice data and converts it to text using the Google Cloud Speech-to-Text API. The system then uses IBM Watson Tone Analyzer to analyze the user's emotional state. For example, if the system detects that the user is tired, it might suggest, "How about a Margherita pizza?"

[1727] Prompt Sentence Examples

[1728] "If a user says, 'I want to order a pizza':

[1729] "User wants to order pizza. Emotional state is relaxed."

[1730] "Yes, this is the menu we recommend for users in a relaxed state."

[1731] The system allows users to have a personalized ordering experience that is tailored to their emotional state, resulting in increased user satisfaction and a more intuitive experience.

[1732] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1733] Step 1:

[1734] The user speaks into the smartphone's microphone. For example, they say, "I'd like to order a pizza." The input here is the user's voice, and the output is the voice data captured by the smartphone. The smartphone sends this voice data to the Google Cloud Speech-to-Text API, which converts it into text.

[1735] Step 2:

[1736] The smartphone uses the Google Cloud Speech-to-Text API to convert the acquired voice data into text data. The converted text is in the form of "I would like to order a pizza." The input of this process is voice data, and the output is text data.

[1737] Step 3:

[1738] The smartphone uses IBM Watson Tone Analyzer to analyze the user's emotional state from the text data and the original voice data. For example, it determines whether the user is relaxed or tired. The input is text data and voice data, and the output is data indicating the user's emotional state.

[1739] Step 4:

[1740] The smartphone sends text data and emotion data to the Google Natural Language API for analysis. The analyzed data is used to identify the tasks the system should perform (e.g., display a menu or process an order). The inputs for this step are text data and emotion data, and the output is the identified tasks.

[1741] Step 5:

[1742] The server accesses the food delivery backend system and retrieves menu information from the database based on the user's emotional state and voice instructions. For example, if the user is tired, a concise menu is displayed. The input is the specified task and the user's emotional state, and the output is the adjusted menu information.

[1743] Step 6:

[1744] The server sends the adjusted menu information to the smartphone, which then displays this information to the user, for example, "Today's special is Margherita pizza." The input is the adjusted menu information, and the output is the user's visual feedback.

[1745] Step 7:

[1746] The user makes a selection from the displayed menu and confirms the order by voice, for example, "I'd like to order a Margherita pizza." The smartphone converts this speech to text, again using the Google Cloud Speech-to-Text API. The input is the user's voice, and the output is text data.

[1747] Step 8:

[1748] The smartphone sends a final confirmation text to the food delivery backend system to confirm the order. The server receives the order and starts processing it. The input is the order confirmation text, and the output is the food delivery order data.

[1749] This process allows users to have a personalized ordering experience that is tailored to their emotional state, resulting in increased user satisfaction and a more intuitive experience.

[1750] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1751] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1752] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1753] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1754] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1755] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1756] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1757] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1758] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1759] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1760] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1761] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1762] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1763] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1764] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1765] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1766] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1767] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1768] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1769] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1770] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1771] The following is further disclosed regarding the above embodiment.

[1772] (Claim 1)

[1773] a means for inputting original data into the data processing system for controlling the system by voice operation;

[1774] means for converting voice instructions obtained from voice input into text data;

[1775] A means for analyzing text data and identifying multiple tasks for data extraction, data processing, data analysis, and result deployment;

[1776] A means for extracting information from the original data based on specified conditions;

[1777] means for processing the extracted information;

[1778] a means for analyzing the processed information;

[1779] a means for converting the analysis results into a specified format;

[1780] means for presenting the analysis results to a user;

[1781] A system including:

[1782] (Claim 2)

[1783] 10. The system of claim 1, which automatically performs all steps of data extraction, processing, analysis, and result development based on voice instructions.

[1784] (Claim 3)

[1785] 10. The system of claim 1, utilizing a natural language processing engine for identifying and performing a plurality of tasks.

[1786] "Example 1"

[1787] (Claim 1)

[1788] a means for inputting original data into the data processing system for controlling the system by voice operation;

[1789] means for converting voice instructions obtained from voice input into text data;

[1790] A means for analyzing text data and identifying multiple tasks for data extraction, data processing, data analysis, and result deployment;

[1791] A means for extracting information from the original data based on specified conditions;

[1792] means for processing the extracted information;

[1793] a means for analyzing the processed information;

[1794] a means for converting the analysis results into a specified format;

[1795] means for presenting the analysis results to a user;

[1796] means for generating and analyzing text through speech recognition software and natural language processing engines;

[1797] means for performing the steps of data extraction, processing, analysis and result development based on the generated text;

[1798] A system including:

[1799] (Claim 2)

[1800] 10. The system of claim 1, which automatically performs all steps of data extraction, processing, analysis, and result development based on voice instructions.

[1801] (Claim 3)

[1802] 10. The system of claim 1, utilizing a natural language processing engine for identifying and performing a plurality of tasks.

[1803] "Application Example 1"

[1804] (Claim 1)

[1805] a means for inputting original data into the data processing system for controlling the system by voice operation;

[1806] means for converting voice instructions obtained from voice input into text data;

[1807] A means for analyzing text data and identifying multiple tasks for data extraction, data processing, data analysis, and result deployment;

[1808] A means for extracting information from the original data based on specified conditions;

[1809] means for processing the extracted information;

[1810] a means for analyzing the processed information;

[1811] a means for converting the analysis results into a specified format;

[1812] means for presenting the analysis results to a user;

[1813] A logistics management method that manages inventory and shipping instructions based on voice commands and provides real-time feedback.

[1814] A system including:

[1815] (Claim 2)

[1816] 10. The system of claim 1, which automatically performs all steps of data extraction, processing, analysis, and result development based on voice instructions.

[1817] (Claim 3)

[1818] 10. The system of claim 1, utilizing a natural language processing engine for identifying and performing a plurality of tasks.

[1819] "Example 2: Combining Emotion Engines"

[1820] (Claim 1)

[1821] a means by which a user inputs raw data into the system;

[1822] means for converting voice instructions obtained from voice input into text data;

[1823] means for analyzing the converted text data and speech parameters to recognize the emotional state of the user;

[1824] A means for analyzing the text data and sentiment data and identifying a plurality of tasks for data extraction, data processing, data analysis, and result deployment;

[1825] A means for extracting information from the original data based on specified conditions;

[1826] means for processing the extracted information;

[1827] a means for analyzing the processed information;

[1828] a means for converting the analysis results into a specified format;

[1829] means for presenting the analysis results to a user;

[1830] means for adjusting the results of the data processing in response to the emotional state of the user;

[1831] A system including:

[1832] (Claim 2)

[1833] The system according to claim 1, which automatically performs all processes of data extraction, processing, analysis, and result development based on voice instructions.

[1834] (Claim 3)

[1835] 10. The system of claim 1, utilizing a natural language processing engine for identifying and performing a plurality of tasks.

[1836] (Claim 4)

[1837] 10. The system of claim 1, providing feedback according to the emotional state of the target user.

[1838] "Application example 2 when combining emotion engines"

[1839] (Claim 1)

[1840] a means for inputting original data into the data processing system for controlling the system by voice operation;

[1841] means for converting voice instructions obtained from voice input into text data;

[1842] means for analyzing the emotional state of a user from text data and voice data;

[1843] A means for analyzing text data and identifying multiple tasks for data extraction, data processing, data analysis, and result deployment;

[1844] A means for extracting information from the original data based on specified conditions;

[1845] means for processing the extracted information;

[1846] a means for analyzing the processed information;

[1847] means for converting information based on the analysis results and the user's emotional state into a specified format;

[1848] means for providing appropriate feedback to the user based on the analysis results and the user's emotional state;

[1849] A system including:

[1850] (Claim 2)

[1851] 10. The system of claim 1, wherein the system automatically performs all steps of extracting, processing, analyzing, and developing results based on voice instructions and emotional states.

[1852] (Claim 3)

[1853] 10. The system of claim 1, utilizing a natural language processing engine and an emotion engine for identifying and performing multiple tasks. [Explanation of symbols]

[1854] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for inputting original data into the data processing system for controlling the system by voice operation; means for converting voice instructions obtained from voice input into text data; A means for analyzing text data and identifying multiple tasks for data extraction, data processing, data analysis, and result deployment; A means for extracting information from the original data based on specified conditions; means for processing the extracted information; a means for analyzing the processed information; a means for converting the analysis results into a specified format; means for presenting the analysis results to a user; A system including:

2. 2. The system according to claim 1, wherein all steps of data extraction, processing, analysis, and result development are automatically performed based on voice instructions.

3. The system of claim 1 , wherein the system utilizes a natural language processing engine for identifying and performing a plurality of tasks.

Images