System

Abstract

A system is provided.SOLUTION: The system includes means for receiving an instruction input by a user in natural language, means for analyzing the received instruction by natural language processing, means for acquiring necessary data from a data warehouse based on an analysis result, means for preprocessing the acquired data, means for generating a graph based on the preprocessed data, and means for transmitting the generated graph to a terminal of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current data analysis methods require a high level of expertise and manual labor to retrieve the necessary data from a data warehouse and generate appropriate graphs. This effort and expertise is a barrier for many users and hinders rapid data analysis and decision-making.

[0005] Furthermore, the lack of a system that automatically processes data in response to instructions in natural language and provides visual output to users reduces the efficiency of data utilization. Therefore, it is a challenge to provide a system that can quickly and effectively visualize required data with simple verbal instructions. [Means for solving the problem]

[0006] To achieve this object, a system is provided, comprising:

[0007] A means of receiving instructions entered by the user in natural language

[0008] A means of analyzing received instructions using natural language processing

[0009] A means of obtaining the necessary data from the data warehouse based on the analysis results

[0010] A means of preprocessing the acquired data

[0011] A means of generating graphs based on preprocessed data

[0012] A means to send the generated graph to the user's device

[0013] This allows users to quickly obtain and visualize the necessary data with simple textual instructions, without the need for advanced expertise. The system uses natural language processing technology to analyze the data period, data type, and graph format, automatically generating graphs that correspond to user instructions. It also uses a graph drawing library to provide visual output, greatly streamlining the user's analysis work.

[0014] A "user" is a person or entity that utilizes the system to obtain and visualize data.

[0015] A "natural language" is a language used by humans on a daily basis, as opposed to a formal programming language.

[0016] An "instruction" is a command statement that indicates the operation or processing that the user wants the system to perform.

[0017] "Natural language processing" is a technology that allows computers to understand and analyze natural language.

[0018] "Analysis" is the process of carefully analyzing input data and information and understanding it as structured information.

[0019] A "data warehouse" is a specialized database system for efficiently managing, storing, and analyzing large amounts of data collected from various data sources within an organization.

[0020] "Data" is a collection of information expressed in the form of symbols, letters, numbers, etc.

[0021] "Preprocessing" is a series of operations performed to prepare data in a format suitable for analysis.

[0022] A "graph" is a diagram or chart that visually represents data.

[0023] "Means" are the methods or techniques necessary to achieve a particular purpose.

[0024] A "terminal" is a device such as a computer or smartphone that can be directly operated by a user. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0026] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0029] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0030] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0031] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0033] [First embodiment]

[0034] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0035] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0036] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0038] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0041] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0043] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0044] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0045] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] In this invention, the user simply issues a command in natural language to acquire and visualize data, and the system automatically processes the data and generates and displays a graph. Specific embodiments of the system are described below.

[0047] Overall overview

[0048] This system automates a series of processes: interpreting user instructions, retrieving the necessary data from a data warehouse, performing preprocessing, generating graphs, and finally displaying the results on the user's device.

[0049] User input

[0050] For example, a user inputs a command in natural language into their device, such as: "Show me this month's sales data in a bar graph." The user can freely specify the period, type, and display format of the data.

[0051] Natural Language Processing

[0052] The device sends the user's instructions to the server, which first analyzes them using a natural language processing engine (e.g., spaCy or NLTK). Through the analysis, information such as the data period (e.g., this month), data type (e.g., sales data), and graph format (e.g., bar graph) is extracted.

[0053] Data Acquisition

[0054] The server generates a query to the data warehouse (DWH) based on the analysis results. This query is to obtain the specific data the user is looking for (this month's sales data). As a result of the query, the required data is sent from the DWH to the server.

[0055] Data Preprocessing

[0056] The server pre-processes the data it receives, which includes data cleansing (removing duplicates and imputing missing values), filtering, and aggregation, making the data suitable for generating graphs in the specified format.

[0057] Graph Generation

[0058] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a bar graph) according to the user's instructions. To generate the graph, it uses a graph drawing library such as Matplotlib or Plotly.

[0059] Sending and displaying graphs

[0060] The generated graphs are exported in image format (such as PNG or JPEG) or interactive format (such as HTML or JavaScript (registered trademark)) and sent from the server to the terminal. The terminal then displays the received graph data so that the user can visually check it.

[0061] Specific examples

[0062] For example, if a user inputs "Display last month's sales data as a line graph," the system operates as follows: First, the user's device sends the instruction to the server, which analyzes it. As a result of the analysis, the information extracted is "last month," "sales data," and "line graph." Based on this information, the server sends a query to the DWH to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. The final graph is sent to the user's device and displayed.

[0063] In this way, this system automatically acquires and visualizes the necessary data based on the user's verbal instructions, making data analysis possible without requiring specialized knowledge.The user simply commands, "Display the data for XX in a XX graph," and can quickly obtain a visual display of the data.

[0064] The processing flow will be explained below.

[0065] Step 1:

[0066] The user inputs a command into their own device, such as "Display this month's sales data in a bar graph." The user inputs the specific data period, data type, graph format, etc. in natural language.

[0067] Step 2:

[0068] The terminal sends the instructions entered by the user to the server. The user's terminal sends the input data to the server as an HTTP request.

[0069] Step 3:

[0070] The server parses the received user instructions using a natural language processing engine (e.g., spaCy, NLTK), and extracts important information from the instructions, such as the time period of the data (e.g., this month), the type of data (e.g., sales data), and the graph format (e.g., bar graph).

[0071] Step 4:

[0072] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. Specifically, the server generates an SQL query and executes the query against the data warehouse.

[0073] Step 5:

[0074] The server receives the data retrieved from the data warehouse and performs pre-processing, which includes data cleansing (e.g., removing duplicate data and imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating the total sales by month).

[0075] Step 6:

[0076] The server generates a graph in the specified format (e.g., a bar graph) based on the preprocessed data, using a graph drawing library such as Matplotlib or Plotly.

[0077] Step 7:

[0078] The server converts the generated graphs into a suitable format for transmission to the user's device. Graphs can be exported into image formats (e.g. PNG, JPEG) or interactive formats (e.g. HTML, JavaScript).

[0079] Step 8:

[0080] The server sends the exported graph data to the user's terminal as an HTTP response.

[0081] Step 9:

[0082] The terminal displays the received graph data. The user's terminal renders the received data on the screen and displays it so that the user can visually confirm it.

[0083] This series of steps allows users to quickly retrieve and visualize data using simple natural language commands.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] In today's information society, users need to quickly and visually understand the information they need from large amounts of data. However, conventional data analysis tools require specialized knowledge, making data analysis and visualization difficult and preventing rapid response. Furthermore, the time and effort required for data preprocessing and graph generation makes it difficult to achieve efficient data analysis.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes means for receiving instructions input by a user in natural language, means for analyzing the received instructions using natural language processing, means for retrieving necessary information from a data storage device based on the analysis result, means for preprocessing the retrieved information, means for generating a visual display based on the preprocessed information, and means for transmitting the generated visual display to the user's device, thereby enabling the user to quickly analyze and visualize data through simple instructions in natural language without requiring specialized knowledge.

[0089] A "user" is a person or entity that operates the system and inputs instructions in natural language.

[0090] "Natural language" refers to language used by humans on a daily basis, and refers to common words and phrases that do not require specific technical skills or knowledge.

[0091] "Instructions" refer to requests or commands given by a user to a system, specifically input in natural language for the purpose of obtaining or visualizing data.

[0092] "Natural language processing" is a technology that allows computers to understand and analyze human language (natural language), and includes the process of extracting meaning from text data.

[0093] A "data storage device" is a computer system for storing and managing large amounts of data, and includes technologies such as data warehouses and databases.

[0094] "Information" refers to data retrieved from a data storage device based on a user's instruction, and includes data of a specific period and type.

[0095] "Preprocessing" refers to a series of operations that prepare acquired information in a form suitable for generating a visual display, including data cleansing and filtering.

[0096] "Visual displays" refer to data display formats such as graphs and charts that are created to make acquired information easier to understand visually.

[0097] "Visual representation rendering tools" refers to software libraries and platforms used to efficiently generate visual representations.

[0098] In this invention, a user can use natural language commands to acquire and visualize data, and the system automatically processes the data based on the commands and generates and displays graphs. Specific embodiments of the system are described below.

[0099] Overall overview

[0100] This system automates a series of processes: interpreting a user's natural language instructions, retrieving the necessary information from a data storage device, performing preprocessing, generating a graph as a visual display, and finally displaying it on the user's device.

[0101] Natural language input

[0102] The user inputs a command in natural language into their own terminal, such as "Display this month's sales data in a bar graph." The user can freely specify the period, type, display format, etc. of the data.

[0103] Natural Language Processing

[0104] The device sends the instructions entered by the user to the server, which then analyzes them using a natural language processing engine (e.g., spaCy or NLTK). The analysis extracts information such as the data period (e.g., this month), the type of data (e.g., sales data), and the graph format (e.g., bar graph).

[0105] Data Acquisition

[0106] The server generates a query to a data store (such as a data warehouse) based on the analysis results. The query is to obtain the specific information the user is looking for (e.g., this month's sales data). As a result of the query, the data store sends the required information to the server.

[0107] Data Preprocessing

[0108] The server pre-processes the information it receives, which includes data cleansing (removing duplicates and imputing missing values), filtering, and aggregation, making the information suitable for generating graphs in the specified format.

[0109] Graph Generation

[0110] Based on the preprocessed information, the server generates a visual representation in the specified format (e.g., a bar graph) according to the user's instructions, using a graph drawing library such as Matplotlib or Plotly.

[0111] Sending and displaying graphs

[0112] The generated graphs are exported in image format (e.g. PNG or JPEG files) or interactive format (e.g. HTML or JavaScript) and sent from the server to the user's device, where the received graph data is displayed so that the user can visually check it.

[0113] Specific examples

[0114] For example, consider the case where a user inputs "Display last month's sales data as a line graph." In this case, the user's device sends the instruction to the server, which analyzes the instruction using a natural language processing engine. As a result of the analysis, the information "last month," "sales data," and "line graph" is extracted, and the server sends a query to the data storage device based on this information to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. The final graph is sent to the user's device and displayed.

[0115] In this way, this system automatically obtains and visualizes the necessary information based on the user's natural language instructions, making data analysis possible without requiring specialized knowledge.The user simply commands, "Display the data for XX in a XX graph," and can quickly obtain a visual display of the data.

[0116] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0117] Step 1:

[0118] A user inputs a command in natural language into his / her terminal. For example, the user inputs "Show me this month's sales data in a bar graph." The input data is a natural language command in the form of a string.

[0119] Input: "Show me this month's sales data in a bar chart"

[0120] Output: Natural language instructions sent from the device to the server

[0121] Step 2:

[0122] The terminal sends the received natural language instruction to the server, where it sends the instruction as is without converting its format.

[0123] Input: User's natural language instructions

[0124] Output: Natural language instructions sent to the server

[0125] Step 3:

[0126] The server analyzes the received instructions using a natural language processing engine (e.g., spaCy or NLTK), extracting information such as the data period, type, and graph format.

[0127] Input: Natural language instructions

[0128] Output: Data period (e.g., this month), data type (e.g., sales data), graph format (e.g., bar graph)

[0129] Step 4:

[0130] Based on the analysis results, the server generates a query to a data storage device (such as a DWH). This query is written in a format such as SQL and is intended to obtain the required information.

[0131] Input: Analysis results (data period, type, graph format)

[0132] Output: Generated query (e.g. "SELECT FROM sales WHERE date >= '2023-10-01' AND date <= '2023-10-31'")

[0133] Step 5:

[0134] The server sends the generated query to the data storage device to retrieve the specified data (sales data for this month). The query results are returned from the data storage device.

[0135] Input: Generated query

[0136] Output: Retrieved data (e.g., this month's sales data)

[0137] Step 6:

[0138] The server pre-processes the acquired data, which includes data cleansing (removing duplicate data and filling in missing values), filtering, and aggregation.

[0139] Input: Retrieved data

[0140] Output: Preprocessed data

[0141] Step 7:

[0142] The server generates a visual display in the specified format (e.g., a bar graph) based on the preprocessed data, using a visual display drawing tool such as Matplotlib or Plotly.

[0143] Input: Preprocessed data

[0144] Output: Generated graph (e.g. bar graph in PNG format)

[0145] Step 8:

[0146] The server sends the generated graph to the user's terminal in the form of an image file (e.g. PNG file) or an interactive HTML file.

[0147] Input: Generated graph

[0148] Output: Graph data sent to the terminal

[0149] Step 9:

[0150] The terminal displays the received graph data, including image files and interactive HTML files, in an appropriate viewer or browser.

[0151] Input: Graph data sent from the server

[0152] Output: A graphical representation on the screen that can be viewed by the user

[0153] For example, if a user types "Show me last month's sales data as a line graph," the system will do the following:

[0154] Step 1-2: The user inputs a natural language instruction, which is sent from the terminal to the server.

[0155] Step 3: The server parses the instructions and extracts the sales data for the last month in the form of a line graph.

[0156] Step 4-5: The server queries the data store to retrieve last month's sales data.

[0157] Step 6: Preprocess the data retrieved by the server.

[0158] Step 7: Generate a line graph based on the preprocessed data.

[0159] Step 8-9: The generated graph is sent to the user's terminal and displayed.

[0160] In this way, the system can automatically acquire and visualize the necessary data based on the user's instructions.

[0161] (Application example 1)

[0162] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0163] Modern production management systems generate massive amounts of data, but visually understanding this data requires specialized knowledge, making it difficult to identify and address problems in real time. Furthermore, there is a lack of tools that allow on-site managers to intuitively understand the data and respond quickly. This leads to problems such as reduced production efficiency and delayed early detection of abnormalities. The present invention aims to solve these problems by providing a system that allows users to easily instruct data visualization in natural language and respond immediately.

[0164] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0165] In this invention, the server includes means for receiving instructions entered by a user in natural language, means for analyzing the received instructions using natural language processing, means for retrieving necessary data from a data warehouse based on the analysis results, means for preprocessing the retrieved data, means for generating a graph based on the preprocessed data, means for transmitting the generated graph to the user's terminal, means for analyzing the natural language instructions and extracting a specific data type and time period, means for retrieving production management system data based on the extracted specific information, means for generating a graph in a specified format (e.g., pie chart, bar graph) based on the production management system data, and means for displaying the generated graph on a smartphone display. This simplifies the visualization of data in the production management system, enabling on-site managers to intuitively understand the data and respond quickly.

[0166] "Natural language" refers to a language that humans use on a daily basis, including written and spoken languages.

[0167] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language, and includes technology for text and voice data.

[0168] A "data warehouse" is a database system that allows companies and organizations to centrally store large amounts of data and integrate and analyze data from various data sources.

[0169] "Preprocessing" is the process of converting raw data into a form suitable for analysis and visualization, and includes data cleansing, filtering, and aggregation.

[0170] A "graph" is a diagram that visually represents data and comes in various forms, such as bar graphs, line graphs, and pie charts.

[0171] A "smartphone" is a highly functional mobile phone that can connect to the Internet and use a variety of applications.

[0172] A "production management system" is a system for managing production activities in factories and production lines, and includes production planning, progress management, quality control, etc.

[0173] A "terminal" is a device that allows a user to input information and display results, and includes smartphones, tablets, and personal computers.

[0174] A "graph drawing library" is a software library for generating graphs in programs, and includes Matplotlib and Plotly.

[0175] A "server" is a computer system that processes and provides data, and provides services in response to requests from clients.

[0176] The present invention provides a system that allows a user to use natural language to instruct data visualization in a production management system, and a server automatically processes the data and generates graphs, displaying the results on the user's terminal.

[0177] Overall overview

[0178] This system automates a series of processes: interpreting user instructions, acquiring and preprocessing specified production management data, generating graphs, and finally displaying them on a smartphone screen.

[0179] User input

[0180] Users use their smartphones to input instructions in natural language, such as "Display today's operating status of production line 1 in a pie chart." Such instructions allow users to specify a specific data period, data type, and graph format.

[0181] Natural Language Processing

[0182] The user's device sends the received instructions to the server, which then analyzes them using a natural language processing engine (e.g., spaCy or NLTK). This analysis extracts information such as the data period (e.g., today), the data type (e.g., the operating status of production line 1), and the graph format (e.g., pie chart).

[0183] Data Acquisition

[0184] The server generates a query to the production management system (data warehouse) based on the analysis results and retrieves the necessary data. This query is intended to retrieve data including operation information for a specific production line.

[0185] Data Preprocessing

[0186] The server pre-processes the data it receives, which includes cleansing, filtering, and aggregating the data, converting it into an optimal format for generating graphs.

[0187] Graph Generation

[0188] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a pie chart) using a graph drawing library such as Matplotlib or Plotly.

[0189] Sending and displaying graphs

[0190] The generated graph is exported as an image file (e.g., PNG) and sent from the server to the user's smartphone, where it displays the received graph data for the user to visually check.

[0191] For example, if a user inputs "Display this month's downtime data for production line 2 as a bar graph," the system operates as follows: The user's device sends the instruction to the server, which analyzes it. As a result of the analysis, the following information is extracted: "This month," "Production line 2," "Downtime data," and "Bar graph." Based on this information, the server sends a query to the data warehouse to obtain "This month's downtime data for production line 2." Next, the server preprocesses the data and then generates a bar graph. The final graph is sent to the user's smartphone and displayed.

[0192] This system automatically acquires and visualizes the necessary data based on the user's natural language instructions, making it possible to analyze production data without requiring specialized knowledge. Users can quickly obtain a visual display of data by simply instructing, "Display the data for XX in a XX graph."

[0193] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0194] Step 1:

[0195] A user uses a smartphone to input instructions in natural language, such as "Display today's operating status of production line 1 in a pie chart." The input instruction is saved in an input field on the smartphone and is ready to be sent to the server.

[0196] Step 2:

[0197] The device sends the user's instructions to the server. The smartphone sends the input text data to the server using an HTTP request or WebSocket. The server receives the received natural language instructions and prepares them for analysis.

[0198] Step 3:

[0199] The server uses a natural language processing engine (e.g., spaCy or NLTK) to analyze the received instructions. Specifically, it tokenizes the text data, identifies nouns and verbs, and extracts the time period (e.g., today), data type (e.g., the operation status of production line 1), and graph format (e.g., pie chart) based on the context. The extracted information is used in the next step.

[0200] Step 4:

[0201] The server generates a specific query for the production management system (data warehouse) based on the analysis results. The generated query is sent as an SQL query or API request to obtain today's data on the operating status of production line 1. The production management system returns data that matches the specified conditions.

[0202] Step 5:

[0203] The server receives the acquired data and performs preprocessing, which includes data cleansing (e.g., filling in missing values ​​and removing duplicates), filtering (e.g., excluding unnecessary data), and aggregation (e.g., aggregating operating status by time). The preprocessed data is then converted into a format suitable for graph generation.

[0204] Step 6:

[0205] The server generates a graph in the specified format (e.g., pie chart) based on the preprocessed data. Graph drawing libraries such as Matplotlib and Plotly are used to generate the graph. Specifically, data points are mapped to each part of the graph and visual elements (color, label, etc.) are added. The generated graph is exported as an image format (e.g., PNG).

[0206] Step 7:

[0207] The server sends the generated graph to the user's smartphone. The image file is sent as an HTTP response or WebSocket message. The smartphone displays the received image file in the specified display area.

[0208] Step 8:

[0209] Users can visually check the generated graphs on their smartphone screen, which allows them to intuitively understand the operating status of the specified production line and take appropriate action.

[0210] This series of steps enables users to easily use natural language to instruct data visualization and grasp the status of production management in real time.

[0211] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0212] This invention combines a system that automatically acquires and visualizes data based on instructions entered by a user in natural language with an emotion engine that recognizes the user's emotions. An embodiment of this invention will be described in detail below.

[0213] Overall overview

[0214] The system allows users to input instructions in natural language, retrieves the necessary data based on those instructions, and generates and displays graphs. It can also recognize the emotions contained in the user's instructions and optimize the color and format of the displayed graphs based on those emotions.

[0215] User input

[0216] For example, a user inputs a command in natural language into their device, such as: "Show me this month's sales data in a bar graph." The user can freely specify the period, type, and display format of the data.

[0217] Natural Language Processing and Emotion Recognition

[0218] The device sends the user's instructions to the server, which first analyzes them using a natural language processing engine (e.g., spaCy or NLTK) and an emotion engine (e.g., Sentiment Analysis API). The natural language processing extracts information such as the data period (e.g., this month), data type (e.g., sales data), and graph format (e.g., bar graph), and the emotion engine recognizes the emotion (e.g., joy, sadness, anger) contained in the user's instructions.

[0219] Data Acquisition and Emotional Response

[0220] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. The server also takes into account the results of the emotion engine and optimizes the data display format and graph colors according to the user's emotions. For example, if the user is expressing positive emotions, a bright color graph will be selected.

[0221] Data Preprocessing

[0222] The server pre-processes the retrieved data. Pre-processing includes data cleansing (e.g., removing duplicates, imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating monthly sales totals). At this stage, user sentiment also influences data processing. For example, dissatisfied users may be provided with more detailed data to help find the root cause of the problem.

[0223] Graph Generation

[0224] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a bar graph). Graph drawing libraries such as Matplotlib and Plotly are used to generate the graph. Again, the color and format of the graph are adjusted based on the results of the emotion engine.

[0225] Sending and displaying graphs

[0226] The generated graphs are exported in image formats (such as PNG or JPEG) or interactive formats (such as HTML or JavaScript) and sent from the server to the terminal, where the received graph data is displayed so that the user can visually check it.

[0227] Specific examples

[0228] For example, if a user inputs, "Display last month's sales data as a line graph, and do not use bright colors because sales were low," the system operates as follows: First, the user's device sends the instruction to the server, which analyzes it. As a result of the analysis, information such as "last month," "sales data," "line graph," and "negative sentiment" is extracted. Based on this information, the server sends a query to the DWH to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. Based on the results of the sentiment engine, a graph with darker colors is selected. The final graph is sent to the user's device and displayed.

[0229] In this way, the system can quickly and effectively acquire and visualize data based on the user's verbal instructions and emotions. Users do not need advanced expertise; they can simply input an instruction, including emotional input, such as "Display the data for XX in a XX graph," to obtain an optimized visual data display.

[0230] The processing flow will be explained below.

[0231] Step 1:

[0232] The user inputs a command into their device, such as "Display this month's sales data as a bar graph." The user can also input specific commands, including emotions (e.g., "I'm disappointed with the low sales figures, so I want to use dark colors").

[0233] Step 2:

[0234] The terminal sends the instructions entered by the user to the server. The user's terminal sends the input data to the server as an HTTP request.

[0235] Step 3:

[0236] The server parses the received user instructions using a natural language processing engine (e.g., spaCy, NLTK), and extracts important information from the instructions, such as the time period of the data (e.g., this month), the type of data (e.g., sales data), and the graph format (e.g., bar graph).

[0237] Step 4:

[0238] The server uses an emotion engine (e.g., Sentiment Analysis API) to recognize emotions from the analysis results of the natural language processing engine. For example, the emotion engine extracts negative emotions such as "disappointed" or "depressed."

[0239] Step 5:

[0240] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. Specifically, the server generates an SQL query and executes the query against the data warehouse.

[0241] Step 6:

[0242] The server receives the data retrieved from the data warehouse and performs pre-processing, which includes data cleansing (e.g., removing duplicate data and imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating the total sales by month).

[0243] Step 7:

[0244] The server generates graphs in the specified format (e.g., bar graphs) based on the preprocessed data and in accordance with the user's instructions. Graph drawing libraries such as Matplotlib and Plotly are used to generate graphs. The results of the emotion engine are reflected here, and if the user's instructions indicate negative emotions, the graph's color scheme is set to a darker shade.

[0245] Step 8:

[0246] The server converts the generated graphs into a suitable format for transmission to the user's device. Graphs can be exported into image formats (e.g. PNG, JPEG) or interactive formats (e.g. HTML, JavaScript).

[0247] Step 9:

[0248] The server sends the exported graph data to the user's terminal as an HTTP response.

[0249] Step 10:

[0250] The terminal displays the received graph data. The user's terminal renders the received data on the screen and displays it so that the user can visually confirm it.

[0251] Through this series of steps, users simply communicate simple natural language instructions and emotions to the system, which then automatically retrieves the data and visualizes it in an appropriate format.

[0252] Example 2

[0253] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0254] Conventional data visualization systems display data without considering the user's emotions, which means they are unable to provide optimal visualizations that reflect the user's intentions and emotions. While systems exist that can respond to natural language instructions, there is a lack of technology that can recognize emotions based on those instructions and reflect them in data visualizations. Therefore, there is a need for a system that allows users to input instructions in natural language, recognizes emotions based on those instructions, and provides appropriate data visualization.

[0255] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0256] In this invention, the server includes means for receiving instructions input by a user in natural language, means for analyzing the received instructions using natural language processing, means for recognizing the user's emotions from the analyzed instructions, means for acquiring necessary data from an information set based on the analysis results, means for preprocessing the acquired data, means for generating a graph by optimizing the format and color of the graph based on the preprocessed data in accordance with the user's emotions, and means for transmitting the generated graph to the user's terminal, thereby enabling optimal data visualization in accordance with the user's emotions.

[0257] "User" refers to a person who inputs instructions to the system in natural language.

[0258] "Natural language" refers to the words and written forms that humans use on a daily basis.

[0259] "Instructions" refer to requests made by the user to the system regarding data acquisition and display format.

[0260] "Server" refers to a central processing unit that receives user instructions, processes them, and returns the results to the user.

[0261] "Terminal" refers to an electronic device that allows a user to input instructions.

[0262] "Natural language processing" refers to the technology that allows computers to understand and analyze human language.

[0263] "Emotion recognition" refers to technology that analyzes and identifies the emotions contained in a user's instructions.

[0264] An "information collection" refers to a place that stores large amounts of data, such as a data warehouse or database.

[0265] "Preprocessing" refers to the process of organizing and shaping acquired data and converting it into a format suitable for analysis and visualization.

[0266] A "graph" refers to a diagram or chart that visually represents data.

[0267] "Graph optimization" refers to adjusting the format, color, and display method of a graph based on user sentiment.

[0268] "Generation" refers to the act of creating a graph in a specified format based on preprocessed data.

[0269] "Transmission" refers to the act of sending the graph generated by the server to the terminal and displaying it to the user.

[0270] "Visualization" refers to the presentation of information or data to a user in a graphical format.

[0271] This invention combines a system that automatically acquires and visualizes data based on instructions entered by a user in natural language with an emotion engine that recognizes the user's emotions. An embodiment of this invention will be described in detail below.

[0272] Overall overview

[0273] The system allows users to input instructions in natural language, retrieves the necessary data based on those instructions, and generates and displays graphs. It can also recognize the emotions contained in the user's instructions and optimize the color and format of the displayed graphs based on those emotions.

[0274] Hardware and software used

[0275] Server: A central processing unit that receives and processes user instructions.

[0276] Terminal: An electronic device (e.g., computer, smartphone) that allows a user to input instructions and display results.

[0277] Natural language processing engine: Software for parsing a user's natural language instructions, such as spaCy or NLTK.

[0278] Emotion recognition engine: Software for analyzing user emotions, such as the Sentiment Analysis API.

[0279] Data Warehouse (DWH): A database system for storing data and retrieving required data using queries.

[0280] Graph drawing libraries: Software for visualizing data in the form of graphs, such as Matplotlib or Plotly.

[0281] User input

[0282] For example, a user inputs a command in natural language into their device, such as: "Show me this month's sales data in a bar graph." The user can freely specify the period, type, and display format of the data.

[0283] Natural Language Processing and Emotion Recognition

[0284] The device sends the user's instructions to the server, which first analyzes them using a natural language processing engine (e.g., spaCy or NLTK) and an emotion engine (e.g., Sentiment Analysis API). The natural language processing extracts information such as the data period (e.g., this month), data type (e.g., sales data), and graph format (e.g., bar graph), and the emotion engine recognizes the emotion (e.g., joy, sadness, anger) contained in the user's instructions.

[0285] Data Acquisition and Emotional Response

[0286] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. The server also takes into account the results of the emotion engine and optimizes the data display format and graph colors according to the user's emotions. For example, if the user is expressing positive emotions, a bright color graph will be selected.

[0287] Data Preprocessing

[0288] The server pre-processes the retrieved data. Pre-processing includes data cleansing (e.g., removing duplicates, imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating monthly sales totals). At this stage, user sentiment also influences data processing. For example, dissatisfied users may be provided with more detailed data to help find the root cause of the problem.

[0289] Graph Generation

[0290] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a bar graph) using a graph drawing library such as Matplotlib or Plotly. Again, the color and format of the graph are adjusted based on the results of the emotion engine.

[0291] Sending and displaying graphs

[0292] The generated graphs are exported in image formats (such as PNG or JPEG) or interactive formats (such as HTML or JavaScript) and sent from the server to the terminal, where the received graph data is displayed so that the user can visually check it.

[0293] Specific examples

[0294] For example, if a user inputs, "Display last month's sales data as a line graph, and do not use bright colors because sales were low," the system operates as follows: First, the user's device sends the instruction to the server, which analyzes it. As a result of the analysis, information such as "last month," "sales data," "line graph," and "negative sentiment" is extracted. Based on this information, the server sends a query to the DWH to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. Based on the results of the sentiment engine, a graph with darker colors is selected. The final graph is sent to the user's device and displayed.

[0295] In this way, the system can quickly and effectively acquire and visualize data based on the user's verbal instructions and emotions. Users do not need advanced expertise; they can simply input an instruction, including emotional input, such as "Display the data for XX in a XX graph," to obtain an optimized visual data display.

[0296] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0297] Step 1: User input

[0298] The user inputs a command in natural language using a terminal. For example, the command might say, "Display this month's sales data in a bar graph." This command includes the data period ("this month"), type ("sales data"), and display format ("bar graph").

[0299] Input: User instructions in natural language

[0300] Output: Typed instructions retained on the terminal

[0301] Step 2: Send instructions to the server

[0302] The terminal transmits the input instructions to the server. This data transmission is carried out via the Internet.

[0303] Input: User's natural language instructions

[0304] Output: Instruction data sent to the server

[0305] Step 3: Parsing instructions (natural language processing and emotion recognition)

[0306] The server analyzes the received instructions using a natural language processing engine (e.g., spaCy). This analysis extracts information such as the data period, type, and display format. At the same time, it analyzes the user's emotions using an emotion recognition engine (e.g., Sentiment Analysis API).

[0307] Input: Instruction data sent to the server

[0308] Output: Analysis result data period, type, display format, emotional information

[0309] Step 4: Obtaining the necessary data

[0310] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. The server executes SQL queries according to the period and type and retrieves the relevant data from the database.

[0311] Input: Analysis results

[0312] Output: Required data obtained from DWH

[0313] Step 5: Preprocessing the data

[0314] The server preprocesses the acquired data, including removing duplicate data, imputing missing values, filtering periods, and aggregating data. It also adjusts the level of detail of the data according to the user's sentiment.

[0315] Input: Required data

[0316] Output: Preprocessed data

[0317] Step 6: Generate the graph

[0318] The server generates graphs in the specified format (e.g., bar graphs) based on the preprocessed data. It uses Matplotlib and Plotly to generate graphs, optimizing the color and format of the graphs based on the user's preferences.

[0319] Input: Preprocessed data, emotion information

[0320] Output: The generated graph

[0321] Step 7: Send and display the graph

[0322] The server sends the generated graph to the terminal, where it can be exported in image format (PNG or JPEG) or interactive format (HTML or JavaScript) and displayed on the terminal for the user to view.

[0323] Input: Generated graph

[0324] Output: Graph displayed on the user's terminal

[0325] Specific examples

[0326] For example, if a user inputs "Display this month's sales data in a bar graph," the system operates as follows: First, the user's device sends the instruction to the server. The server analyzes the instruction and extracts information such as "this month," "sales data," and "bar graph." It then retrieves "this month's sales data" from the DWH based on the analysis results. The server preprocesses the data and generates a bar graph based on the preprocessed data. The final graph is sent to the user's device and displayed. This allows users to easily obtain visualized data based on instructions in natural language.

[0327] (Application example 2)

[0328] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0329] Conventional data visualization systems analyze users' natural language instructions to visualize data, but they do not optimize the system to take into account the user's emotions, resulting in a lack of improvement in the user experience. Furthermore, particularly in the entertainment industry, users often seek information and content that reflects their current emotions, creating a need for appropriate data and content that takes emotions into account.

[0330] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving instructions entered by a user in natural language, means for analyzing the received instructions using natural language processing, means for retrieving necessary data from a data warehouse based on the analysis result, means for preprocessing the retrieved data, means for generating a graph based on the preprocessed data, means for transmitting the generated graph to the user's terminal, means for recognizing the user's emotion based on the analyzed instructions, and means for optimizing the format and color of displayed data based on the recognized emotion. This enables optimal data visualization and content display according to the user's emotion.

[0331] A "user" is a person who uses the system to input instructions in natural language.

[0332] A "natural language" is a language that humans use on a daily basis, and is not a specific programming language.

[0333] An "instruction" represents a request or command given by a user to a system.

[0334] "Analysis" is the process of mechanically breaking down instructions entered in natural language and understanding their meaning.

[0335] A "data warehouse" is a system for centrally storing and managing large amounts of data.

[0336] "Preprocessing" is the process of processing acquired data through cleansing, filtering, and other processes to prepare it in a format suitable for analysis and visualization.

[0337] A "graph" is a diagram that visually represents numerical data.

[0338] A "terminal" is a device through which a user accesses and operates the system.

[0339] "Emotion" refers to the psychological state of the user.

[0340] "Optimization" means adjusting something to the most effective state for a specific purpose.

[0341] This invention combines a system that allows a user to input instructions in natural language and automatically acquires and visualizes data based on the instructions with an emotion engine that recognizes the user's emotions. The system includes the following steps:

[0342] First, the user inputs a command in natural language into their device. For example, they might say, "Tell me the next interesting movie" or "Tell me a movie that will help me relax when I'm tired." This is the initial input stage performed by the user.

[0343] The device sends these instructions to the server, which then uses a natural language processing engine (e.g., spaCy) to analyze the user's instructions. At this stage, the server extracts the content of the instructions (e.g., the genre and characteristics of the movie being searched for) and simultaneously analyzes the user's emotions using an emotion engine (e.g., Sentiment Analysis API).

[0344] Based on the analysis results, the server retrieves the necessary data from the data warehouse, including movie information and rating data from the content database. For example, if a user requests "relaxing movies," the server will extract movies that meet that criteria.

[0345] The acquired data is pre-processed by the server, which includes data cleansing, filtering, and optionally aggregating the data. The server then optimizes the data based on the results of the emotion engine. If the user's emotion is relaxation, a list of movies that match that emotion is generated. The visual display format and colors are also adjusted based on the emotion.

[0346] Finally, the preprocessed and optimized data may be graphed using a graph drawing library (e.g., Matplotlib or Plotly). This visual graph or content list is sent to the user's device and displayed to the user.

[0347] Specific examples

[0348] For example, if a user types, "I've been feeling stressed lately, can you recommend some relaxing movies?", the system analyzes the instruction, extracts "relaxing movies" as a movie genre, and uses an emotion engine to recognize the user's stress level. It then retrieves information about relaxing movies from a data warehouse and generates an optimized list based on the user's emotions. Finally, the movie list, with its background color and display format adjusted, is displayed on the user's device.

[0349] Prompt Sentence Examples

[0350] "I've been feeling stressed lately, so can you recommend a relaxing movie?"

[0351] This invention allows users to easily obtain data visualization and content display that best fits their emotional state. The system takes into account the user's emotions and provides the most appropriate information at that time, greatly improving the user experience.

[0352] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0353] Step 1:

[0354] The user inputs instructions in natural language, such as "Tell me the next good movie." This is the initial input that occurs on the terminal. The input instructions are sent directly to the server.

[0355] Step 2:

[0356] The server receives instructions in natural language sent from the device. The server analyzes the instructions using a natural language processing engine. As a result of the analysis, it extracts information such as the search target and display format. For example, if the instruction is "Tell me the next good movie," the search target will be "good movies."

[0357] Step 3:

[0358] The server uses an emotion engine to recognize the user's emotions based on the analysis results. Emotions (e.g., positive or negative emotions) are extracted from the content of the instruction. For example, in the case of "Tell me a movie that helps me relax when I'm tired," "fatigue" is recognized as the emotion.

[0359] Step 4:

[0360] The server retrieves the necessary data (e.g., movie information) from the data warehouse based on the analysis results and the recognized emotions. The server then sends a query to the database to retrieve the corresponding movie list.

[0361] Step 5:

[0362] The server preprocesses the retrieved data. Preprocessing includes data cleansing, filtering, and, if necessary, data aggregation. Data cleansing involves filling in missing values ​​and removing duplicates, while filtering extracts only data that corresponds to the user's instructions. For example, if the search term is "relaxing movies," only movie information that falls into this category is filtered.

[0363] Step 6:

[0364] The server then uses the pre-processed data to optimize the data based on the results of the emotion engine, for example, visualizing positive emotions with bright colors and negative emotions with muted colors, as well as adjusting the order of the list display and the emphasis of information.

[0365] Step 7:

[0366] The server uses a graph drawing library (such as Matplotlib or Plotly) to generate graphs based on the optimized data. When the user requests a visualization, a graph in the appropriate format is drawn.

[0367] Step 8:

[0368] The server sends the generated graph or optimized content list to the user's device. The device visually displays the received data. The display format and color are optimized according to the user's emotions.

[0369] Step 9:

[0370] The user checks the results displayed on the device, such as a list of relaxing movies and graphs, which are expected to have a psychologically soothing effect.

[0371] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0372] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0373] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0374] [Second embodiment]

[0375] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0376] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0377] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0378] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0379] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0380] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0381] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0382] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0383] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0384] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0385] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0386] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0387] In this invention, the user simply issues a command in natural language to acquire and visualize data, and the system automatically processes the data and generates and displays a graph. Specific embodiments of the system are described below.

[0388] Overall overview

[0389] This system automates a series of processes: interpreting user instructions, retrieving the necessary data from a data warehouse, performing preprocessing, generating graphs, and finally displaying the results on the user's device.

[0390] User input

[0391] For example, a user inputs a command in natural language into their device, such as: "Show me this month's sales data in a bar graph." The user can freely specify the period, type, and display format of the data.

[0392] Natural Language Processing

[0393] The device sends the user's instructions to the server, which first analyzes them using a natural language processing engine (e.g., spaCy or NLTK). Through the analysis, information such as the data period (e.g., this month), data type (e.g., sales data), and graph format (e.g., bar graph) is extracted.

[0394] Data Acquisition

[0395] The server generates a query to the data warehouse (DWH) based on the analysis results. This query is to obtain the specific data the user is looking for (this month's sales data). As a result of the query, the required data is sent from the DWH to the server.

[0396] Data Preprocessing

[0397] The server pre-processes the data it receives, which includes data cleansing (removing duplicates and imputing missing values), filtering, and aggregation, making the data suitable for generating graphs in the specified format.

[0398] Graph Generation

[0399] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a bar graph) according to the user's instructions. To generate the graph, it uses a graph drawing library such as Matplotlib or Plotly.

[0400] Sending and displaying graphs

[0401] The generated graphs are exported in image formats (e.g. PNG or JPEG) or interactive formats (e.g. HTML or JavaScript) and sent from the server to the terminal, which then displays the received graph data so that the user can visually check it.

[0402] Specific examples

[0403] For example, if a user inputs "Display last month's sales data as a line graph," the system operates as follows: First, the user's device sends the instruction to the server, which analyzes it. As a result of the analysis, the information extracted is "last month," "sales data," and "line graph." Based on this information, the server sends a query to the DWH to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. The final graph is sent to the user's device and displayed.

[0404] In this way, this system automatically acquires and visualizes the necessary data based on the user's verbal instructions, making data analysis possible without requiring specialized knowledge.The user simply commands, "Display the data for XX in a XX graph," and can quickly obtain a visual display of the data.

[0405] The processing flow will be explained below.

[0406] Step 1:

[0407] The user inputs a command into their own device, such as "Display this month's sales data in a bar graph." The user inputs the specific data period, data type, graph format, etc. in natural language.

[0408] Step 2:

[0409] The terminal sends the instructions entered by the user to the server. The user's terminal sends the input data to the server as an HTTP request.

[0410] Step 3:

[0411] The server parses the received user instructions using a natural language processing engine (e.g., spaCy, NLTK), and extracts important information from the instructions, such as the time period of the data (e.g., this month), the type of data (e.g., sales data), and the graph format (e.g., bar graph).

[0412] Step 4:

[0413] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. Specifically, the server generates an SQL query and executes the query against the data warehouse.

[0414] Step 5:

[0415] The server receives the data retrieved from the data warehouse and performs pre-processing, which includes data cleansing (e.g., removing duplicate data and imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating the total sales by month).

[0416] Step 6:

[0417] The server generates a graph in the specified format (e.g., a bar graph) based on the preprocessed data, using a graph drawing library such as Matplotlib or Plotly.

[0418] Step 7:

[0419] The server converts the generated graphs into a suitable format for transmission to the user's device. Graphs can be exported into image formats (e.g. PNG, JPEG) or interactive formats (e.g. HTML, JavaScript).

[0420] Step 8:

[0421] The server sends the exported graph data to the user's terminal as an HTTP response.

[0422] Step 9:

[0423] The terminal displays the received graph data. The user's terminal renders the received data on the screen and displays it so that the user can visually confirm it.

[0424] This series of steps allows users to quickly retrieve and visualize data using simple natural language commands.

[0425] Example 1

[0426] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0427] In today's information society, users need to quickly and visually understand the information they need from large amounts of data. However, conventional data analysis tools require specialized knowledge, making data analysis and visualization difficult and preventing rapid response. Furthermore, the time and effort required for data preprocessing and graph generation makes it difficult to achieve efficient data analysis.

[0428] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0429] In this invention, the server includes means for receiving instructions input by a user in natural language, means for analyzing the received instructions using natural language processing, means for retrieving necessary information from a data storage device based on the analysis result, means for preprocessing the retrieved information, means for generating a visual display based on the preprocessed information, and means for transmitting the generated visual display to the user's device, thereby enabling the user to quickly analyze and visualize data through simple instructions in natural language without requiring specialized knowledge.

[0430] A "user" is a person or entity that operates the system and inputs instructions in natural language.

[0431] "Natural language" refers to language used by humans on a daily basis, and refers to common words and phrases that do not require specific technical skills or knowledge.

[0432] "Instructions" refer to requests or commands given by a user to a system, specifically input in natural language for the purpose of obtaining or visualizing data.

[0433] "Natural language processing" is a technology that allows computers to understand and analyze human language (natural language), and includes the process of extracting meaning from text data.

[0434] A "data storage device" is a computer system for storing and managing large amounts of data, and includes technologies such as data warehouses and databases.

[0435] "Information" refers to data retrieved from a data storage device based on a user's instruction, and includes data of a specific period and type.

[0436] "Preprocessing" refers to a series of operations that prepare acquired information in a form suitable for generating a visual display, including data cleansing and filtering.

[0437] "Visual displays" refer to data display formats such as graphs and charts that are created to make acquired information easier to understand visually.

[0438] "Visual representation rendering tools" refers to software libraries and platforms used to efficiently generate visual representations.

[0439] In this invention, a user can use natural language commands to acquire and visualize data, and the system automatically processes the data based on the commands and generates and displays graphs. Specific embodiments of the system are described below.

[0440] Overall overview

[0441] This system automates a series of processes: interpreting a user's natural language instructions, retrieving the necessary information from a data storage device, performing preprocessing, generating a graph as a visual display, and finally displaying it on the user's device.

[0442] Natural language input

[0443] The user inputs a command in natural language into their own terminal, such as "Display this month's sales data in a bar graph." The user can freely specify the period, type, display format, etc. of the data.

[0444] Natural Language Processing

[0445] The device sends the instructions entered by the user to the server, which then analyzes them using a natural language processing engine (e.g., spaCy or NLTK). The analysis extracts information such as the data period (e.g., this month), the type of data (e.g., sales data), and the graph format (e.g., bar graph).

[0446] Data Acquisition

[0447] The server generates a query to a data store (such as a data warehouse) based on the analysis results. The query is to obtain the specific information the user is looking for (e.g., this month's sales data). As a result of the query, the data store sends the required information to the server.

[0448] Data Preprocessing

[0449] The server pre-processes the information it receives, which includes data cleansing (removing duplicates and imputing missing values), filtering, and aggregation, making the information suitable for generating graphs in the specified format.

[0450] Graph Generation

[0451] Based on the preprocessed information, the server generates a visual representation in the specified format (e.g., a bar graph) according to the user's instructions, using a graph drawing library such as Matplotlib or Plotly.

[0452] Sending and displaying graphs

[0453] The generated graphs are exported in image format (e.g. PNG or JPEG files) or interactive format (e.g. HTML or JavaScript) and sent from the server to the user's device, where the received graph data is displayed so that the user can visually check it.

[0454] Specific examples

[0455] For example, consider the case where a user inputs "Display last month's sales data as a line graph." In this case, the user's device sends the instruction to the server, which analyzes the instruction using a natural language processing engine. As a result of the analysis, the information "last month," "sales data," and "line graph" is extracted, and the server sends a query to the data storage device based on this information to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. The final graph is sent to the user's device and displayed.

[0456] In this way, this system automatically obtains and visualizes the necessary information based on the user's natural language instructions, making data analysis possible without requiring specialized knowledge.The user simply commands, "Display the data for XX in a XX graph," and can quickly obtain a visual display of the data.

[0457] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0458] Step 1:

[0459] A user inputs a command in natural language into his / her terminal. For example, the user inputs "Show me this month's sales data in a bar graph." The input data is a natural language command in the form of a string.

[0460] Input: "Show me this month's sales data in a bar chart"

[0461] Output: Natural language instructions sent from the device to the server

[0462] Step 2:

[0463] The terminal sends the received natural language instruction to the server, where it sends the instruction as is without converting its format.

[0464] Input: User's natural language instructions

[0465] Output: Natural language instructions sent to the server

[0466] Step 3:

[0467] The server analyzes the received instructions using a natural language processing engine (e.g., spaCy or NLTK), extracting information such as the data period, type, and graph format.

[0468] Input: Natural language instructions

[0469] Output: Data period (e.g., this month), data type (e.g., sales data), graph format (e.g., bar graph)

[0470] Step 4:

[0471] Based on the analysis results, the server generates a query to a data storage device (such as a DWH). This query is written in a format such as SQL and is intended to obtain the required information.

[0472] Input: Analysis results (data period, type, graph format)

[0473] Output: Generated query (e.g. "SELECT FROM sales WHERE date >= '2023-10-01' AND date <= '2023-10-31'")

[0474] Step 5:

[0475] The server sends the generated query to the data storage device to retrieve the specified data (sales data for this month). The query results are returned from the data storage device.

[0476] Input: Generated query

[0477] Output: Retrieved data (e.g., this month's sales data)

[0478] Step 6:

[0479] The server pre-processes the acquired data, which includes data cleansing (removing duplicate data and filling in missing values), filtering, and aggregation.

[0480] Input: Retrieved data

[0481] Output: Preprocessed data

[0482] Step 7:

[0483] The server generates a visual display in the specified format (e.g., a bar graph) based on the preprocessed data, using a visual display drawing tool such as Matplotlib or Plotly.

[0484] Input: Preprocessed data

[0485] Output: Generated graph (e.g. bar graph in PNG format)

[0486] Step 8:

[0487] The server sends the generated graph to the user's terminal in the form of an image file (e.g. PNG file) or an interactive HTML file.

[0488] Input: Generated graph

[0489] Output: Graph data sent to the terminal

[0490] Step 9:

[0491] The terminal displays the received graph data, including image files and interactive HTML files, in an appropriate viewer or browser.

[0492] Input: Graph data sent from the server

[0493] Output: A graphical representation on the screen that can be viewed by the user

[0494] For example, if a user types "Show me last month's sales data as a line graph," the system will do the following:

[0495] Step 1-2: The user inputs a natural language instruction, which is sent from the terminal to the server.

[0496] Step 3: The server parses the instructions and extracts the sales data for the last month in the form of a line graph.

[0497] Step 4-5: The server queries the data store to retrieve last month's sales data.

[0498] Step 6: Preprocess the data retrieved by the server.

[0499] Step 7: Generate a line graph based on the preprocessed data.

[0500] Step 8-9: The generated graph is sent to the user's terminal and displayed.

[0501] In this way, the system can automatically acquire and visualize the necessary data based on the user's instructions.

[0502] (Application example 1)

[0503] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0504] Modern production management systems generate massive amounts of data, but visually understanding this data requires specialized knowledge, making it difficult to identify and address problems in real time. Furthermore, there is a lack of tools that allow on-site managers to intuitively understand the data and respond quickly. This leads to problems such as reduced production efficiency and delayed early detection of abnormalities. The present invention aims to solve these problems by providing a system that allows users to easily instruct data visualization in natural language and respond immediately.

[0505] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0506] In this invention, the server includes means for receiving instructions entered by a user in natural language, means for analyzing the received instructions using natural language processing, means for retrieving necessary data from a data warehouse based on the analysis results, means for preprocessing the retrieved data, means for generating a graph based on the preprocessed data, means for transmitting the generated graph to the user's terminal, means for analyzing the natural language instructions and extracting a specific data type and time period, means for retrieving production management system data based on the extracted specific information, means for generating a graph in a specified format (e.g., pie chart, bar graph) based on the production management system data, and means for displaying the generated graph on a smartphone display. This simplifies the visualization of data in the production management system, enabling on-site managers to intuitively understand the data and respond quickly.

[0507] "Natural language" refers to a language that humans use on a daily basis, including written and spoken languages.

[0508] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language, and includes technology for text and voice data.

[0509] A "data warehouse" is a database system that allows companies and organizations to centrally store large amounts of data and integrate and analyze data from various data sources.

[0510] "Preprocessing" is the process of converting raw data into a form suitable for analysis and visualization, and includes data cleansing, filtering, and aggregation.

[0511] A "graph" is a diagram that visually represents data and comes in various forms, such as bar graphs, line graphs, and pie charts.

[0512] A "smartphone" is a highly functional mobile phone that can connect to the Internet and use a variety of applications.

[0513] A "production management system" is a system for managing production activities in factories and production lines, and includes production planning, progress management, quality control, etc.

[0514] A "terminal" is a device that allows a user to input information and display results, and includes smartphones, tablets, and personal computers.

[0515] A "graph drawing library" is a software library for generating graphs in programs, and includes Matplotlib and Plotly.

[0516] A "server" is a computer system that processes and provides data, and provides services in response to requests from clients.

[0517] The present invention provides a system that allows a user to use natural language to instruct data visualization in a production management system, and a server automatically processes the data and generates graphs, displaying the results on the user's terminal.

[0518] Overall overview

[0519] This system automates a series of processes: interpreting user instructions, acquiring and preprocessing specified production management data, generating graphs, and finally displaying them on a smartphone screen.

[0520] User input

[0521] Users use their smartphones to input instructions in natural language, such as "Display today's operating status of production line 1 in a pie chart." Such instructions allow users to specify a specific data period, data type, and graph format.

[0522] Natural Language Processing

[0523] The user's device sends the received instructions to the server, which then analyzes them using a natural language processing engine (e.g., spaCy or NLTK). This analysis extracts information such as the data period (e.g., today), the data type (e.g., the operating status of production line 1), and the graph format (e.g., pie chart).

[0524] Data Acquisition

[0525] The server generates a query to the production management system (data warehouse) based on the analysis results and retrieves the necessary data. This query is intended to retrieve data including operation information for a specific production line.

[0526] Data Preprocessing

[0527] The server pre-processes the data it receives, which includes cleansing, filtering, and aggregating the data, converting it into an optimal format for generating graphs.

[0528] Graph Generation

[0529] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a pie chart) using a graph drawing library such as Matplotlib or Plotly.

[0530] Sending and displaying graphs

[0531] The generated graph is exported as an image file (e.g., PNG) and sent from the server to the user's smartphone, where it displays the received graph data for the user to visually check.

[0532] For example, if a user inputs "Display this month's downtime data for production line 2 as a bar graph," the system operates as follows: The user's device sends the instruction to the server, which analyzes it. As a result of the analysis, the following information is extracted: "This month," "Production line 2," "Downtime data," and "Bar graph." Based on this information, the server sends a query to the data warehouse to obtain "This month's downtime data for production line 2." Next, the server preprocesses the data and then generates a bar graph. The final graph is sent to the user's smartphone and displayed.

[0533] This system automatically acquires and visualizes the necessary data based on the user's natural language instructions, making it possible to analyze production data without requiring specialized knowledge. Users can quickly obtain a visual display of data by simply instructing, "Display the data for XX in a XX graph."

[0534] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0535] Step 1:

[0536] A user uses a smartphone to input instructions in natural language, such as "Display today's operating status of production line 1 in a pie chart." The input instruction is saved in an input field on the smartphone and is ready to be sent to the server.

[0537] Step 2:

[0538] The device sends the user's instructions to the server. The smartphone sends the input text data to the server using an HTTP request or WebSocket. The server receives the received natural language instructions and prepares them for analysis.

[0539] Step 3:

[0540] The server uses a natural language processing engine (e.g., spaCy or NLTK) to analyze the received instructions. Specifically, it tokenizes the text data, identifies nouns and verbs, and extracts the time period (e.g., today), data type (e.g., the operation status of production line 1), and graph format (e.g., pie chart) based on the context. The extracted information is used in the next step.

[0541] Step 4:

[0542] The server generates a specific query for the production management system (data warehouse) based on the analysis results. The generated query is sent as an SQL query or API request to obtain today's data on the operating status of production line 1. The production management system returns data that matches the specified conditions.

[0543] Step 5:

[0544] The server receives the acquired data and performs preprocessing, which includes data cleansing (e.g., filling in missing values ​​and removing duplicates), filtering (e.g., excluding unnecessary data), and aggregation (e.g., aggregating operating status by time). The preprocessed data is then converted into a format suitable for graph generation.

[0545] Step 6:

[0546] The server generates a graph in the specified format (e.g., pie chart) based on the preprocessed data. Graph drawing libraries such as Matplotlib and Plotly are used to generate the graph. Specifically, data points are mapped to each part of the graph and visual elements (color, label, etc.) are added. The generated graph is exported as an image format (e.g., PNG).

[0547] Step 7:

[0548] The server sends the generated graph to the user's smartphone. The image file is sent as an HTTP response or WebSocket message. The smartphone displays the received image file in the specified display area.

[0549] Step 8:

[0550] Users can visually check the generated graphs on their smartphone screen, which allows them to intuitively understand the operating status of the specified production line and take appropriate action.

[0551] This series of steps enables users to easily use natural language to instruct data visualization and grasp the status of production management in real time.

[0552] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0553] This invention combines a system that automatically acquires and visualizes data based on instructions entered by a user in natural language with an emotion engine that recognizes the user's emotions. An embodiment of this invention will be described in detail below.

[0554] Overall overview

[0555] The system allows users to input instructions in natural language, retrieves the necessary data based on those instructions, and generates and displays graphs. It can also recognize the emotions contained in the user's instructions and optimize the color and format of the displayed graphs based on those emotions.

[0556] User input

[0557] For example, a user inputs a command in natural language into their device, such as: "Show me this month's sales data in a bar graph." The user can freely specify the period, type, and display format of the data.

[0558] Natural Language Processing and Emotion Recognition

[0559] The device sends the user's instructions to the server, which first analyzes them using a natural language processing engine (e.g., spaCy or NLTK) and an emotion engine (e.g., Sentiment Analysis API). The natural language processing extracts information such as the data period (e.g., this month), data type (e.g., sales data), and graph format (e.g., bar graph), and the emotion engine recognizes the emotion (e.g., joy, sadness, anger) contained in the user's instructions.

[0560] Data Acquisition and Emotional Response

[0561] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. The server also takes into account the results of the emotion engine and optimizes the data display format and graph colors according to the user's emotions. For example, if the user is expressing positive emotions, a bright color graph will be selected.

[0562] Data Preprocessing

[0563] The server pre-processes the retrieved data. Pre-processing includes data cleansing (e.g., removing duplicates, imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating monthly sales totals). At this stage, user sentiment also influences data processing. For example, dissatisfied users may be provided with more detailed data to help find the root cause of the problem.

[0564] Graph Generation

[0565] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a bar graph). Graph drawing libraries such as Matplotlib and Plotly are used to generate the graph. Again, the color and format of the graph are adjusted based on the results of the emotion engine.

[0566] Sending and displaying graphs

[0567] The generated graphs are exported in image formats (such as PNG or JPEG) or interactive formats (such as HTML or JavaScript) and sent from the server to the terminal, where the received graph data is displayed so that the user can visually check it.

[0568] Specific examples

[0569] For example, if a user inputs, "Display last month's sales data as a line graph, and do not use bright colors because sales were low," the system operates as follows: First, the user's device sends the instruction to the server, which analyzes it. As a result of the analysis, information such as "last month," "sales data," "line graph," and "negative sentiment" is extracted. Based on this information, the server sends a query to the DWH to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. Based on the results of the sentiment engine, a graph with darker colors is selected. The final graph is sent to the user's device and displayed.

[0570] In this way, the system can quickly and effectively acquire and visualize data based on the user's verbal instructions and emotions. Users do not need advanced expertise; they can simply input an instruction, including emotional input, such as "Display the data for XX in a XX graph," to obtain an optimized visual data display.

[0571] The processing flow will be explained below.

[0572] Step 1:

[0573] The user inputs a command into their device, such as "Display this month's sales data as a bar graph." The user can also input specific commands, including emotions (e.g., "I'm disappointed with the low sales figures, so I want to use dark colors").

[0574] Step 2:

[0575] The terminal sends the instructions entered by the user to the server. The user's terminal sends the input data to the server as an HTTP request.

[0576] Step 3:

[0577] The server parses the received user instructions using a natural language processing engine (e.g., spaCy, NLTK), and extracts important information from the instructions, such as the time period of the data (e.g., this month), the type of data (e.g., sales data), and the graph format (e.g., bar graph).

[0578] Step 4:

[0579] The server uses an emotion engine (e.g., Sentiment Analysis API) to recognize emotions from the analysis results of the natural language processing engine. For example, the emotion engine extracts negative emotions such as "disappointed" or "depressed."

[0580] Step 5:

[0581] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. Specifically, the server generates an SQL query and executes the query against the data warehouse.

[0582] Step 6:

[0583] The server receives the data retrieved from the data warehouse and performs pre-processing, which includes data cleansing (e.g., removing duplicate data and imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating the total sales by month).

[0584] Step 7:

[0585] The server generates graphs in the specified format (e.g., bar graphs) based on the preprocessed data and in accordance with the user's instructions. Graph drawing libraries such as Matplotlib and Plotly are used to generate graphs. The results of the emotion engine are reflected here, and if the user's instructions indicate negative emotions, the graph's color scheme is set to a darker shade.

[0586] Step 8:

[0587] The server converts the generated graphs into a suitable format for transmission to the user's device. Graphs can be exported into image formats (e.g. PNG, JPEG) or interactive formats (e.g. HTML, JavaScript).

[0588] Step 9:

[0589] The server sends the exported graph data to the user's terminal as an HTTP response.

[0590] Step 10:

[0591] The terminal displays the received graph data. The user's terminal renders the received data on the screen and displays it so that the user can visually confirm it.

[0592] Through this series of steps, users simply communicate simple natural language instructions and emotions to the system, which then automatically retrieves the data and visualizes it in an appropriate format.

[0593] Example 2

[0594] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0595] Conventional data visualization systems display data without considering the user's emotions, which means they are unable to provide optimal visualizations that reflect the user's intentions and emotions. While systems exist that can respond to natural language instructions, there is a lack of technology that can recognize emotions based on those instructions and reflect them in data visualizations. Therefore, there is a need for a system that allows users to input instructions in natural language, recognizes emotions based on those instructions, and provides appropriate data visualization.

[0596] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0597] In this invention, the server includes means for receiving instructions input by a user in natural language, means for analyzing the received instructions using natural language processing, means for recognizing the user's emotions from the analyzed instructions, means for acquiring necessary data from an information set based on the analysis results, means for preprocessing the acquired data, means for generating a graph by optimizing the format and color of the graph based on the preprocessed data in accordance with the user's emotions, and means for transmitting the generated graph to the user's terminal, thereby enabling optimal data visualization in accordance with the user's emotions.

[0598] "User" refers to a person who inputs instructions to the system in natural language.

[0599] "Natural language" refers to the words and written forms that humans use on a daily basis.

[0600] "Instructions" refer to requests made by the user to the system regarding data acquisition and display format.

[0601] "Server" refers to a central processing unit that receives user instructions, processes them, and returns the results to the user.

[0602] "Terminal" refers to an electronic device that allows a user to input instructions.

[0603] "Natural language processing" refers to the technology that allows computers to understand and analyze human language.

[0604] "Emotion recognition" refers to technology that analyzes and identifies the emotions contained in a user's instructions.

[0605] An "information collection" refers to a place that stores large amounts of data, such as a data warehouse or database.

[0606] "Preprocessing" refers to the process of organizing and shaping acquired data and converting it into a format suitable for analysis and visualization.

[0607] A "graph" refers to a diagram or chart that visually represents data.

[0608] "Graph optimization" refers to adjusting the format, color, and display method of a graph based on user sentiment.

[0609] "Generation" refers to the act of creating a graph in a specified format based on preprocessed data.

[0610] "Transmission" refers to the act of sending the graph generated by the server to the terminal and displaying it to the user.

[0611] "Visualization" refers to the presentation of information or data to a user in a graphical format.

[0612] This invention combines a system that automatically acquires and visualizes data based on instructions entered by a user in natural language with an emotion engine that recognizes the user's emotions. An embodiment of this invention will be described in detail below.

[0613] Overall overview

[0614] The system allows users to input instructions in natural language, retrieves the necessary data based on those instructions, and generates and displays graphs. It can also recognize the emotions contained in the user's instructions and optimize the color and format of the displayed graphs based on those emotions.

[0615] Hardware and software used

[0616] Server: A central processing unit that receives and processes user instructions.

[0617] Terminal: An electronic device (e.g., computer, smartphone) that allows a user to input instructions and display results.

[0618] Natural language processing engine: Software for parsing a user's natural language instructions, such as spaCy or NLTK.

[0619] Emotion recognition engine: Software for analyzing user emotions, such as the Sentiment Analysis API.

[0620] Data Warehouse (DWH): A database system for storing data and retrieving required data using queries.

[0621] Graph drawing libraries: Software for visualizing data in the form of graphs, such as Matplotlib or Plotly.

[0622] User input

[0623] For example, a user inputs a command in natural language into their device, such as: "Show me this month's sales data in a bar graph." The user can freely specify the period, type, and display format of the data.

[0624] Natural Language Processing and Emotion Recognition

[0625] The device sends the user's instructions to the server, which first analyzes them using a natural language processing engine (e.g., spaCy or NLTK) and an emotion engine (e.g., Sentiment Analysis API). The natural language processing extracts information such as the data period (e.g., this month), data type (e.g., sales data), and graph format (e.g., bar graph), and the emotion engine recognizes the emotion (e.g., joy, sadness, anger) contained in the user's instructions.

[0626] Data Acquisition and Emotional Response

[0627] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. The server also takes into account the results of the emotion engine and optimizes the data display format and graph colors according to the user's emotions. For example, if the user is expressing positive emotions, a bright color graph will be selected.

[0628] Data Preprocessing

[0629] The server pre-processes the retrieved data. Pre-processing includes data cleansing (e.g., removing duplicates, imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating monthly sales totals). At this stage, user sentiment also influences data processing. For example, dissatisfied users may be provided with more detailed data to help find the root cause of the problem.

[0630] Graph Generation

[0631] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a bar graph) using a graph drawing library such as Matplotlib or Plotly. Again, the color and format of the graph are adjusted based on the results of the emotion engine.

[0632] Sending and displaying graphs

[0633] The generated graphs are exported in image formats (such as PNG or JPEG) or interactive formats (such as HTML or JavaScript) and sent from the server to the terminal, where the received graph data is displayed so that the user can visually check it.

[0634] Specific examples

[0635] For example, if a user inputs, "Display last month's sales data as a line graph, and do not use bright colors because sales were low," the system operates as follows: First, the user's device sends the instruction to the server, which analyzes it. As a result of the analysis, information such as "last month," "sales data," "line graph," and "negative sentiment" is extracted. Based on this information, the server sends a query to the DWH to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. Based on the results of the sentiment engine, a graph with darker colors is selected. The final graph is sent to the user's device and displayed.

[0636] In this way, the system can quickly and effectively acquire and visualize data based on the user's verbal instructions and emotions. Users do not need advanced expertise; they can simply input an instruction, including emotional input, such as "Display the data for XX in a XX graph," to obtain an optimized visual data display.

[0637] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0638] Step 1: User input

[0639] The user inputs a command in natural language using a terminal. For example, the command might say, "Display this month's sales data in a bar graph." This command includes the data period ("this month"), type ("sales data"), and display format ("bar graph").

[0640] Input: User instructions in natural language

[0641] Output: Typed instructions retained on the terminal

[0642] Step 2: Send instructions to the server

[0643] The terminal transmits the input instructions to the server. This data transmission is carried out via the Internet.

[0644] Input: User's natural language instructions

[0645] Output: Instruction data sent to the server

[0646] Step 3: Parsing instructions (natural language processing and emotion recognition)

[0647] The server analyzes the received instructions using a natural language processing engine (e.g., spaCy). This analysis extracts information such as the data period, type, and display format. At the same time, it analyzes the user's emotions using an emotion recognition engine (e.g., Sentiment Analysis API).

[0648] Input: Instruction data sent to the server

[0649] Output: Analysis result data period, type, display format, emotional information

[0650] Step 4: Obtaining the necessary data

[0651] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. The server executes SQL queries according to the period and type and retrieves the relevant data from the database.

[0652] Input: Analysis results

[0653] Output: Required data obtained from DWH

[0654] Step 5: Preprocessing the data

[0655] The server preprocesses the acquired data, including removing duplicate data, imputing missing values, filtering periods, and aggregating data. It also adjusts the level of detail of the data according to the user's sentiment.

[0656] Input: Required data

[0657] Output: Preprocessed data

[0658] Step 6: Generate the graph

[0659] The server generates graphs in the specified format (e.g., bar graphs) based on the preprocessed data. It uses Matplotlib and Plotly to generate graphs, optimizing the color and format of the graphs based on the user's preferences.

[0660] Input: Preprocessed data, emotion information

[0661] Output: The generated graph

[0662] Step 7: Send and display the graph

[0663] The server sends the generated graph to the terminal, where it can be exported in image format (PNG or JPEG) or interactive format (HTML or JavaScript) and displayed on the terminal for the user to view.

[0664] Input: Generated graph

[0665] Output: Graph displayed on the user's terminal

[0666] Specific examples

[0667] For example, if a user inputs "Display this month's sales data in a bar graph," the system operates as follows: First, the user's device sends the instruction to the server. The server analyzes the instruction and extracts information such as "this month," "sales data," and "bar graph." It then retrieves "this month's sales data" from the DWH based on the analysis results. The server preprocesses the data and generates a bar graph based on the preprocessed data. The final graph is sent to the user's device and displayed. This allows users to easily obtain visualized data based on instructions in natural language.

[0668] (Application example 2)

[0669] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0670] Conventional data visualization systems analyze users' natural language instructions to visualize data, but they do not optimize the system to take into account the user's emotions, resulting in a lack of improvement in the user experience. Furthermore, particularly in the entertainment industry, users often seek information and content that reflects their current emotions, creating a need for appropriate data and content that takes emotions into account.

[0671] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving instructions entered by a user in natural language, means for analyzing the received instructions using natural language processing, means for retrieving necessary data from a data warehouse based on the analysis result, means for preprocessing the retrieved data, means for generating a graph based on the preprocessed data, means for transmitting the generated graph to the user's terminal, means for recognizing the user's emotion based on the analyzed instructions, and means for optimizing the format and color of displayed data based on the recognized emotion. This enables optimal data visualization and content display according to the user's emotion.

[0672] A "user" is a person who uses the system to input instructions in natural language.

[0673] A "natural language" is a language that humans use on a daily basis, and is not a specific programming language.

[0674] An "instruction" represents a request or command given by a user to a system.

[0675] "Analysis" is the process of mechanically breaking down instructions entered in natural language and understanding their meaning.

[0676] A "data warehouse" is a system for centrally storing and managing large amounts of data.

[0677] "Preprocessing" is the process of processing acquired data through cleansing, filtering, and other processes to prepare it in a format suitable for analysis and visualization.

[0678] A "graph" is a diagram that visually represents numerical data.

[0679] A "terminal" is a device through which a user accesses and operates the system.

[0680] "Emotion" refers to the psychological state of the user.

[0681] "Optimization" means adjusting something to the most effective state for a specific purpose.

[0682] This invention combines a system that allows a user to input instructions in natural language and automatically acquires and visualizes data based on the instructions with an emotion engine that recognizes the user's emotions. The system includes the following steps:

[0683] First, the user inputs a command in natural language into their device. For example, they might say, "Tell me the next interesting movie" or "Tell me a movie that will help me relax when I'm tired." This is the initial input stage performed by the user.

[0684] The device sends these instructions to the server, which then uses a natural language processing engine (e.g., spaCy) to analyze the user's instructions. At this stage, the server extracts the content of the instructions (e.g., the genre and characteristics of the movie being searched for) and simultaneously analyzes the user's emotions using an emotion engine (e.g., Sentiment Analysis API).

[0685] Based on the analysis results, the server retrieves the necessary data from the data warehouse, including movie information and rating data from the content database. For example, if a user requests "relaxing movies," the server will extract movies that meet that criteria.

[0686] The acquired data is pre-processed by the server, which includes data cleansing, filtering, and optionally aggregating the data. The server then optimizes the data based on the results of the emotion engine. If the user's emotion is relaxation, a list of movies that match that emotion is generated. The visual display format and colors are also adjusted based on the emotion.

[0687] Finally, the preprocessed and optimized data may be graphed using a graph drawing library (e.g., Matplotlib or Plotly). This visual graph or content list is sent to the user's device and displayed to the user.

[0688] Specific examples

[0689] For example, if a user types, "I've been feeling stressed lately, can you recommend some relaxing movies?", the system analyzes the instruction, extracts "relaxing movies" as a movie genre, and uses an emotion engine to recognize the user's stress level. It then retrieves information about relaxing movies from a data warehouse and generates an optimized list based on the user's emotions. Finally, the movie list, with its background color and display format adjusted, is displayed on the user's device.

[0690] Prompt Sentence Examples

[0691] "I've been feeling stressed lately, so can you recommend a relaxing movie?"

[0692] This invention allows users to easily obtain data visualization and content display that best fits their emotional state. The system takes into account the user's emotions and provides the most appropriate information at that time, greatly improving the user experience.

[0693] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0694] Step 1:

[0695] The user inputs instructions in natural language, such as "Tell me the next good movie." This is the initial input that occurs on the terminal. The input instructions are sent directly to the server.

[0696] Step 2:

[0697] The server receives instructions in natural language sent from the device. The server analyzes the instructions using a natural language processing engine. As a result of the analysis, it extracts information such as the search target and display format. For example, if the instruction is "Tell me the next good movie," the search target will be "good movies."

[0698] Step 3:

[0699] The server uses an emotion engine to recognize the user's emotions based on the analysis results. Emotions (e.g., positive or negative emotions) are extracted from the content of the instruction. For example, in the case of "Tell me a movie that helps me relax when I'm tired," "fatigue" is recognized as the emotion.

[0700] Step 4:

[0701] The server retrieves the necessary data (e.g., movie information) from the data warehouse based on the analysis results and the recognized emotions. The server then sends a query to the database to retrieve the corresponding movie list.

[0702] Step 5:

[0703] The server preprocesses the retrieved data. Preprocessing includes data cleansing, filtering, and, if necessary, data aggregation. Data cleansing involves filling in missing values ​​and removing duplicates, while filtering extracts only data that corresponds to the user's instructions. For example, if the search term is "relaxing movies," only movie information that falls into this category is filtered.

[0704] Step 6:

[0705] The server then uses the pre-processed data to optimize the data based on the results of the emotion engine, for example, visualizing positive emotions with bright colors and negative emotions with muted colors, as well as adjusting the order of the list display and the emphasis of information.

[0706] Step 7:

[0707] The server uses a graph drawing library (such as Matplotlib or Plotly) to generate graphs based on the optimized data. When the user requests a visualization, a graph in the appropriate format is drawn.

[0708] Step 8:

[0709] The server sends the generated graph or optimized content list to the user's device. The device visually displays the received data. The display format and color are optimized according to the user's emotions.

[0710] Step 9:

[0711] The user checks the results displayed on the device, such as a list of relaxing movies and graphs, which are expected to have a psychologically soothing effect.

[0712] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0713] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0714] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0715] [Third embodiment]

[0716] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0717] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0718] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0719] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0720] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0721] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0722] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0723] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0724] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0725] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0726] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0727] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0728] In this invention, the user simply issues a command in natural language to acquire and visualize data, and the system automatically processes the data and generates and displays a graph. Specific embodiments of the system are described below.

[0729] Overall overview

[0730] This system automates a series of processes: interpreting user instructions, retrieving the necessary data from a data warehouse, performing preprocessing, generating graphs, and finally displaying the results on the user's device.

[0731] User input

[0732] For example, a user inputs a command in natural language into their device, such as: "Show me this month's sales data in a bar graph." The user can freely specify the period, type, and display format of the data.

[0733] Natural Language Processing

[0734] The device sends the user's instructions to the server, which first analyzes them using a natural language processing engine (e.g., spaCy or NLTK). Through the analysis, information such as the data period (e.g., this month), data type (e.g., sales data), and graph format (e.g., bar graph) is extracted.

[0735] Data Acquisition

[0736] The server generates a query to the data warehouse (DWH) based on the analysis results. This query is to obtain the specific data the user is looking for (this month's sales data). As a result of the query, the required data is sent from the DWH to the server.

[0737] Data Preprocessing

[0738] The server pre-processes the data it receives, which includes data cleansing (removing duplicates and imputing missing values), filtering, and aggregation, making the data suitable for generating graphs in the specified format.

[0739] Graph Generation

[0740] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a bar graph) according to the user's instructions. To generate the graph, it uses a graph drawing library such as Matplotlib or Plotly.

[0741] Sending and displaying graphs

[0742] The generated graphs are exported in image formats (e.g. PNG or JPEG) or interactive formats (e.g. HTML or JavaScript) and sent from the server to the terminal, which then displays the received graph data so that the user can visually check it.

[0743] Specific examples

[0744] For example, if a user inputs "Display last month's sales data as a line graph," the system operates as follows: First, the user's device sends the instruction to the server, which analyzes it. As a result of the analysis, the information extracted is "last month," "sales data," and "line graph." Based on this information, the server sends a query to the DWH to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. The final graph is sent to the user's device and displayed.

[0745] In this way, this system automatically acquires and visualizes the necessary data based on the user's verbal instructions, making data analysis possible without requiring specialized knowledge.The user simply commands, "Display the data for XX in a XX graph," and can quickly obtain a visual display of the data.

[0746] The processing flow will be explained below.

[0747] Step 1:

[0748] The user inputs a command into their own device, such as "Display this month's sales data in a bar graph." The user inputs the specific data period, data type, graph format, etc. in natural language.

[0749] Step 2:

[0750] The terminal sends the instructions entered by the user to the server. The user's terminal sends the input data to the server as an HTTP request.

[0751] Step 3:

[0752] The server parses the received user instructions using a natural language processing engine (e.g., spaCy, NLTK), and extracts important information from the instructions, such as the time period of the data (e.g., this month), the type of data (e.g., sales data), and the graph format (e.g., bar graph).

[0753] Step 4:

[0754] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. Specifically, the server generates an SQL query and executes the query against the data warehouse.

[0755] Step 5:

[0756] The server receives the data retrieved from the data warehouse and performs pre-processing, which includes data cleansing (e.g., removing duplicate data and imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating the total sales by month).

[0757] Step 6:

[0758] The server generates a graph in the specified format (e.g., a bar graph) based on the preprocessed data, using a graph drawing library such as Matplotlib or Plotly.

[0759] Step 7:

[0760] The server converts the generated graphs into a suitable format for transmission to the user's device. Graphs can be exported into image formats (e.g. PNG, JPEG) or interactive formats (e.g. HTML, JavaScript).

[0761] Step 8:

[0762] The server sends the exported graph data to the user's terminal as an HTTP response.

[0763] Step 9:

[0764] The terminal displays the received graph data. The user's terminal renders the received data on the screen and displays it so that the user can visually confirm it.

[0765] This series of steps allows users to quickly retrieve and visualize data using simple natural language commands.

[0766] Example 1

[0767] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0768] In today's information society, users need to quickly and visually understand the information they need from large amounts of data. However, conventional data analysis tools require specialized knowledge, making data analysis and visualization difficult and preventing rapid response. Furthermore, the time and effort required for data preprocessing and graph generation makes it difficult to achieve efficient data analysis.

[0769] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0770] In this invention, the server includes means for receiving instructions input by a user in natural language, means for analyzing the received instructions using natural language processing, means for retrieving necessary information from a data storage device based on the analysis result, means for preprocessing the retrieved information, means for generating a visual display based on the preprocessed information, and means for transmitting the generated visual display to the user's device, thereby enabling the user to quickly analyze and visualize data through simple instructions in natural language without requiring specialized knowledge.

[0771] A "user" is a person or entity that operates the system and inputs instructions in natural language.

[0772] "Natural language" refers to language used by humans on a daily basis, and refers to common words and phrases that do not require specific technical skills or knowledge.

[0773] "Instructions" refer to requests or commands given by a user to a system, specifically input in natural language for the purpose of obtaining or visualizing data.

[0774] "Natural language processing" is a technology that allows computers to understand and analyze human language (natural language), and includes the process of extracting meaning from text data.

[0775] A "data storage device" is a computer system for storing and managing large amounts of data, and includes technologies such as data warehouses and databases.

[0776] "Information" refers to data retrieved from a data storage device based on a user's instruction, and includes data of a specific period and type.

[0777] "Preprocessing" refers to a series of operations that prepare acquired information in a form suitable for generating a visual display, including data cleansing and filtering.

[0778] "Visual displays" refer to data display formats such as graphs and charts that are created to make acquired information easier to understand visually.

[0779] "Visual representation rendering tools" refers to software libraries and platforms used to efficiently generate visual representations.

[0780] In this invention, a user can use natural language commands to acquire and visualize data, and the system automatically processes the data based on the commands and generates and displays graphs. Specific embodiments of the system are described below.

[0781] Overall overview

[0782] This system automates a series of processes: interpreting a user's natural language instructions, retrieving the necessary information from a data storage device, performing preprocessing, generating a graph as a visual display, and finally displaying it on the user's device.

[0783] Natural language input

[0784] The user inputs a command in natural language into their own terminal, such as "Display this month's sales data in a bar graph." The user can freely specify the period, type, display format, etc. of the data.

[0785] Natural Language Processing

[0786] The device sends the instructions entered by the user to the server, which then analyzes them using a natural language processing engine (e.g., spaCy or NLTK). The analysis extracts information such as the data period (e.g., this month), the type of data (e.g., sales data), and the graph format (e.g., bar graph).

[0787] Data Acquisition

[0788] The server generates a query to a data store (such as a data warehouse) based on the analysis results. The query is to obtain the specific information the user is looking for (e.g., this month's sales data). As a result of the query, the data store sends the required information to the server.

[0789] Data Preprocessing

[0790] The server pre-processes the information it receives, which includes data cleansing (removing duplicates and imputing missing values), filtering, and aggregation, making the information suitable for generating graphs in the specified format.

[0791] Graph Generation

[0792] Based on the preprocessed information, the server generates a visual representation in the specified format (e.g., a bar graph) according to the user's instructions, using a graph drawing library such as Matplotlib or Plotly.

[0793] Sending and displaying graphs

[0794] The generated graphs are exported in image format (e.g. PNG or JPEG files) or interactive format (e.g. HTML or JavaScript) and sent from the server to the user's device, where the received graph data is displayed so that the user can visually check it.

[0795] Specific examples

[0796] For example, consider the case where a user inputs "Display last month's sales data as a line graph." In this case, the user's device sends the instruction to the server, which analyzes the instruction using a natural language processing engine. As a result of the analysis, the information "last month," "sales data," and "line graph" is extracted, and the server sends a query to the data storage device based on this information to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. The final graph is sent to the user's device and displayed.

[0797] In this way, this system automatically obtains and visualizes the necessary information based on the user's natural language instructions, making data analysis possible without requiring specialized knowledge.The user simply commands, "Display the data for XX in a XX graph," and can quickly obtain a visual display of the data.

[0798] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0799] Step 1:

[0800] A user inputs a command in natural language into his / her terminal. For example, the user inputs "Show me this month's sales data in a bar graph." The input data is a natural language command in the form of a string.

[0801] Input: "Show me this month's sales data in a bar chart"

[0802] Output: Natural language instructions sent from the device to the server

[0803] Step 2:

[0804] The terminal sends the received natural language instruction to the server, where it sends the instruction as is without converting its format.

[0805] Input: User's natural language instructions

[0806] Output: Natural language instructions sent to the server

[0807] Step 3:

[0808] The server analyzes the received instructions using a natural language processing engine (e.g., spaCy or NLTK), extracting information such as the data period, type, and graph format.

[0809] Input: Natural language instructions

[0810] Output: Data period (e.g., this month), data type (e.g., sales data), graph format (e.g., bar graph)

[0811] Step 4:

[0812] Based on the analysis results, the server generates a query to a data storage device (such as a DWH). This query is written in a format such as SQL and is intended to obtain the required information.

[0813] Input: Analysis results (data period, type, graph format)

[0814] Output: Generated query (e.g. "SELECT FROM sales WHERE date >= '2023-10-01' AND date <= '2023-10-31'")

[0815] Step 5:

[0816] The server sends the generated query to the data storage device to retrieve the specified data (sales data for this month). The query results are returned from the data storage device.

[0817] Input: Generated query

[0818] Output: Retrieved data (e.g., this month's sales data)

[0819] Step 6:

[0820] The server pre-processes the acquired data, which includes data cleansing (removing duplicate data and filling in missing values), filtering, and aggregation.

[0821] Input: Retrieved data

[0822] Output: Preprocessed data

[0823] Step 7:

[0824] The server generates a visual display in the specified format (e.g., a bar graph) based on the preprocessed data, using a visual display drawing tool such as Matplotlib or Plotly.

[0825] Input: Preprocessed data

[0826] Output: Generated graph (e.g. bar graph in PNG format)

[0827] Step 8:

[0828] The server sends the generated graph to the user's terminal in the form of an image file (e.g. PNG file) or an interactive HTML file.

[0829] Input: Generated graph

[0830] Output: Graph data sent to the terminal

[0831] Step 9:

[0832] The terminal displays the received graph data, including image files and interactive HTML files, in an appropriate viewer or browser.

[0833] Input: Graph data sent from the server

[0834] Output: A graphical representation on the screen that can be viewed by the user

[0835] For example, if a user types "Show me last month's sales data as a line graph," the system will do the following:

[0836] Step 1-2: The user inputs a natural language instruction, which is sent from the terminal to the server.

[0837] Step 3: The server parses the instructions and extracts the sales data for the last month in the form of a line graph.

[0838] Step 4-5: The server queries the data store to retrieve last month's sales data.

[0839] Step 6: Preprocess the data retrieved by the server.

[0840] Step 7: Generate a line graph based on the preprocessed data.

[0841] Step 8-9: The generated graph is sent to the user's terminal and displayed.

[0842] In this way, the system can automatically acquire and visualize the necessary data based on the user's instructions.

[0843] (Application example 1)

[0844] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0845] Modern production management systems generate massive amounts of data, but visually understanding this data requires specialized knowledge, making it difficult to identify and address problems in real time. Furthermore, there is a lack of tools that allow on-site managers to intuitively understand the data and respond quickly. This leads to problems such as reduced production efficiency and delayed early detection of abnormalities. The present invention aims to solve these problems by providing a system that allows users to easily instruct data visualization in natural language and respond immediately.

[0846] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0847] In this invention, the server includes means for receiving instructions entered by a user in natural language, means for analyzing the received instructions using natural language processing, means for retrieving necessary data from a data warehouse based on the analysis results, means for preprocessing the retrieved data, means for generating a graph based on the preprocessed data, means for transmitting the generated graph to the user's terminal, means for analyzing the natural language instructions and extracting a specific data type and time period, means for retrieving production management system data based on the extracted specific information, means for generating a graph in a specified format (e.g., pie chart, bar graph) based on the production management system data, and means for displaying the generated graph on a smartphone display. This simplifies the visualization of data in the production management system, enabling on-site managers to intuitively understand the data and respond quickly.

[0848] "Natural language" refers to a language that humans use on a daily basis, including written and spoken languages.

[0849] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language, and includes technology for text and voice data.

[0850] A "data warehouse" is a database system that allows companies and organizations to centrally store large amounts of data and integrate and analyze data from various data sources.

[0851] "Preprocessing" is the process of converting raw data into a form suitable for analysis and visualization, and includes data cleansing, filtering, and aggregation.

[0852] A "graph" is a diagram that visually represents data and comes in various forms, such as bar graphs, line graphs, and pie charts.

[0853] A "smartphone" is a highly functional mobile phone that can connect to the Internet and use a variety of applications.

[0854] A "production management system" is a system for managing production activities in factories and production lines, and includes production planning, progress management, quality control, etc.

[0855] A "terminal" is a device that allows a user to input information and display results, and includes smartphones, tablets, and personal computers.

[0856] A "graph drawing library" is a software library for generating graphs in programs, and includes Matplotlib and Plotly.

[0857] A "server" is a computer system that processes and provides data, and provides services in response to requests from clients.

[0858] The present invention provides a system that allows a user to use natural language to instruct data visualization in a production management system, and a server automatically processes the data and generates graphs, displaying the results on the user's terminal.

[0859] Overall overview

[0860] This system automates a series of processes: interpreting user instructions, acquiring and preprocessing specified production management data, generating graphs, and finally displaying them on a smartphone screen.

[0861] User input

[0862] Users use their smartphones to input instructions in natural language, such as "Display today's operating status of production line 1 in a pie chart." Such instructions allow users to specify a specific data period, data type, and graph format.

[0863] Natural Language Processing

[0864] The user's device sends the received instructions to the server, which then analyzes them using a natural language processing engine (e.g., spaCy or NLTK). This analysis extracts information such as the data period (e.g., today), the data type (e.g., the operating status of production line 1), and the graph format (e.g., pie chart).

[0865] Data Acquisition

[0866] The server generates a query to the production management system (data warehouse) based on the analysis results and retrieves the necessary data. This query is intended to retrieve data including operation information for a specific production line.

[0867] Data Preprocessing

[0868] The server pre-processes the data it receives, which includes cleansing, filtering, and aggregating the data, converting it into an optimal format for generating graphs.

[0869] Graph Generation

[0870] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a pie chart) using a graph drawing library such as Matplotlib or Plotly.

[0871] Sending and displaying graphs

[0872] The generated graph is exported as an image file (e.g., PNG) and sent from the server to the user's smartphone, where it displays the received graph data for the user to visually check.

[0873] For example, if a user inputs "Display this month's downtime data for production line 2 as a bar graph," the system operates as follows: The user's device sends the instruction to the server, which analyzes it. As a result of the analysis, the following information is extracted: "This month," "Production line 2," "Downtime data," and "Bar graph." Based on this information, the server sends a query to the data warehouse to obtain "This month's downtime data for production line 2." Next, the server preprocesses the data and then generates a bar graph. The final graph is sent to the user's smartphone and displayed.

[0874] This system automatically acquires and visualizes the necessary data based on the user's natural language instructions, making it possible to analyze production data without requiring specialized knowledge. Users can quickly obtain a visual display of data by simply instructing, "Display the data for XX in a XX graph."

[0875] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0876] Step 1:

[0877] A user uses a smartphone to input instructions in natural language, such as "Display today's operating status of production line 1 in a pie chart." The input instruction is saved in an input field on the smartphone and is ready to be sent to the server.

[0878] Step 2:

[0879] The device sends the user's instructions to the server. The smartphone sends the input text data to the server using an HTTP request or WebSocket. The server receives the received natural language instructions and prepares them for analysis.

[0880] Step 3:

[0881] The server uses a natural language processing engine (e.g., spaCy or NLTK) to analyze the received instructions. Specifically, it tokenizes the text data, identifies nouns and verbs, and extracts the time period (e.g., today), data type (e.g., the operation status of production line 1), and graph format (e.g., pie chart) based on the context. The extracted information is used in the next step.

[0882] Step 4:

[0883] The server generates a specific query for the production management system (data warehouse) based on the analysis results. The generated query is sent as an SQL query or API request to obtain today's data on the operating status of production line 1. The production management system returns data that matches the specified conditions.

[0884] Step 5:

[0885] The server receives the acquired data and performs preprocessing, which includes data cleansing (e.g., filling in missing values ​​and removing duplicates), filtering (e.g., excluding unnecessary data), and aggregation (e.g., aggregating operating status by time). The preprocessed data is then converted into a format suitable for graph generation.

[0886] Step 6:

[0887] The server generates a graph in the specified format (e.g., pie chart) based on the preprocessed data. Graph drawing libraries such as Matplotlib and Plotly are used to generate the graph. Specifically, data points are mapped to each part of the graph and visual elements (color, label, etc.) are added. The generated graph is exported as an image format (e.g., PNG).

[0888] Step 7:

[0889] The server sends the generated graph to the user's smartphone. The image file is sent as an HTTP response or WebSocket message. The smartphone displays the received image file in the specified display area.

[0890] Step 8:

[0891] Users can visually check the generated graphs on their smartphone screen, which allows them to intuitively understand the operating status of the specified production line and take appropriate action.

[0892] This series of steps enables users to easily use natural language to instruct data visualization and grasp the status of production management in real time.

[0893] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0894] This invention combines a system that automatically acquires and visualizes data based on instructions entered by a user in natural language with an emotion engine that recognizes the user's emotions. An embodiment of this invention will be described in detail below.

[0895] Overall overview

[0896] The system allows users to input instructions in natural language, retrieves the necessary data based on those instructions, and generates and displays graphs. It can also recognize the emotions contained in the user's instructions and optimize the color and format of the displayed graphs based on those emotions.

[0897] User input

[0898] For example, a user inputs a command in natural language into their device, such as: "Show me this month's sales data in a bar graph." The user can freely specify the period, type, and display format of the data.

[0899] Natural Language Processing and Emotion Recognition

[0900] The device sends the user's instructions to the server, which first analyzes them using a natural language processing engine (e.g., spaCy or NLTK) and an emotion engine (e.g., Sentiment Analysis API). The natural language processing extracts information such as the data period (e.g., this month), data type (e.g., sales data), and graph format (e.g., bar graph), and the emotion engine recognizes the emotion (e.g., joy, sadness, anger) contained in the user's instructions.

[0901] Data Acquisition and Emotional Response

[0902] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. The server also takes into account the results of the emotion engine and optimizes the data display format and graph colors according to the user's emotions. For example, if the user is expressing positive emotions, a bright color graph will be selected.

[0903] Data Preprocessing

[0904] The server pre-processes the retrieved data. Pre-processing includes data cleansing (e.g., removing duplicates, imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating monthly sales totals). At this stage, user sentiment also influences data processing. For example, dissatisfied users may be provided with more detailed data to help find the root cause of the problem.

[0905] Graph Generation

[0906] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a bar graph). Graph drawing libraries such as Matplotlib and Plotly are used to generate the graph. Again, the color and format of the graph are adjusted based on the results of the emotion engine.

[0907] Sending and displaying graphs

[0908] The generated graphs are exported in image formats (such as PNG or JPEG) or interactive formats (such as HTML or JavaScript) and sent from the server to the terminal, where the received graph data is displayed so that the user can visually check it.

[0909] Specific examples

[0910] For example, if a user inputs, "Display last month's sales data as a line graph, and do not use bright colors because sales were low," the system operates as follows: First, the user's device sends the instruction to the server, which analyzes it. As a result of the analysis, information such as "last month," "sales data," "line graph," and "negative sentiment" is extracted. Based on this information, the server sends a query to the DWH to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. Based on the results of the sentiment engine, a graph with darker colors is selected. The final graph is sent to the user's device and displayed.

[0911] In this way, the system can quickly and effectively acquire and visualize data based on the user's verbal instructions and emotions. Users do not need advanced expertise; they can simply input an instruction, including emotional input, such as "Display the data for XX in a XX graph," to obtain an optimized visual data display.

[0912] The processing flow will be explained below.

[0913] Step 1:

[0914] The user inputs a command into their device, such as "Display this month's sales data as a bar graph." The user can also input specific commands, including emotions (e.g., "I'm disappointed with the low sales figures, so I want to use dark colors").

[0915] Step 2:

[0916] The terminal sends the instructions entered by the user to the server. The user's terminal sends the input data to the server as an HTTP request.

[0917] Step 3:

[0918] The server parses the received user instructions using a natural language processing engine (e.g., spaCy, NLTK), and extracts important information from the instructions, such as the time period of the data (e.g., this month), the type of data (e.g., sales data), and the graph format (e.g., bar graph).

[0919] Step 4:

[0920] The server uses an emotion engine (e.g., Sentiment Analysis API) to recognize emotions from the analysis results of the natural language processing engine. For example, the emotion engine extracts negative emotions such as "disappointed" or "depressed."

[0921] Step 5:

[0922] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. Specifically, the server generates an SQL query and executes the query against the data warehouse.

[0923] Step 6:

[0924] The server receives the data retrieved from the data warehouse and performs pre-processing, which includes data cleansing (e.g., removing duplicate data and imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating the total sales by month).

[0925] Step 7:

[0926] The server generates graphs in the specified format (e.g., bar graphs) based on the preprocessed data and in accordance with the user's instructions. Graph drawing libraries such as Matplotlib and Plotly are used to generate graphs. The results of the emotion engine are reflected here, and if the user's instructions indicate negative emotions, the graph's color scheme is set to a darker shade.

[0927] Step 8:

[0928] The server converts the generated graphs into a suitable format for transmission to the user's device. Graphs can be exported into image formats (e.g. PNG, JPEG) or interactive formats (e.g. HTML, JavaScript).

[0929] Step 9:

[0930] The server sends the exported graph data to the user's terminal as an HTTP response.

[0931] Step 10:

[0932] The terminal displays the received graph data. The user's terminal renders the received data on the screen and displays it so that the user can visually confirm it.

[0933] Through this series of steps, users simply communicate simple natural language instructions and emotions to the system, which then automatically retrieves the data and visualizes it in an appropriate format.

[0934] Example 2

[0935] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0936] Conventional data visualization systems display data without considering the user's emotions, which means they are unable to provide optimal visualizations that reflect the user's intentions and emotions. While systems exist that can respond to natural language instructions, there is a lack of technology that can recognize emotions based on those instructions and reflect them in data visualizations. Therefore, there is a need for a system that allows users to input instructions in natural language, recognizes emotions based on those instructions, and provides appropriate data visualization.

[0937] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0938] In this invention, the server includes means for receiving instructions input by a user in natural language, means for analyzing the received instructions using natural language processing, means for recognizing the user's emotions from the analyzed instructions, means for acquiring necessary data from an information set based on the analysis results, means for preprocessing the acquired data, means for generating a graph by optimizing the format and color of the graph based on the preprocessed data in accordance with the user's emotions, and means for transmitting the generated graph to the user's terminal, thereby enabling optimal data visualization in accordance with the user's emotions.

[0939] "User" refers to a person who inputs instructions to the system in natural language.

[0940] "Natural language" refers to the words and written forms that humans use on a daily basis.

[0941] "Instructions" refer to requests made by the user to the system regarding data acquisition and display format.

[0942] "Server" refers to a central processing unit that receives user instructions, processes them, and returns the results to the user.

[0943] "Terminal" refers to an electronic device that allows a user to input instructions.

[0944] "Natural language processing" refers to the technology that allows computers to understand and analyze human language.

[0945] "Emotion recognition" refers to technology that analyzes and identifies the emotions contained in a user's instructions.

[0946] An "information collection" refers to a place that stores large amounts of data, such as a data warehouse or database.

[0947] "Preprocessing" refers to the process of organizing and shaping acquired data and converting it into a format suitable for analysis and visualization.

[0948] A "graph" refers to a diagram or chart that visually represents data.

[0949] "Graph optimization" refers to adjusting the format, color, and display method of a graph based on user sentiment.

[0950] "Generation" refers to the act of creating a graph in a specified format based on preprocessed data.

[0951] "Transmission" refers to the act of sending the graph generated by the server to the terminal and displaying it to the user.

[0952] "Visualization" refers to the presentation of information or data to a user in a graphical format.

[0953] This invention combines a system that automatically acquires and visualizes data based on instructions entered by a user in natural language with an emotion engine that recognizes the user's emotions. An embodiment of this invention will be described in detail below.

[0954] Overall overview

[0955] The system allows users to input instructions in natural language, retrieves the necessary data based on those instructions, and generates and displays graphs. It can also recognize the emotions contained in the user's instructions and optimize the color and format of the displayed graphs based on those emotions.

[0956] Hardware and software used

[0957] Server: A central processing unit that receives and processes user instructions.

[0958] Terminal: An electronic device (e.g., computer, smartphone) that allows a user to input instructions and display results.

[0959] Natural language processing engine: Software for parsing a user's natural language instructions, such as spaCy or NLTK.

[0960] Emotion recognition engine: Software for analyzing user emotions, such as the Sentiment Analysis API.

[0961] Data Warehouse (DWH): A database system for storing data and retrieving required data using queries.

[0962] Graph drawing libraries: Software for visualizing data in the form of graphs, such as Matplotlib or Plotly.

[0963] User input

[0964] For example, a user inputs a command in natural language into their device, such as: "Show me this month's sales data in a bar graph." The user can freely specify the period, type, and display format of the data.

[0965] Natural Language Processing and Emotion Recognition

[0966] The device sends the user's instructions to the server, which first analyzes them using a natural language processing engine (e.g., spaCy or NLTK) and an emotion engine (e.g., Sentiment Analysis API). The natural language processing extracts information such as the data period (e.g., this month), data type (e.g., sales data), and graph format (e.g., bar graph), and the emotion engine recognizes the emotion (e.g., joy, sadness, anger) contained in the user's instructions.

[0967] Data Acquisition and Emotional Response

[0968] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. The server also takes into account the results of the emotion engine and optimizes the data display format and graph colors according to the user's emotions. For example, if the user is expressing positive emotions, a bright color graph will be selected.

[0969] Data Preprocessing

[0970] The server pre-processes the retrieved data. Pre-processing includes data cleansing (e.g., removing duplicates, imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating monthly sales totals). At this stage, user sentiment also influences data processing. For example, dissatisfied users may be provided with more detailed data to help find the root cause of the problem.

[0971] Graph Generation

[0972] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a bar graph) using a graph drawing library such as Matplotlib or Plotly. Again, the color and format of the graph are adjusted based on the results of the emotion engine.

[0973] Sending and displaying graphs

[0974] The generated graphs are exported in image formats (such as PNG or JPEG) or interactive formats (such as HTML or JavaScript) and sent from the server to the terminal, where the received graph data is displayed so that the user can visually check it.

[0975] Specific examples

[0976] For example, if a user inputs, "Display last month's sales data as a line graph, and do not use bright colors because sales were low," the system operates as follows: First, the user's device sends the instruction to the server, which analyzes it. As a result of the analysis, information such as "last month," "sales data," "line graph," and "negative sentiment" is extracted. Based on this information, the server sends a query to the DWH to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. Based on the results of the sentiment engine, a graph with darker colors is selected. The final graph is sent to the user's device and displayed.

[0977] In this way, the system can quickly and effectively acquire and visualize data based on the user's verbal instructions and emotions. Users do not need advanced expertise; they can simply input an instruction, including emotional input, such as "Display the data for XX in a XX graph," to obtain an optimized visual data display.

[0978] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0979] Step 1: User input

[0980] The user inputs a command in natural language using a terminal. For example, the command might say, "Display this month's sales data in a bar graph." This command includes the data period ("this month"), type ("sales data"), and display format ("bar graph").

[0981] Input: User instructions in natural language

[0982] Output: Typed instructions retained on the terminal

[0983] Step 2: Send instructions to the server

[0984] The terminal transmits the input instructions to the server. This data transmission is carried out via the Internet.

[0985] Input: User's natural language instructions

[0986] Output: Instruction data sent to the server

[0987] Step 3: Parsing instructions (natural language processing and emotion recognition)

[0988] The server analyzes the received instructions using a natural language processing engine (e.g., spaCy). This analysis extracts information such as the data period, type, and display format. At the same time, it analyzes the user's emotions using an emotion recognition engine (e.g., Sentiment Analysis API).

[0989] Input: Instruction data sent to the server

[0990] Output: Analysis result data period, type, display format, emotional information

[0991] Step 4: Obtaining the necessary data

[0992] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. The server executes SQL queries according to the period and type and retrieves the relevant data from the database.

[0993] Input: Analysis results

[0994] Output: Required data obtained from DWH

[0995] Step 5: Preprocessing the data

[0996] The server preprocesses the acquired data, including removing duplicate data, imputing missing values, filtering periods, and aggregating data. It also adjusts the level of detail of the data according to the user's sentiment.

[0997] Input: Required data

[0998] Output: Preprocessed data

[0999] Step 6: Generate the graph

[1000] The server generates graphs in the specified format (e.g., bar graphs) based on the preprocessed data. It uses Matplotlib and Plotly to generate graphs, optimizing the color and format of the graphs based on the user's preferences.

[1001] Input: Preprocessed data, emotion information

[1002] Output: The generated graph

[1003] Step 7: Send and display the graph

[1004] The server sends the generated graph to the terminal, where it can be exported in image format (PNG or JPEG) or interactive format (HTML or JavaScript) and displayed on the terminal for the user to view.

[1005] Input: Generated graph

[1006] Output: Graph displayed on the user's terminal

[1007] Specific examples

[1008] For example, if a user inputs "Display this month's sales data in a bar graph," the system operates as follows: First, the user's device sends the instruction to the server. The server analyzes the instruction and extracts information such as "this month," "sales data," and "bar graph." It then retrieves "this month's sales data" from the DWH based on the analysis results. The server preprocesses the data and generates a bar graph based on the preprocessed data. The final graph is sent to the user's device and displayed. This allows users to easily obtain visualized data based on instructions in natural language.

[1009] (Application example 2)

[1010] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1011] Conventional data visualization systems analyze users' natural language instructions to visualize data, but they do not optimize the system to take into account the user's emotions, resulting in a lack of improvement in the user experience. Furthermore, particularly in the entertainment industry, users often seek information and content that reflects their current emotions, creating a need for appropriate data and content that takes emotions into account.

[1012] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving instructions entered by a user in natural language, means for analyzing the received instructions using natural language processing, means for retrieving necessary data from a data warehouse based on the analysis result, means for preprocessing the retrieved data, means for generating a graph based on the preprocessed data, means for transmitting the generated graph to the user's terminal, means for recognizing the user's emotion based on the analyzed instructions, and means for optimizing the format and color of displayed data based on the recognized emotion. This enables optimal data visualization and content display according to the user's emotion.

[1013] A "user" is a person who uses the system to input instructions in natural language.

[1014] A "natural language" is a language that humans use on a daily basis, and is not a specific programming language.

[1015] An "instruction" represents a request or command given by a user to a system.

[1016] "Analysis" is the process of mechanically breaking down instructions entered in natural language and understanding their meaning.

[1017] A "data warehouse" is a system for centrally storing and managing large amounts of data.

[1018] "Preprocessing" is the process of processing acquired data through cleansing, filtering, and other processes to prepare it in a format suitable for analysis and visualization.

[1019] A "graph" is a diagram that visually represents numerical data.

[1020] A "terminal" is a device through which a user accesses and operates the system.

[1021] "Emotion" refers to the psychological state of the user.

[1022] "Optimization" means adjusting something to the most effective state for a specific purpose.

[1023] This invention combines a system that allows a user to input instructions in natural language and automatically acquires and visualizes data based on the instructions with an emotion engine that recognizes the user's emotions. The system includes the following steps:

[1024] First, the user inputs a command in natural language into their device. For example, they might say, "Tell me the next interesting movie" or "Tell me a movie that will help me relax when I'm tired." This is the initial input stage performed by the user.

[1025] The device sends these instructions to the server, which then uses a natural language processing engine (e.g., spaCy) to analyze the user's instructions. At this stage, the server extracts the content of the instructions (e.g., the genre and characteristics of the movie being searched for) and simultaneously analyzes the user's emotions using an emotion engine (e.g., Sentiment Analysis API).

[1026] Based on the analysis results, the server retrieves the necessary data from the data warehouse, including movie information and rating data from the content database. For example, if a user requests "relaxing movies," the server will extract movies that meet that criteria.

[1027] The acquired data is pre-processed by the server, which includes data cleansing, filtering, and optionally aggregating the data. The server then optimizes the data based on the results of the emotion engine. If the user's emotion is relaxation, a list of movies that match that emotion is generated. The visual display format and colors are also adjusted based on the emotion.

[1028] Finally, the preprocessed and optimized data may be graphed using a graph drawing library (e.g., Matplotlib or Plotly). This visual graph or content list is sent to the user's device and displayed to the user.

[1029] Specific examples

[1030] For example, if a user types, "I've been feeling stressed lately, can you recommend some relaxing movies?", the system analyzes the instruction, extracts "relaxing movies" as a movie genre, and uses an emotion engine to recognize the user's stress level. It then retrieves information about relaxing movies from a data warehouse and generates an optimized list based on the user's emotions. Finally, the movie list, with its background color and display format adjusted, is displayed on the user's device.

[1031] Prompt Sentence Examples

[1032] "I've been feeling stressed lately, so can you recommend a relaxing movie?"

[1033] This invention allows users to easily obtain data visualization and content display that best fits their emotional state. The system takes into account the user's emotions and provides the most appropriate information at that time, greatly improving the user experience.

[1034] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1035] Step 1:

[1036] The user inputs instructions in natural language, such as "Tell me the next good movie." This is the initial input that occurs on the terminal. The input instructions are sent directly to the server.

[1037] Step 2:

[1038] The server receives instructions in natural language sent from the device. The server analyzes the instructions using a natural language processing engine. As a result of the analysis, it extracts information such as the search target and display format. For example, if the instruction is "Tell me the next good movie," the search target will be "good movies."

[1039] Step 3:

[1040] The server uses an emotion engine to recognize the user's emotions based on the analysis results. Emotions (e.g., positive or negative emotions) are extracted from the content of the instruction. For example, in the case of "Tell me a movie that helps me relax when I'm tired," "fatigue" is recognized as the emotion.

[1041] Step 4:

[1042] The server retrieves the necessary data (e.g., movie information) from the data warehouse based on the analysis results and the recognized emotions. The server then sends a query to the database to retrieve the corresponding movie list.

[1043] Step 5:

[1044] The server preprocesses the retrieved data. Preprocessing includes data cleansing, filtering, and, if necessary, data aggregation. Data cleansing involves filling in missing values ​​and removing duplicates, while filtering extracts only data that corresponds to the user's instructions. For example, if the search term is "relaxing movies," only movie information that falls into this category is filtered.

[1045] Step 6:

[1046] The server then uses the pre-processed data to optimize the data based on the results of the emotion engine, for example, visualizing positive emotions with bright colors and negative emotions with muted colors, as well as adjusting the order of the list display and the emphasis of information.

[1047] Step 7:

[1048] The server uses a graph drawing library (such as Matplotlib or Plotly) to generate graphs based on the optimized data. When the user requests a visualization, a graph in the appropriate format is drawn.

[1049] Step 8:

[1050] The server sends the generated graph or optimized content list to the user's device. The device visually displays the received data. The display format and color are optimized according to the user's emotions.

[1051] Step 9:

[1052] The user checks the results displayed on the device, such as a list of relaxing movies and graphs, which are expected to have a psychologically soothing effect.

[1053] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1054] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1055] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1056] [Fourth embodiment]

[1057] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1058] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1059] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1060] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1061] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1062] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1063] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1064] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1065] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1066] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1067] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1068] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1069] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1070] In this invention, the user simply issues a command in natural language to acquire and visualize data, and the system automatically processes the data and generates and displays a graph. Specific embodiments of the system are described below.

[1071] Overall overview

[1072] This system automates a series of processes: interpreting user instructions, retrieving the necessary data from a data warehouse, performing preprocessing, generating graphs, and finally displaying the results on the user's device.

[1073] User input

[1074] For example, a user inputs a command in natural language into their device, such as: "Show me this month's sales data in a bar graph." The user can freely specify the period, type, and display format of the data.

[1075] Natural Language Processing

[1076] The device sends the user's instructions to the server, which first analyzes them using a natural language processing engine (e.g., spaCy or NLTK). Through the analysis, information such as the data period (e.g., this month), data type (e.g., sales data), and graph format (e.g., bar graph) is extracted.

[1077] Data Acquisition

[1078] The server generates a query to the data warehouse (DWH) based on the analysis results. This query is to obtain the specific data the user is looking for (this month's sales data). As a result of the query, the required data is sent from the DWH to the server.

[1079] Data Preprocessing

[1080] The server pre-processes the data it receives, which includes data cleansing (removing duplicates and imputing missing values), filtering, and aggregation, making the data suitable for generating graphs in the specified format.

[1081] Graph Generation

[1082] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a bar graph) according to the user's instructions. To generate the graph, it uses a graph drawing library such as Matplotlib or Plotly.

[1083] Sending and displaying graphs

[1084] The generated graphs are exported in image formats (e.g. PNG or JPEG) or interactive formats (e.g. HTML or JavaScript) and sent from the server to the terminal, which then displays the received graph data so that the user can visually check it.

[1085] Specific examples

[1086] For example, if a user inputs "Display last month's sales data as a line graph," the system operates as follows: First, the user's device sends the instruction to the server, which analyzes it. As a result of the analysis, the information extracted is "last month," "sales data," and "line graph." Based on this information, the server sends a query to the DWH to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. The final graph is sent to the user's device and displayed.

[1087] In this way, this system automatically acquires and visualizes the necessary data based on the user's verbal instructions, making data analysis possible without requiring specialized knowledge.The user simply commands, "Display the data for XX in a XX graph," and can quickly obtain a visual display of the data.

[1088] The processing flow will be explained below.

[1089] Step 1:

[1090] The user inputs a command into their own device, such as "Display this month's sales data in a bar graph." The user inputs the specific data period, data type, graph format, etc. in natural language.

[1091] Step 2:

[1092] The terminal sends the instructions entered by the user to the server. The user's terminal sends the input data to the server as an HTTP request.

[1093] Step 3:

[1094] The server parses the received user instructions using a natural language processing engine (e.g., spaCy, NLTK), and extracts important information from the instructions, such as the time period of the data (e.g., this month), the type of data (e.g., sales data), and the graph format (e.g., bar graph).

[1095] Step 4:

[1096] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. Specifically, the server generates an SQL query and executes the query against the data warehouse.

[1097] Step 5:

[1098] The server receives the data retrieved from the data warehouse and performs pre-processing, which includes data cleansing (e.g., removing duplicate data and imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating the total sales by month).

[1099] Step 6:

[1100] The server generates a graph in the specified format (e.g., a bar graph) based on the preprocessed data, using a graph drawing library such as Matplotlib or Plotly.

[1101] Step 7:

[1102] The server converts the generated graphs into a suitable format for transmission to the user's device. Graphs can be exported into image formats (e.g. PNG, JPEG) or interactive formats (e.g. HTML, JavaScript).

[1103] Step 8:

[1104] The server sends the exported graph data to the user's terminal as an HTTP response.

[1105] Step 9:

[1106] The terminal displays the received graph data. The user's terminal renders the received data on the screen and displays it so that the user can visually confirm it.

[1107] This series of steps allows users to quickly retrieve and visualize data using simple natural language commands.

[1108] Example 1

[1109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1110] In today's information society, users need to quickly and visually understand the information they need from large amounts of data. However, conventional data analysis tools require specialized knowledge, making data analysis and visualization difficult and preventing rapid response. Furthermore, the time and effort required for data preprocessing and graph generation makes it difficult to achieve efficient data analysis.

[1111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1112] In this invention, the server includes means for receiving instructions input by a user in natural language, means for analyzing the received instructions using natural language processing, means for retrieving necessary information from a data storage device based on the analysis result, means for preprocessing the retrieved information, means for generating a visual display based on the preprocessed information, and means for transmitting the generated visual display to the user's device, thereby enabling the user to quickly analyze and visualize data through simple instructions in natural language without requiring specialized knowledge.

[1113] A "user" is a person or entity that operates the system and inputs instructions in natural language.

[1114] "Natural language" refers to language used by humans on a daily basis, and refers to common words and phrases that do not require specific technical skills or knowledge.

[1115] "Instructions" refer to requests or commands given by a user to a system, specifically input in natural language for the purpose of obtaining or visualizing data.

[1116] "Natural language processing" is a technology that allows computers to understand and analyze human language (natural language), and includes the process of extracting meaning from text data.

[1117] A "data storage device" is a computer system for storing and managing large amounts of data, and includes technologies such as data warehouses and databases.

[1118] "Information" refers to data retrieved from a data storage device based on a user's instruction, and includes data of a specific period and type.

[1119] "Preprocessing" refers to a series of operations that prepare acquired information in a form suitable for generating a visual display, including data cleansing and filtering.

[1120] "Visual displays" refer to data display formats such as graphs and charts that are created to make acquired information easier to understand visually.

[1121] "Visual representation rendering tools" refers to software libraries and platforms used to efficiently generate visual representations.

[1122] In this invention, a user can use natural language commands to acquire and visualize data, and the system automatically processes the data based on the commands and generates and displays graphs. Specific embodiments of the system are described below.

[1123] Overall overview

[1124] This system automates a series of processes: interpreting a user's natural language instructions, retrieving the necessary information from a data storage device, performing preprocessing, generating a graph as a visual display, and finally displaying it on the user's device.

[1125] Natural language input

[1126] The user inputs a command in natural language into their own terminal, such as "Display this month's sales data in a bar graph." The user can freely specify the period, type, display format, etc. of the data.

[1127] Natural Language Processing

[1128] The device sends the instructions entered by the user to the server, which then analyzes them using a natural language processing engine (e.g., spaCy or NLTK). The analysis extracts information such as the data period (e.g., this month), the type of data (e.g., sales data), and the graph format (e.g., bar graph).

[1129] Data Acquisition

[1130] The server generates a query to a data store (such as a data warehouse) based on the analysis results. The query is to obtain the specific information the user is looking for (e.g., this month's sales data). As a result of the query, the data store sends the required information to the server.

[1131] Data Preprocessing

[1132] The server pre-processes the information it receives, which includes data cleansing (removing duplicates and imputing missing values), filtering, and aggregation, making the information suitable for generating graphs in the specified format.

[1133] Graph Generation

[1134] Based on the preprocessed information, the server generates a visual representation in the specified format (e.g., a bar graph) according to the user's instructions, using a graph drawing library such as Matplotlib or Plotly.

[1135] Sending and displaying graphs

[1136] The generated graphs are exported in image format (e.g. PNG or JPEG files) or interactive format (e.g. HTML or JavaScript) and sent from the server to the user's device, where the received graph data is displayed so that the user can visually check it.

[1137] Specific examples

[1138] For example, consider the case where a user inputs "Display last month's sales data as a line graph." In this case, the user's device sends the instruction to the server, which analyzes the instruction using a natural language processing engine. As a result of the analysis, the information "last month," "sales data," and "line graph" is extracted, and the server sends a query to the data storage device based on this information to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. The final graph is sent to the user's device and displayed.

[1139] In this way, this system automatically obtains and visualizes the necessary information based on the user's natural language instructions, making data analysis possible without requiring specialized knowledge.The user simply commands, "Display the data for XX in a XX graph," and can quickly obtain a visual display of the data.

[1140] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1141] Step 1:

[1142] A user inputs a command in natural language into his / her terminal. For example, the user inputs "Show me this month's sales data in a bar graph." The input data is a natural language command in the form of a string.

[1143] Input: "Show me this month's sales data in a bar chart"

[1144] Output: Natural language instructions sent from the device to the server

[1145] Step 2:

[1146] The terminal sends the received natural language instruction to the server, where it sends the instruction as is without converting its format.

[1147] Input: User's natural language instructions

[1148] Output: Natural language instructions sent to the server

[1149] Step 3:

[1150] The server analyzes the received instructions using a natural language processing engine (e.g., spaCy or NLTK), extracting information such as the data period, type, and graph format.

[1151] Input: Natural language instructions

[1152] Output: Data period (e.g., this month), data type (e.g., sales data), graph format (e.g., bar graph)

[1153] Step 4:

[1154] Based on the analysis results, the server generates a query to a data storage device (such as a DWH). This query is written in a format such as SQL and is intended to obtain the required information.

[1155] Input: Analysis results (data period, type, graph format)

[1156] Output: Generated query (e.g. "SELECT FROM sales WHERE date >= '2023-10-01' AND date <= '2023-10-31'")

[1157] Step 5:

[1158] The server sends the generated query to the data storage device to retrieve the specified data (sales data for this month). The query results are returned from the data storage device.

[1159] Input: Generated query

[1160] Output: Retrieved data (e.g., this month's sales data)

[1161] Step 6:

[1162] The server pre-processes the acquired data, which includes data cleansing (removing duplicate data and filling in missing values), filtering, and aggregation.

[1163] Input: Retrieved data

[1164] Output: Preprocessed data

[1165] Step 7:

[1166] The server generates a visual display in the specified format (e.g., a bar graph) based on the preprocessed data, using a visual display drawing tool such as Matplotlib or Plotly.

[1167] Input: Preprocessed data

[1168] Output: Generated graph (e.g. bar graph in PNG format)

[1169] Step 8:

[1170] The server sends the generated graph to the user's terminal in the form of an image file (e.g. PNG file) or an interactive HTML file.

[1171] Input: Generated graph

[1172] Output: Graph data sent to the terminal

[1173] Step 9:

[1174] The terminal displays the received graph data, including image files and interactive HTML files, in an appropriate viewer or browser.

[1175] Input: Graph data sent from the server

[1176] Output: A graphical representation on the screen that can be viewed by the user

[1177] For example, if a user types "Show me last month's sales data as a line graph," the system will do the following:

[1178] Step 1-2: The user inputs a natural language instruction, which is sent from the terminal to the server.

[1179] Step 3: The server parses the instructions and extracts the sales data for the last month in the form of a line graph.

[1180] Step 4-5: The server queries the data store to retrieve last month's sales data.

[1181] Step 6: Preprocess the data retrieved by the server.

[1182] Step 7: Generate a line graph based on the preprocessed data.

[1183] Step 8-9: The generated graph is sent to the user's terminal and displayed.

[1184] In this way, the system can automatically acquire and visualize the necessary data based on the user's instructions.

[1185] (Application example 1)

[1186] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1187] Modern production management systems generate massive amounts of data, but visually understanding this data requires specialized knowledge, making it difficult to identify and address problems in real time. Furthermore, there is a lack of tools that allow on-site managers to intuitively understand the data and respond quickly. This leads to problems such as reduced production efficiency and delayed early detection of abnormalities. The present invention aims to solve these problems by providing a system that allows users to easily instruct data visualization in natural language and respond immediately.

[1188] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1189] In this invention, the server includes means for receiving instructions entered by a user in natural language, means for analyzing the received instructions using natural language processing, means for retrieving necessary data from a data warehouse based on the analysis results, means for preprocessing the retrieved data, means for generating a graph based on the preprocessed data, means for transmitting the generated graph to the user's terminal, means for analyzing the natural language instructions and extracting a specific data type and time period, means for retrieving production management system data based on the extracted specific information, means for generating a graph in a specified format (e.g., pie chart, bar graph) based on the production management system data, and means for displaying the generated graph on a smartphone display. This simplifies the visualization of data in the production management system, enabling on-site managers to intuitively understand the data and respond quickly.

[1190] "Natural language" refers to a language that humans use on a daily basis, including written and spoken languages.

[1191] "Natural language processing" is a technology that allows computers to understand, analyze, and generate human language, and includes technology for text and voice data.

[1192] A "data warehouse" is a database system that allows companies and organizations to centrally store large amounts of data and integrate and analyze data from various data sources.

[1193] "Preprocessing" is the process of converting raw data into a form suitable for analysis and visualization, and includes data cleansing, filtering, and aggregation.

[1194] A "graph" is a diagram that visually represents data and comes in various forms, such as bar graphs, line graphs, and pie charts.

[1195] A "smartphone" is a highly functional mobile phone that can connect to the Internet and use a variety of applications.

[1196] A "production management system" is a system for managing production activities in factories and production lines, and includes production planning, progress management, quality control, etc.

[1197] A "terminal" is a device that allows a user to input information and display results, and includes smartphones, tablets, and personal computers.

[1198] A "graph drawing library" is a software library for generating graphs in programs, and includes Matplotlib and Plotly.

[1199] A "server" is a computer system that processes and provides data, and provides services in response to requests from clients.

[1200] The present invention provides a system that allows a user to use natural language to instruct data visualization in a production management system, and a server automatically processes the data and generates graphs, displaying the results on the user's terminal.

[1201] Overall overview

[1202] This system automates a series of processes: interpreting user instructions, acquiring and preprocessing specified production management data, generating graphs, and finally displaying them on a smartphone screen.

[1203] User input

[1204] Users use their smartphones to input instructions in natural language, such as "Display today's operating status of production line 1 in a pie chart." Such instructions allow users to specify a specific data period, data type, and graph format.

[1205] Natural Language Processing

[1206] The user's device sends the received instructions to the server, which then analyzes them using a natural language processing engine (e.g., spaCy or NLTK). This analysis extracts information such as the data period (e.g., today), the data type (e.g., the operating status of production line 1), and the graph format (e.g., pie chart).

[1207] Data Acquisition

[1208] The server generates a query to the production management system (data warehouse) based on the analysis results and retrieves the necessary data. This query is intended to retrieve data including operation information for a specific production line.

[1209] Data Preprocessing

[1210] The server pre-processes the data it receives, which includes cleansing, filtering, and aggregating the data, converting it into an optimal format for generating graphs.

[1211] Graph Generation

[1212] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a pie chart) using a graph drawing library such as Matplotlib or Plotly.

[1213] Sending and displaying graphs

[1214] The generated graph is exported as an image file (e.g., PNG) and sent from the server to the user's smartphone, where it displays the received graph data for the user to visually check.

[1215] For example, if a user inputs "Display this month's downtime data for production line 2 as a bar graph," the system operates as follows: The user's device sends the instruction to the server, which analyzes it. As a result of the analysis, the following information is extracted: "This month," "Production line 2," "Downtime data," and "Bar graph." Based on this information, the server sends a query to the data warehouse to obtain "This month's downtime data for production line 2." Next, the server preprocesses the data and then generates a bar graph. The final graph is sent to the user's smartphone and displayed.

[1216] This system automatically acquires and visualizes the necessary data based on the user's natural language instructions, making it possible to analyze production data without requiring specialized knowledge. Users can quickly obtain a visual display of data by simply instructing, "Display the data for XX in a XX graph."

[1217] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1218] Step 1:

[1219] A user uses a smartphone to input instructions in natural language, such as "Display today's operating status of production line 1 in a pie chart." The input instruction is saved in an input field on the smartphone and is ready to be sent to the server.

[1220] Step 2:

[1221] The device sends the user's instructions to the server. The smartphone sends the input text data to the server using an HTTP request or WebSocket. The server receives the received natural language instructions and prepares them for analysis.

[1222] Step 3:

[1223] The server uses a natural language processing engine (e.g., spaCy or NLTK) to analyze the received instructions. Specifically, it tokenizes the text data, identifies nouns and verbs, and extracts the time period (e.g., today), data type (e.g., the operation status of production line 1), and graph format (e.g., pie chart) based on the context. The extracted information is used in the next step.

[1224] Step 4:

[1225] The server generates a specific query for the production management system (data warehouse) based on the analysis results. The generated query is sent as an SQL query or API request to obtain today's data on the operating status of production line 1. The production management system returns data that matches the specified conditions.

[1226] Step 5:

[1227] The server receives the acquired data and performs preprocessing, which includes data cleansing (e.g., filling in missing values ​​and removing duplicates), filtering (e.g., excluding unnecessary data), and aggregation (e.g., aggregating operating status by time). The preprocessed data is then converted into a format suitable for graph generation.

[1228] Step 6:

[1229] The server generates a graph in the specified format (e.g., pie chart) based on the preprocessed data. Graph drawing libraries such as Matplotlib and Plotly are used to generate the graph. Specifically, data points are mapped to each part of the graph and visual elements (color, label, etc.) are added. The generated graph is exported as an image format (e.g., PNG).

[1230] Step 7:

[1231] The server sends the generated graph to the user's smartphone. The image file is sent as an HTTP response or WebSocket message. The smartphone displays the received image file in the specified display area.

[1232] Step 8:

[1233] Users can visually check the generated graphs on their smartphone screen, which allows them to intuitively understand the operating status of the specified production line and take appropriate action.

[1234] This series of steps enables users to easily use natural language to instruct data visualization and grasp the status of production management in real time.

[1235] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1236] This invention combines a system that automatically acquires and visualizes data based on instructions entered by a user in natural language with an emotion engine that recognizes the user's emotions. An embodiment of this invention will be described in detail below.

[1237] Overall overview

[1238] The system allows users to input instructions in natural language, retrieves the necessary data based on those instructions, and generates and displays graphs. It can also recognize the emotions contained in the user's instructions and optimize the color and format of the displayed graphs based on those emotions.

[1239] User input

[1240] For example, a user inputs a command in natural language into their device, such as: "Show me this month's sales data in a bar graph." The user can freely specify the period, type, and display format of the data.

[1241] Natural Language Processing and Emotion Recognition

[1242] The device sends the user's instructions to the server, which first analyzes them using a natural language processing engine (e.g., spaCy or NLTK) and an emotion engine (e.g., Sentiment Analysis API). The natural language processing extracts information such as the data period (e.g., this month), data type (e.g., sales data), and graph format (e.g., bar graph), and the emotion engine recognizes the emotion (e.g., joy, sadness, anger) contained in the user's instructions.

[1243] Data Acquisition and Emotional Response

[1244] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. The server also takes into account the results of the emotion engine and optimizes the data display format and graph colors according to the user's emotions. For example, if the user is expressing positive emotions, a bright color graph will be selected.

[1245] Data Preprocessing

[1246] The server pre-processes the retrieved data. Pre-processing includes data cleansing (e.g., removing duplicates, imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating monthly sales totals). At this stage, user sentiment also influences data processing. For example, dissatisfied users may be provided with more detailed data to help find the root cause of the problem.

[1247] Graph Generation

[1248] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a bar graph). Graph drawing libraries such as Matplotlib and Plotly are used to generate the graph. Again, the color and format of the graph are adjusted based on the results of the emotion engine.

[1249] Sending and displaying graphs

[1250] The generated graphs are exported in image formats (such as PNG or JPEG) or interactive formats (such as HTML or JavaScript) and sent from the server to the terminal, where the received graph data is displayed so that the user can visually check it.

[1251] Specific examples

[1252] For example, if a user inputs, "Display last month's sales data as a line graph, and do not use bright colors because sales were low," the system operates as follows: First, the user's device sends the instruction to the server, which analyzes it. As a result of the analysis, information such as "last month," "sales data," "line graph," and "negative sentiment" is extracted. Based on this information, the server sends a query to the DWH to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. Based on the results of the sentiment engine, a graph with darker colors is selected. The final graph is sent to the user's device and displayed.

[1253] In this way, the system can quickly and effectively acquire and visualize data based on the user's verbal instructions and emotions. Users do not need advanced expertise; they can simply input an instruction, including emotional input, such as "Display the data for XX in a XX graph," to obtain an optimized visual data display.

[1254] The processing flow will be explained below.

[1255] Step 1:

[1256] The user inputs a command into their device, such as "Display this month's sales data as a bar graph." The user can also input specific commands, including emotions (e.g., "I'm disappointed with the low sales figures, so I want to use dark colors").

[1257] Step 2:

[1258] The terminal sends the instructions entered by the user to the server. The user's terminal sends the input data to the server as an HTTP request.

[1259] Step 3:

[1260] The server parses the received user instructions using a natural language processing engine (e.g., spaCy, NLTK), and extracts important information from the instructions, such as the time period of the data (e.g., this month), the type of data (e.g., sales data), and the graph format (e.g., bar graph).

[1261] Step 4:

[1262] The server uses an emotion engine (e.g., Sentiment Analysis API) to recognize emotions from the analysis results of the natural language processing engine. For example, the emotion engine extracts negative emotions such as "disappointed" or "depressed."

[1263] Step 5:

[1264] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. Specifically, the server generates an SQL query and executes the query against the data warehouse.

[1265] Step 6:

[1266] The server receives the data retrieved from the data warehouse and performs pre-processing, which includes data cleansing (e.g., removing duplicate data and imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating the total sales by month).

[1267] Step 7:

[1268] The server generates graphs in the specified format (e.g., bar graphs) based on the preprocessed data and in accordance with the user's instructions. Graph drawing libraries such as Matplotlib and Plotly are used to generate graphs. The results of the emotion engine are reflected here, and if the user's instructions indicate negative emotions, the graph's color scheme is set to a darker shade.

[1269] Step 8:

[1270] The server converts the generated graphs into a suitable format for transmission to the user's device. Graphs can be exported into image formats (e.g. PNG, JPEG) or interactive formats (e.g. HTML, JavaScript).

[1271] Step 9:

[1272] The server sends the exported graph data to the user's terminal as an HTTP response.

[1273] Step 10:

[1274] The terminal displays the received graph data. The user's terminal renders the received data on the screen and displays it so that the user can visually confirm it.

[1275] Through this series of steps, users simply communicate simple natural language instructions and emotions to the system, which then automatically retrieves the data and visualizes it in an appropriate format.

[1276] Example 2

[1277] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1278] Conventional data visualization systems display data without considering the user's emotions, which means they are unable to provide optimal visualizations that reflect the user's intentions and emotions. While systems exist that can respond to natural language instructions, there is a lack of technology that can recognize emotions based on those instructions and reflect them in data visualizations. Therefore, there is a need for a system that allows users to input instructions in natural language, recognizes emotions based on those instructions, and provides appropriate data visualization.

[1279] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1280] In this invention, the server includes means for receiving instructions input by a user in natural language, means for analyzing the received instructions using natural language processing, means for recognizing the user's emotions from the analyzed instructions, means for acquiring necessary data from an information set based on the analysis results, means for preprocessing the acquired data, means for generating a graph by optimizing the format and color of the graph based on the preprocessed data in accordance with the user's emotions, and means for transmitting the generated graph to the user's terminal, thereby enabling optimal data visualization in accordance with the user's emotions.

[1281] "User" refers to a person who inputs instructions to the system in natural language.

[1282] "Natural language" refers to the words and written forms that humans use on a daily basis.

[1283] "Instructions" refer to requests made by the user to the system regarding data acquisition and display format.

[1284] "Server" refers to a central processing unit that receives user instructions, processes them, and returns the results to the user.

[1285] "Terminal" refers to an electronic device that allows a user to input instructions.

[1286] "Natural language processing" refers to the technology that allows computers to understand and analyze human language.

[1287] "Emotion recognition" refers to technology that analyzes and identifies the emotions contained in a user's instructions.

[1288] An "information collection" refers to a place that stores large amounts of data, such as a data warehouse or database.

[1289] "Preprocessing" refers to the process of organizing and shaping acquired data and converting it into a format suitable for analysis and visualization.

[1290] A "graph" refers to a diagram or chart that visually represents data.

[1291] "Graph optimization" refers to adjusting the format, color, and display method of a graph based on user sentiment.

[1292] "Generation" refers to the act of creating a graph in a specified format based on preprocessed data.

[1293] "Transmission" refers to the act of sending the graph generated by the server to the terminal and displaying it to the user.

[1294] "Visualization" refers to the presentation of information or data to a user in a graphical format.

[1295] This invention combines a system that automatically acquires and visualizes data based on instructions entered by a user in natural language with an emotion engine that recognizes the user's emotions. An embodiment of this invention will be described in detail below.

[1296] Overall overview

[1297] The system allows users to input instructions in natural language, retrieves the necessary data based on those instructions, and generates and displays graphs. It can also recognize the emotions contained in the user's instructions and optimize the color and format of the displayed graphs based on those emotions.

[1298] Hardware and software used

[1299] Server: A central processing unit that receives and processes user instructions.

[1300] Terminal: An electronic device (e.g., computer, smartphone) that allows a user to input instructions and display results.

[1301] Natural language processing engine: Software for parsing a user's natural language instructions, such as spaCy or NLTK.

[1302] Emotion recognition engine: Software for analyzing user emotions, such as the Sentiment Analysis API.

[1303] Data Warehouse (DWH): A database system for storing data and retrieving required data using queries.

[1304] Graph drawing libraries: Software for visualizing data in the form of graphs, such as Matplotlib or Plotly.

[1305] User input

[1306] For example, a user inputs a command in natural language into their device, such as: "Show me this month's sales data in a bar graph." The user can freely specify the period, type, and display format of the data.

[1307] Natural Language Processing and Emotion Recognition

[1308] The device sends the user's instructions to the server, which first analyzes them using a natural language processing engine (e.g., spaCy or NLTK) and an emotion engine (e.g., Sentiment Analysis API). The natural language processing extracts information such as the data period (e.g., this month), data type (e.g., sales data), and graph format (e.g., bar graph), and the emotion engine recognizes the emotion (e.g., joy, sadness, anger) contained in the user's instructions.

[1309] Data Acquisition and Emotional Response

[1310] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. The server also takes into account the results of the emotion engine and optimizes the data display format and graph colors according to the user's emotions. For example, if the user is expressing positive emotions, a bright color graph will be selected.

[1311] Data Preprocessing

[1312] The server pre-processes the retrieved data. Pre-processing includes data cleansing (e.g., removing duplicates, imputing missing values), filtering (e.g., filtering data for a specified period), and aggregation (e.g., calculating monthly sales totals). At this stage, user sentiment also influences data processing. For example, dissatisfied users may be provided with more detailed data to help find the root cause of the problem.

[1313] Graph Generation

[1314] After the preprocessing is complete, the server generates a graph in the specified format (e.g., a bar graph) using a graph drawing library such as Matplotlib or Plotly. Again, the color and format of the graph are adjusted based on the results of the emotion engine.

[1315] Sending and displaying graphs

[1316] The generated graphs are exported in image formats (such as PNG or JPEG) or interactive formats (such as HTML or JavaScript) and sent from the server to the terminal, where the received graph data is displayed so that the user can visually check it.

[1317] Specific examples

[1318] For example, if a user inputs, "Display last month's sales data as a line graph, and do not use bright colors because sales were low," the system operates as follows: First, the user's device sends the instruction to the server, which analyzes it. As a result of the analysis, information such as "last month," "sales data," "line graph," and "negative sentiment" is extracted. Based on this information, the server sends a query to the DWH to obtain "last month's sales data." Next, the server preprocesses the data and then generates a line graph. Based on the results of the sentiment engine, a graph with darker colors is selected. The final graph is sent to the user's device and displayed.

[1319] In this way, the system can quickly and effectively acquire and visualize data based on the user's verbal instructions and emotions. Users do not need advanced expertise; they can simply input an instruction, including emotional input, such as "Display the data for XX in a XX graph," to obtain an optimized visual data display.

[1320] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1321] Step 1: User input

[1322] The user inputs a command in natural language using a terminal. For example, the command might say, "Display this month's sales data in a bar graph." This command includes the data period ("this month"), type ("sales data"), and display format ("bar graph").

[1323] Input: User instructions in natural language

[1324] Output: Typed instructions retained on the terminal

[1325] Step 2: Send instructions to the server

[1326] The terminal transmits the input instructions to the server. This data transmission is carried out via the Internet.

[1327] Input: User's natural language instructions

[1328] Output: Instruction data sent to the server

[1329] Step 3: Parsing instructions (natural language processing and emotion recognition)

[1330] The server analyzes the received instructions using a natural language processing engine (e.g., spaCy). This analysis extracts information such as the data period, type, and display format. At the same time, it analyzes the user's emotions using an emotion recognition engine (e.g., Sentiment Analysis API).

[1331] Input: Instruction data sent to the server

[1332] Output: Analysis result data period, type, display format, emotional information

[1333] Step 4: Obtaining the necessary data

[1334] The server retrieves the necessary data from the data warehouse (DWH) based on the analysis results. The server executes SQL queries according to the period and type and retrieves the relevant data from the database.

[1335] Input: Analysis results

[1336] Output: Required data obtained from DWH

[1337] Step 5: Preprocessing the data

[1338] The server preprocesses the acquired data, including removing duplicate data, imputing missing values, filtering periods, and aggregating data. It also adjusts the level of detail of the data according to the user's sentiment.

[1339] Input: Required data

[1340] Output: Preprocessed data

[1341] Step 6: Generate the graph

[1342] The server generates graphs in the specified format (e.g., bar graphs) based on the preprocessed data. It uses Matplotlib and Plotly to generate graphs, optimizing the color and format of the graphs based on the user's preferences.

[1343] Input: Preprocessed data, emotion information

[1344] Output: The generated graph

[1345] Step 7: Send and display the graph

[1346] The server sends the generated graph to the terminal, where it can be exported in image format (PNG or JPEG) or interactive format (HTML or JavaScript) and displayed on the terminal for the user to view.

[1347] Input: Generated graph

[1348] Output: Graph displayed on the user's terminal

[1349] Specific examples

[1350] For example, if a user inputs "Display this month's sales data in a bar graph," the system operates as follows: First, the user's device sends the instruction to the server. The server analyzes the instruction and extracts information such as "this month," "sales data," and "bar graph." It then retrieves "this month's sales data" from the DWH based on the analysis results. The server preprocesses the data and generates a bar graph based on the preprocessed data. The final graph is sent to the user's device and displayed. This allows users to easily obtain visualized data based on instructions in natural language.

[1351] (Application example 2)

[1352] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1353] Conventional data visualization systems analyze users' natural language instructions to visualize data, but they do not optimize the system to take into account the user's emotions, resulting in a lack of improvement in the user experience. Furthermore, particularly in the entertainment industry, users often seek information and content that reflects their current emotions, creating a need for appropriate data and content that takes emotions into account.

[1354] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving instructions entered by a user in natural language, means for analyzing the received instructions using natural language processing, means for retrieving necessary data from a data warehouse based on the analysis result, means for preprocessing the retrieved data, means for generating a graph based on the preprocessed data, means for transmitting the generated graph to the user's terminal, means for recognizing the user's emotion based on the analyzed instructions, and means for optimizing the format and color of displayed data based on the recognized emotion. This enables optimal data visualization and content display according to the user's emotion.

[1355] A "user" is a person who uses the system to input instructions in natural language.

[1356] A "natural language" is a language that humans use on a daily basis, and is not a specific programming language.

[1357] An "instruction" represents a request or command given by a user to a system.

[1358] "Analysis" is the process of mechanically breaking down instructions entered in natural language and understanding their meaning.

[1359] A "data warehouse" is a system for centrally storing and managing large amounts of data.

[1360] "Preprocessing" is the process of processing acquired data through cleansing, filtering, and other processes to prepare it in a format suitable for analysis and visualization.

[1361] A "graph" is a diagram that visually represents numerical data.

[1362] A "terminal" is a device through which a user accesses and operates the system.

[1363] "Emotion" refers to the psychological state of the user.

[1364] "Optimization" means adjusting something to the most effective state for a specific purpose.

[1365] This invention combines a system that allows a user to input instructions in natural language and automatically acquires and visualizes data based on the instructions with an emotion engine that recognizes the user's emotions. The system includes the following steps:

[1366] First, the user inputs a command in natural language into their device. For example, they might say, "Tell me the next interesting movie" or "Tell me a movie that will help me relax when I'm tired." This is the initial input stage performed by the user.

[1367] The device sends these instructions to the server, which then uses a natural language processing engine (e.g., spaCy) to analyze the user's instructions. At this stage, the server extracts the content of the instructions (e.g., the genre and characteristics of the movie being searched for) and simultaneously analyzes the user's emotions using an emotion engine (e.g., Sentiment Analysis API).

[1368] Based on the analysis results, the server retrieves the necessary data from the data warehouse, including movie information and rating data from the content database. For example, if a user requests "relaxing movies," the server will extract movies that meet that criteria.

[1369] The acquired data is pre-processed by the server, which includes data cleansing, filtering, and optionally aggregating the data. The server then optimizes the data based on the results of the emotion engine. If the user's emotion is relaxation, a list of movies that match that emotion is generated. The visual display format and colors are also adjusted based on the emotion.

[1370] Finally, the preprocessed and optimized data may be graphed using a graph drawing library (e.g., Matplotlib or Plotly). This visual graph or content list is sent to the user's device and displayed to the user.

[1371] Specific examples

[1372] For example, if a user types, "I've been feeling stressed lately, can you recommend some relaxing movies?", the system analyzes the instruction, extracts "relaxing movies" as a movie genre, and uses an emotion engine to recognize the user's stress level. It then retrieves information about relaxing movies from a data warehouse and generates an optimized list based on the user's emotions. Finally, the movie list, with its background color and display format adjusted, is displayed on the user's device.

[1373] Prompt Sentence Examples

[1374] "I've been feeling stressed lately, so can you recommend a relaxing movie?"

[1375] This invention allows users to easily obtain data visualization and content display that best fits their emotional state. The system takes into account the user's emotions and provides the most appropriate information at that time, greatly improving the user experience.

[1376] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1377] Step 1:

[1378] The user inputs instructions in natural language, such as "Tell me the next good movie." This is the initial input that occurs on the terminal. The input instructions are sent directly to the server.

[1379] Step 2:

[1380] The server receives instructions in natural language sent from the device. The server analyzes the instructions using a natural language processing engine. As a result of the analysis, it extracts information such as the search target and display format. For example, if the instruction is "Tell me the next good movie," the search target will be "good movies."

[1381] Step 3:

[1382] The server uses an emotion engine to recognize the user's emotions based on the analysis results. Emotions (e.g., positive or negative emotions) are extracted from the content of the instruction. For example, in the case of "Tell me a movie that helps me relax when I'm tired," "fatigue" is recognized as the emotion.

[1383] Step 4:

[1384] The server retrieves the necessary data (e.g., movie information) from the data warehouse based on the analysis results and the recognized emotions. The server then sends a query to the database to retrieve the corresponding movie list.

[1385] Step 5:

[1386] The server preprocesses the retrieved data. Preprocessing includes data cleansing, filtering, and, if necessary, data aggregation. Data cleansing involves filling in missing values ​​and removing duplicates, while filtering extracts only data that corresponds to the user's instructions. For example, if the search term is "relaxing movies," only movie information that falls into this category is filtered.

[1387] Step 6:

[1388] The server then uses the pre-processed data to optimize the data based on the results of the emotion engine, for example, visualizing positive emotions with bright colors and negative emotions with muted colors, as well as adjusting the order of the list display and the emphasis of information.

[1389] Step 7:

[1390] The server uses a graph drawing library (such as Matplotlib or Plotly) to generate graphs based on the optimized data. When the user requests a visualization, a graph in the appropriate format is drawn.

[1391] Step 8:

[1392] The server sends the generated graph or optimized content list to the user's device. The device visually displays the received data. The display format and color are optimized according to the user's emotions.

[1393] Step 9:

[1394] The user checks the results displayed on the device, such as a list of relaxing movies and graphs, which are expected to have a psychologically soothing effect.

[1395] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1396] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1397] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1398] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1399] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1400] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1401] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1402] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1403] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1404] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1405] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1406] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1407] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1408] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1409] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1410] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1411] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1412] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1413] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1414] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1415] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1416] The following is further disclosed regarding the above embodiment.

[1417] (Claim 1)

[1418] means for receiving instructions input by a user in natural language;

[1419] means for analyzing the received instructions using natural language processing;

[1420] A means for obtaining necessary data from the data warehouse based on the analysis results;

[1421] means for pre-processing the acquired data;

[1422] means for generating graphs based on the preprocessed data;

[1423] means for transmitting the generated graph to a user's terminal;

[1424] A system including:

[1425] (Claim 2)

[1426] 10. The system of claim 1, further comprising analyzing the time period of the data, the type of data, and the format of the graph.

[1427] (Claim 3)

[1428] 10. The system of claim 1, wherein the graph is generated using a graph drawing library.

[1429] "Example 1"

[1430] (Claim 1)

[1431] means for receiving instructions input by a user in natural language;

[1432] means for analyzing the received instructions using natural language processing;

[1433] means for acquiring necessary information from a data storage device based on the analysis results;

[1434] means for preprocessing the acquired information;

[1435] means for generating a visual display based on the preprocessed information;

[1436] means for transmitting the generated visual representation to a user device;

[1437] A system including:

[1438] (Claim 2)

[1439] 10. The system of claim 1, wherein the system analyzes the duration of the information, the type of information, and the format of the visual representation.

[1440] (Claim 3)

[1441] 10. The system of claim 1, wherein the visual representation is generated using a visual representation drawing tool.

[1442] "Application Example 1"

[1443] (Claim 1)

[1444] means for receiving instructions input by a user in natural language;

[1445] means for analyzing the received instructions using natural language processing;

[1446] A means for obtaining necessary data from the data warehouse based on the analysis results;

[1447] means for pre-processing the acquired data;

[1448] means for generating graphs based on the preprocessed data;

[1449] means for transmitting the generated graph to a user's terminal;

[1450] means for parsing the natural language instructions to extract specific data types and time periods;

[1451] A means for acquiring data of a production control system based on the extracted specific information;

[1452] A means for generating a graph in a specified format (pie chart, bar graph, etc.) based on data from the production management system;

[1453] A means to display the generated graph on a smartphone display

[1454] A system including:

[1455] (Claim 2)

[1456] 10. The system of claim 1, further comprising analyzing the time period of the data, the type of data, and the format of the graph.

[1457] (Claim 3)

[1458] 10. The system of claim 1, wherein the graph is generated using a graph drawing library.

[1459] "Example 2: Combining Emotion Engines"

[1460] (Claim 1)

[1461] means for receiving instructions input by a user in natural language;

[1462] means for analyzing the received instructions using natural language processing;

[1463] means for recognizing a user's emotion from the analyzed instructions;

[1464] A means for acquiring necessary data from the information set based on the analysis results;

[1465] means for pre-processing the acquired data;

[1466] A means for generating a graph by optimizing the format and color of the graph according to the user's feelings based on the preprocessed data;

[1467] means for transmitting the generated graph to a user's terminal;

[1468] A system including:

[1469] (Claim 2)

[1470] The system of claim 1, wherein the system analyzes the period of data, the type of data, and the format of the graph to recognize the user's emotions.

[1471] (Claim 3)

[1472] The system of claim 1, wherein a graph is generated based on the user's emotions using a graph drawing library.

[1473] "Application example 2 when combining emotion engines"

[1474] (Claim 1)

[1475] means for receiving instructions input by a user in natural language;

[1476] means for analyzing the received instructions using natural language processing;

[1477] A means for obtaining necessary data from the data warehouse based on the analysis results;

[1478] means for pre-processing the acquired data;

[1479] means for generating graphs based on the preprocessed data;

[1480] means for transmitting the generated graph to a user's terminal;

[1481] means for recognizing the user's emotion based on the analyzed instructions;

[1482] A means for optimizing the format and color of displayed data based on the recognized emotion; and

[1483] A system including:

[1484] (Claim 2)

[1485] 10. The system of claim 1, further comprising analyzing the time period of the data, the type of data, and the format of the graph.

[1486] (Claim 3)

[1487] 10. The system of claim 1, wherein the graph is generated using a graph drawing library. [Explanation of symbols]

[1488] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving instructions input by a user in natural language; means for analyzing the received instructions using natural language processing; A means for obtaining necessary data from the data warehouse based on the analysis results; means for pre-processing the acquired data; means for generating graphs based on the preprocessed data; means for transmitting the generated graph to a user's terminal; A system including:

2. 10. The system of claim 1, further comprising analyzing the time period of the data, the type of data, and the format of the graph.

3. The system of claim 1 , wherein the system generates the graph using a graph drawing library.

Images