System

Abstract

A system is provided.SOLUTION: The system includes a sentence input means, a voice input means, a transmission means for processing inputted data and transmitting the processed data to a server, an analysis means for analyzing the data and determining an appropriate illustration format by the server, a means for acquiring necessary additional information by the server, a means for transmitting generated illustration data to a terminal, and a display means for displaying the illustration data received by the terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditionally, converting text or audio descriptions into diagrams required specialized tools and time, making it difficult to quickly create diagrams, especially during meetings or brainstorming sessions. This made it difficult to quickly visualize complex information, reducing communication efficiency. Furthermore, because there was no way to select the appropriate diagram format or obtain the necessary additional information, the accuracy and usefulness of the generated diagrams was low. [Means for solving the problem]

[0005] To solve these problems, the present invention provides the following means: A text input means and a voice input means are provided, allowing a user to input text or voice; a transmission means is provided for processing the input data and transmitting it to a server, and the server is provided with an analysis means for analyzing the data using natural language processing technology and determining an appropriate diagram format; the server is provided with a means for obtaining necessary additional information, and includes a means for transmitting the generated diagram data to a terminal; this system also includes a means for selecting the diagram format from Gantt charts, flowcharts, graphs, and tables, and a means for querying the user for additional information and obtaining it; this makes it possible to convert information input by the user into an appropriate diagram in real time and present it in a visually easy-to-understand form, significantly improving the efficiency of communication.

[0006] "Text input means" refers to a device or software that allows a user to input text.

[0007] "Voice input means" refers to a device or software that allows a user to input voice.

[0008] "Transmission means" refers to a device or software that has the function of processing input data and transmitting it to the server.

[0009] "Analysis means" refers to a device or software that has the function of allowing the server to analyze data using natural language processing technology and determine an appropriate graphical format.

[0010] The "means for obtaining additional information" refers to a device or software that has the function of allowing the server to query the user for the necessary additional information and obtain it.

[0011] "Means for transmitting diagram data" refers to a device or software that has the function of transmitting the generated diagram data to a terminal.

[0012] "Display means" refers to a device or software that has the function of displaying to the user the graphical data that the terminal receives from the server. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention relates to a system that converts information input by text or voice into a diagram format and displays it. This system provides a function in which a user inputs information using text or voice, analyzes the information, and generates and displays an appropriate diagram. Specific embodiments of the present invention are described below.

[0035] 1. Text and voice input methods

[0036] Users input information into the device either by text or voice. Text input is done using a keyboard or touchscreen, while voice input is done using a microphone or voice recognition software. For example, a user can type, "I need a Gantt chart showing the progress of a project."

[0037] 2. Transmission Method

[0038] The device preprocesses the input data and sends it to the server. In the case of voice input, the device uses voice recognition technology to convert the voice data into text data, which is then formatted into a standard format and sent to the server.

[0039] 3. Analysis method

[0040] The server uses natural language processing technology to analyze the text data sent. The analysis engine extracts key phrases and keywords and identifies themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted, and based on this, the text is determined to be information about project management.

[0041] 4. Means of obtaining additional information

[0042] To generate a Gantt chart, the server needs detailed information such as specific tasks, deadlines, and progress. If this additional information is missing, the server queries the user to obtain the necessary data. For example, the server might send a message to the user requesting the "project start date," "task details," and "deadline for each task."

[0043] 5. Illustration Generation Method

[0044] After retrieving the necessary data, the server uses a diagram generation engine to generate the appropriate diagram, which supports formats such as flowcharts, Gantt charts, graphs, tables, etc. For example, a Gantt chart showing each task in a project and its duration can be generated.

[0045] 6. Illustrated Data Transmission Methods

[0046] The generated diagram is sent from the server to the terminal. The diagram data is sent in a format that the user can view and edit (e.g., image data, Excel file, Google (registered trademark) Sheets URL, etc.).

[0047] 7. Display means

[0048] The device receives the diagram data from the server and displays it to the user. The user can visually check the diagram on the device screen and edit it as needed. For example, a Gantt chart showing the progress of a project can be displayed on the screen, allowing the user to check the progress and deadlines of each task at a glance.

[0049] In this way, the system of the present invention can analyze the information entered by the user in real time and convert it into an appropriate diagram format for display, thereby enabling complex information to be visualized concisely and greatly improving communication efficiency.

[0050] The processing flow will be explained below.

[0051] Step 1:

[0052] The user inputs text or voice into the device. For example, the user inputs, "I need a Gantt chart showing the progress of a project."

[0053] Step 2:

[0054] The device receives input or speech. In the case of speech input, the device uses speech recognition software to convert the speech data into text.

[0055] Step 3:

[0056] The terminal preprocesses the text data and formats it into a standard format, then sends the formatted data to the server.

[0057] Step 4:

[0058] The server analyzes the received text data. Natural language processing techniques are used to extract key phrases and keywords and identify themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[0059] Step 5:

[0060] The server then determines the appropriate diagram format based on the analysis results. In this example, a Gantt chart is selected as the appropriate diagram format.

[0061] Step 6:

[0062] The server checks the details needed to generate the Gantt chart. If the required information is missing, the server prompts the user for additional information. For example, "What is the start date of the project?"

[0063] Step 7:

[0064] The user answers with additional information. For example, the user answers, "The project started on October 1, 2023."

[0065] Step 8:

[0066] The server receives additional information and adds it to the data used to generate the Gantt chart, such as project tasks, deadlines, and progress.

[0067] Step 9:

[0068] The server generates a Gantt chart using a diagram generation engine, and the generated Gantt chart is converted into image data or an editable format within the server.

[0069] Step 10:

[0070] The server transmits the generated diagram data to the terminal in a format that the user can view and edit.

[0071] Step 11:

[0072] The terminal displays the received graphical data to the user, for example, a Gantt chart is displayed on the terminal screen, allowing the user to visually check the progress of the project.

[0073] Step 12:

[0074] Users can check the displayed diagrams and edit them as necessary, which allows users to communicate more efficiently.

[0075] Example 1

[0076] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0077] Conventional information input systems have difficulty properly analyzing information entered via text or voice and converting it into a visually easy-to-understand diagram format. In particular, when the user's input information is insufficient, there is a lack of a means to effectively acquire additional information and generate a diagram, which reduces user convenience. Furthermore, the inability to update or edit diagram data in real time makes it difficult to make decisions based on the latest information.

[0078] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0079] In this invention, the server includes a means for converting input data into text using speech recognition technology, a means for analyzing the data using natural language processing technology, and a means for updating the generated illustrations in real time. This allows the information input by the user via text or voice to be properly analyzed, and missing information to be effectively acquired and an illustration to be generated. Furthermore, editing and real-time updating of the generated illustration data allows for efficient decision-making based on the latest information.

[0080] A "text input means" is a device and method that allows a user to input text information using a keyboard or touch screen.

[0081] "Voice input means" refers to a device and method including a microphone and voice recognition software for inputting user voice information and processing it as digital data.

[0082] The "transmitting means" refers to a device and method including a network interface and communication protocol for preprocessing input data and transmitting it to a server.

[0083] "Analysis means" refers to software and algorithms that the server uses to analyze the received data using natural language processing techniques and determine an appropriate graphical format.

[0084] The "acquisition means" refers to a device and method for acquiring additional information required by the server to generate the illustration from the user through a query.

[0085] "Illustration generation means" refers to software and algorithms for generating appropriate illustrations (Gantt charts, flow charts, graphs, tables) based on collected data.

[0086] The "transmitting means" refers to an apparatus and method including a network interface and a communication protocol for the server to transmit the generated illustration data to the terminal.

[0087] "Display means" refers to a display and software for visually displaying the graphical data received by the terminal.

[0088] The "query means" refers to a device and method by which the server queries the user for necessary information when additional information is missing.

[0089] "Speech recognition technology" refers to software and algorithms used to convert voice data into text data.

[0090] "Natural language processing technology" refers to software and algorithms that analyze text data, extract key phrases and keywords, and perform semantic analysis.

[0091] "Real-time update means" refers to a device and method for instantly updating generated graphical data based on the latest information.

[0092] The "editing means" refers to software and an interface that allows the user to edit the received graphic data on the terminal.

[0093] The system of the present invention is a system that can convert information input by a user in text or voice into a diagram format and visually display it. To implement this system, the following hardware and software are used.

[0094] Hardware and Software

[0095] Enter information

[0096] Users can input text using a keyboard or touchscreen, such as "I need a Gantt chart showing the progress of a project." A microphone is also used for voice input, which is then converted into text using speech recognition software, such as the Google Speech-to-Text API.

[0097] Data transmission

[0098] The terminal formats the input text data into a standard format and sends it to the server, for example, in JSON format, using an HTTP POST request.

[0099] Data analysis

[0100] The server analyzes the received text data using natural language processing (NLP) techniques. Analysis engines such as SpaCy and the BERT model are used. This allows the server to extract key phrases and keywords and identify themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[0101] Obtaining additional information

[0102] The server queries the user to obtain additional information needed to generate the diagram. This information is obtained via HTTP requests or message transmission. The user enters the requested information into the terminal and resubmits it. For example, the user enters specific details such as "The project start date is October 1, 2023, Task 1 will be planning until October 5, and Task 2 will be design until October 15."

[0103] Illustration generation

[0104] After the server collects all the necessary data, it uses a diagram generation engine, such as D3.js or Syncfusion, to generate diagrams, such as Gantt charts, that show each project task and its duration.

[0105] Sending diagram data

[0106] The generated diagram data is sent from the server to the terminal via a network interface or communication protocol. The diagram data is often sent in image format or an editable format.

[0107] Data Display

[0108] The device then displays the received diagram data to the user. A display and corresponding software are used as the display method. The user can check the diagram on the device screen and edit it as needed. For example, a Gantt chart showing the project progress can be displayed, allowing the progress and deadlines of each task to be confirmed at a glance.

[0109] Specific examples

[0110] Consider the example where a user types into a terminal, "Show me the agenda for my next meeting in a flowchart." The following steps occur:

[0111] 1. The user enters text into the terminal.

[0112] 2. The terminal converts the input text data into a standard format and sends it to the server.

[0113] 3. The server analyzes the data using natural language processing technology and extracts the keywords "meeting agenda" and "flowchart."

[0114] 4. The server queries the user for the missing information (specific agenda items).

[0115] 5. The user enters the specific agenda item into the terminal and submits it.

[0116] 6. The server generates a flowchart based on the collected information.

[0117] 7. The generated flowchart is sent to the terminal and displayed.

[0118] 8. The user can check the displayed flowchart and edit it if necessary.

[0119] This system allows users to easily visualize complex information and communicate effectively based on it.

[0120] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0121] Step 1:

[0122] The user inputs information into the terminal by text or voice.

[0123] Specifically, the user inputs text information using a keyboard or touch screen, or voice information using a microphone. If the input information is text information such as "I need a Gantt chart showing the progress of the project," the device receives it as is. If the input information is voice information, it is received by the microphone and converted into text using voice recognition technology in the next step.

[0124] Step 2:

[0125] The terminal uses voice recognition software to convert the voice data into text data.

[0126] Specifically, speech recognition software (e.g., Google Speech-to-Text API) is used to convert voice data into text data in real time. This converted text data is used in the next step. The output is sentence text data.

[0127] Step 3:

[0128] The terminal formats the input text data into a standard format (e.g., JSON) and sends it to the server.

[0129] Specifically, the input text data is formatted into JSON format and sent to the server using an HTTP POST request. For example, the formatted JSON data looks like this:

[0130] json

[0131] {

[0132] "request": "I need a Gantt chart to show the progress of my project"

[0133] }

[0134] The output is JSON data that is sent to the server.

[0135] Step 4:

[0136] The server analyzes the received text data using natural language processing (NLP) techniques.

[0137] Specifically, the server uses an NLP engine (e.g., SpaCy or a BERT model) to extract key phrases and keywords from the received text data. For example, keywords such as "project progress" and "Gantt chart" are extracted, and the theme of the text is identified as "project management." The output is the analyzed keywords and theme information.

[0138] Step 5:

[0139] The server queries the user to obtain additional information needed to generate the illustration.

[0140] Specifically, the server identifies the missing information and generates a message to inquire about it to the user. For example, it generates a message asking for the "start date of the project," "details of the tasks," and "deadlines of each task," and sends it to the terminal. The output is a query message.

[0141] Step 6:

[0142] The user receives the inquiry message from the server at the terminal, enters additional information, and resends it.

[0143] Specifically, the user inputs the necessary information according to the inquiry displayed on the terminal. Specific details such as "The project start date is October 1, 2023, Task 1 will be planning until October 5, and Task 2 will be design until October 15" are entered, and the terminal sends this to the server. The output is the additional information entered by the user.

[0144] Step 7:

[0145] The server uses a diagram generation engine to generate diagrams based on all the collected information.

[0146] Specifically, the server runs a diagram generation engine (e.g., D3.js or Syncfusion) to generate an appropriate diagram format (e.g., Gantt chart) based on the collected information. For example, a Gantt chart is generated that visually shows each task and its duration. The output is the generated diagram data.

[0147] Step 8:

[0148] The server transmits the generated illustration data to the terminal.

[0149] Specifically, the generated diagram data is sent to the terminal in an image format (e.g., PNG or JPEG) or in an editable format (e.g., an Excel file or a Google Sheets URL). The output is the diagram data sent to the terminal.

[0150] Step 9:

[0151] The terminal displays the received diagram data to the user and edits it as necessary.

[0152] Specifically, the received diagram data is displayed on the terminal display, and the user can visually check it. For example, a Gantt chart showing the progress of a project is displayed on the screen, and the user can check the progress and deadlines of each task and make edits as necessary. The output is the visually displayed diagram data.

[0153] This series of processing steps allows users to quickly and effectively generate and use visual diagrams based on the information they enter.

[0154] (Application example 1)

[0155] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0156] Conventional factory work process management faces the problem of making it difficult to visually grasp complex processes and vast amounts of data. Effective visualization is essential to efficiently manage work processes and enable workers and managers to quickly understand the processes and issue appropriate instructions. Furthermore, process data exists in different formats, and integrating and visualizing them is time-consuming. To solve these issues, a system is needed that uses voice and text input to automatically analyze work processes and convert them into an appropriate diagram format, such as a flowchart.

[0157] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0158] In this invention, the server includes a text input means, a voice input means, a transmission means for processing the input data and transmitting it to the server, an analysis means for the server to analyze the data and determine an appropriate diagram format, a means for the server to acquire necessary additional information, a means for transmitting the generated diagram data to the terminal, a display means for displaying the diagram data received by the terminal, an input information analysis means for managing the work process in the factory, and a flowchart generation means for visualizing the work process based on the analysis results. This makes it possible to efficiently manage complex work processes in a factory and to grasp the processes in a form that is visually easy for workers and managers to understand.

[0159] "Text input means" refers to a means for a user to input text using a keyboard, touch screen, or the like.

[0160] "Voice input means" refers to a means for a user to input voice using a microphone or voice recognition software.

[0161] The "transmission means" is a means for processing input data and transmitting it to the server.

[0162] "Analysis means" means by which the server analyzes the data and determines the appropriate graphical format.

[0163] The "additional information acquisition means" is a means for the server to acquire the necessary additional information.

[0164] The "graphic data transmission means" is a means for transmitting the generated graphic data to the terminal.

[0165] The "display means" is a means for displaying the graphic data received by the terminal to the user.

[0166] The "input information analysis means" is a means for analyzing input information in order to manage the work process in the factory.

[0167] The "flowchart generation means" is a means for generating a flow chart to visualize the work process based on the analysis results.

[0168] The present invention is a system that analyzes information input from voice or text, and automatically generates and displays visual diagrams such as flowcharts based on that information, in order to facilitate the management of factory work processes. Specific embodiments of the system are described below.

[0169] 1. Text and voice input methods

[0170] The user can input text using a keyboard or touchscreen. They can also input speech through a microphone. Speech input uses speech recognition software to convert the speech data into text. At this stage, the user might input a prompt such as, "First, prepare the materials, then move on to the assembly process, and finally perform quality checks."

[0171] 2. Transmission Method

[0172] The device preprocesses the input data and sends it to the server. In particular, in the case of voice input, the voice data is converted into text data using voice recognition technology. The converted text data is then formatted into a standard format and sent to the server.

[0173] 3. Analysis method

[0174] The server uses natural language processing technology to analyze the text data sent. During this process, key phrases and keywords are extracted using an analysis model using the transformers library. For example, keywords such as "material preparation," "assembly process," and "quality check" are extracted.

[0175] 4. Means of obtaining additional information

[0176] Based on the analysis results, the server queries the user for any missing details. This query is in the form of a message, and can obtain additional information from the user. For example, the server may need to know the detailed work content of each process or the time required for each process.

[0177] 5. Flowchart generation method

[0178] After obtaining the necessary additional information, the server generates a flowchart or Gantt chart. The networkx and matplotlib libraries are used to visualize the information. For example, a flowchart from "material preparation" to "assembly process" and "quality check" is generated.

[0179] 6. Illustrated Data Transmission Methods

[0180] The generated diagrams are sent from the server to the terminal and provided in a format that can be viewed and edited by the user.

[0181] 7. Display means

[0182] The terminal displays the diagram data received from the server to the user. By checking the displayed flowcharts and Gantt charts, the user can easily visually understand the factory work processes. The user can also edit the data as needed.

[0183] In this way, the system of the present invention efficiently manages the work processes in a factory, allowing workers and managers to grasp the processes in a visually easy-to-understand format.

[0184] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0185] Step 1:

[0186] The user inputs information into the terminal by text or voice.

[0187] Input: The user types text using a keyboard or touchscreen, or speaks using a microphone.

[0188] Output: The input text data or audio data.

[0189] Specific operation: The user inputs the prompt sentence: "First, prepare the materials, then move on to the assembly process. Finally, perform quality check."

[0190] Step 2:

[0191] The terminal converts the voice data into text data.

[0192] Input: Audio data.

[0193] Output: Text data.

[0194] Specific operation: The device uses voice recognition technology to convert voice data into text, using the speech_recognition library.

[0195] Step 3:

[0196] The terminal preprocesses the text data and sends it to the server.

[0197] Input: Text data.

[0198] Output: Preprocessed text data.

[0199] Specific operation: The terminal performs formatting processing on the text data, converts it into a standard format, and then sends the data to the server.

[0200] Step 4:

[0201] The server analyzes the data using natural language processing technology.

[0202] Input: Preprocessed text data.

[0203] Output: Keyphrases or keywords.

[0204] Specific operation: The server uses the transformers library to extract key phrases and keywords such as "material preparation," "assembly process," and "quality check" from the text data.

[0205] Step 5:

[0206] The server prompts the user for any additional information that is missing.

[0207] Input: Keywords obtained from the analysis results.

[0208] Output: Additional information from the user.

[0209] Specific operation: The server queries the user for missing information in the form of a message, for example, to confirm "detailed work content for each process" and "time required for each process."

[0210] Step 6:

[0211] The server generates the flowchart.

[0212] Input: Additional information collected from the user and analysis results.

[0213] Output: Flowchart data.

[0214] What it does: The server uses the networkx and matplotlib libraries to generate a flowchart based on the information it collects.

[0215] Step 7:

[0216] The server transmits the generated flowchart data to the terminal.

[0217] Input: Flowchart data.

[0218] Output: Illustrated data sent to the terminal.

[0219] Specific operation: The server sends the flowchart data to the terminal and provides it in a format (e.g., image data) that the user can view and edit.

[0220] Step 8:

[0221] The terminal displays the received flowchart data to the user.

[0222] Input: Illustrated data sent from the server.

[0223] Output: The flowchart that is displayed to the user.

[0224] Specific operation: The terminal displays the received flowchart data on the screen, allowing the user to visually check the factory work process.

[0225] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0226] The present invention relates to a system that converts information input via text or voice into a diagram format and adjusts the diagram by recognizing the user's emotions. This system provides a function that allows a user to input information using text or voice, analyzes the information, generates and displays an appropriate diagram, and also a function that recognizes the user's emotions and adjusts the layout and color of the diagram. Specific embodiments of the present invention are described below.

[0227] 1. Text and voice input methods

[0228] Users input information into the device either by text or voice. Text input is done using a keyboard or touchscreen, while voice input is done using a microphone or voice recognition software. For example, a user can type, "I need a Gantt chart showing the progress of a project."

[0229] 2. Transmission Method

[0230] The device preprocesses the input data and sends it to the server. In the case of voice input, the device uses voice recognition technology to convert the voice data into text data, which is then formatted into a standard format and sent to the server.

[0231] 3. Emotion recognition means

[0232] The server is equipped with an emotion engine for recognizing the user's emotion from the received voice data. This engine uses voice analysis technology to determine the emotion from the user's tone of voice and expressions. For example, it can recognize emotions such as nervousness, relaxation, anger, etc. from the user's tone of voice.

[0233] 4. Analysis method

[0234] After emotion recognition, the server uses natural language processing technology to analyze the text data. The analysis engine extracts key phrases and keywords to identify themes within the text. For example, keywords like "project progress" and "Gantt chart" may be extracted.

[0235] 5. Deciding on the illustration format

[0236] The server then determines the appropriate illustration format based on the analysis results. In this example, a Gantt chart is selected as the appropriate illustration format. At the same time, the layout and color of the illustration are adjusted based on the user's emotions. For example, if the user is relaxed, calm colors are used, and if the user is tense, a visually gentle color scheme is used.

[0237] 6. Means of obtaining additional information

[0238] To generate a Gantt chart, the server needs detailed information such as specific tasks, deadlines, progress, etc. If this additional information is missing, the server queries the user to obtain the necessary data. For example, the server might send a message to the user asking for the project start date, task details, and deadlines for each task.

[0239] 7. Illustration Generation Method

[0240] After retrieving the necessary data, the server uses a diagram generation engine to generate the appropriate diagram, which supports formats such as flowcharts, Gantt charts, graphs, and tables. For example, a Gantt chart showing each project task and its duration can be generated and adjusted according to the user's emotions.

[0241] 8. Illustrated Data Transmission Methods

[0242] The generated diagram is sent from the server to the device. The diagram data is sent in a format that the user can view and edit (e.g., image data, Excel file, Google Sheets URL, etc.).

[0243] 9. Display means

[0244] The device displays the diagram data received from the server to the user. The user can visually check the diagram on the device screen and edit it as needed. For example, a Gantt chart showing the progress of a project can be displayed on the screen, allowing the user to check the progress and deadlines of each task at a glance. In addition, layouts and colors are applied according to emotions, making it easier for users to use the diagrams.

[0245] In this way, the system of the present invention can analyze the information entered by the user in real time, convert it into an appropriate diagram format for display, and adjust the diagram taking into account the user's feelings, thereby enabling complex information to be visualized concisely and greatly improving communication efficiency.

[0246] The processing flow will be explained below.

[0247] Step 1:

[0248] The user inputs text or voice into the device. For example, the user inputs, "I need a Gantt chart showing the progress of a project."

[0249] Step 2:

[0250] The device receives input or speech. In the case of speech input, the device uses speech recognition software to convert the speech data into text.

[0251] Step 3:

[0252] The terminal preprocesses the text data and formats it into a standard format, then sends the formatted data to the server.

[0253] Step 4:

[0254] The server analyzes the received text data. Natural language processing techniques are used to extract key phrases and keywords and identify themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[0255] Step 5:

[0256] The server uses an emotion recognition engine to analyze emotions from the voice data sent by the user, for example, determining whether the user is nervous or relaxed from their voice tone.

[0257] Step 6:

[0258] The server determines the appropriate diagram format based on the analysis results. At the same time, it adjusts the layout and color of the diagram based on the user's emotions. For example, a Gantt chart is selected as the appropriate diagram format, and the color scheme is determined according to the user's emotions.

[0259] Step 7:

[0260] The server checks the details required to generate the Gantt chart. If the required information is missing, the server will ask the user questions such as "Please tell me the start date of the project and details of the tasks."

[0261] Step 8:

[0262] The user answers with additional information. For example, the user might say, "The project started on October 1, 2023."

[0263] Step 9:

[0264] The server receives additional information and adds it to the data used to generate the Gantt chart, such as project tasks, deadlines, and progress.

[0265] Step 10:

[0266] The server generates a Gantt chart using a diagram generation engine. The generated Gantt chart is converted into image data or an editable format within the server, and the layout and color are applied according to the user's emotions.

[0267] Step 11:

[0268] The server transmits the generated diagram data to the terminal in a format that the user can view and edit.

[0269] Step 12:

[0270] The device displays the received graphical data to the user. For example, a Gantt chart is displayed on the device screen, allowing the user to visually check the progress of the project. The layout and color are also applied according to the emotion.

[0271] Example 2

[0272] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0273] Conventional information input systems can properly analyze information entered via text or voice and convert it into diagram format, but they lack the ability to recognize the user's emotions and adjust the layout and color of the diagram, limiting their ability to improve the user experience. Furthermore, they lacked efficiency and flexibility when acquiring additional information and transmitting and displaying diagrams.

[0274] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an emotion recognition means, a means for analyzing data using natural language processing technology, and a means for adjusting the layout and color of the illustration based on the user's emotion. This makes it possible to generate an appropriate illustration format taking the user's emotion into consideration and to efficiently and flexibly obtain additional information the user needs.

[0275] "Text input means" refers to a means by which a user inputs text using a keyboard or a touch screen.

[0276] "Voice input means" refers to means for obtaining the user's voice as input data using a microphone or voice recognition software.

[0277] The "transmission means" is a means for preprocessing input data, converting it into a standard format, and transmitting it to the server.

[0278] "Analysis means" refers to means by which the server analyzes the received data and determines an appropriate graphical format.

[0279] The "emotion recognition means" is a means by which the server determines the user's emotions from voice and text expressions.

[0280] The "additional information acquisition means" is a means for the server to inquire of the user and acquire detailed information necessary for generating an illustration.

[0281] The "illustration generating means" is a means for generating an appropriate illustration using all collected data.

[0282] The "display means" is a means for displaying the graphic data received by the terminal to the user.

[0283] "Speech recognition technology" is a technology that converts input speech into text data.

[0284] "Natural language processing technology" is a technology that analyzes input text data and understands key phrases, keywords, and context.

[0285] "Illustration format" refers to a format that visually presents data, such as a Gantt chart, flow chart, graph, or table.

[0286] The "layout adjustment means" is a means for adjusting the layout of illustrations based on the user's feelings.

[0287] The "color adjustment means" is a means for adjusting the color scheme of the illustration based on the user's feelings.

[0288] The present invention relates to a system that converts information input via text or voice into an illustration format and adjusts the illustration by recognizing the user's emotions. This system is intended to be used by an unspecified number of users and aims to generate appropriate illustrations in real time. The following describes in detail the embodiments of the present invention.

[0289] Text and voice input methods

[0290] Users input information through the device, either through a keyboard or touchscreen for text input or through a microphone and voice recognition software for voice input. For example, a user might type, "I need a Gantt chart showing the progress of a project."

[0291] Sending data

[0292] The device preprocesses the data entered by the user, converts it into a standard format, and sends it to the server. In the case of voice input, the device uses voice recognition technology (e.g., Google voice recognition API) to convert the voice data into text data, formats it into JSON format, and sends it to the server.

[0293] emotion recognition means

[0294] The server processes the received data and recognizes the user's emotions using an emotion engine (e.g., IBM Watson® Emotion Analysis) that determines the user's emotions from voice tone and text expressions, for example, identifying whether the user is tense, relaxed, angry, etc.

[0295] Data analysis

[0296] After emotion recognition, the server analyzes the input text data using natural language processing technology (e.g., OpenAI® GPT-4®). This analysis engine extracts key phrases and keywords from the text and identifies the subject and purpose of the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[0297] Deciding on an illustration format

[0298] The server determines the appropriate illustration format based on the analysis results and emotion recognition results. For example, a Gantt chart is deemed appropriate, and the layout and color of the illustration are also adjusted according to the user's emotional state. If the user is relaxed, calm colors are used, and if the user is tense, a color scheme that is less visually demanding is selected.

[0299] Obtaining additional information

[0300] The server collects detailed information (e.g., tasks, deadlines, and progress) required to generate the diagram. If the required information is missing, it prompts the user for additional information. For example, it sends questions such as, "What is the project start date, task details, and deadlines for each task?"

[0301] Generating illustrations

[0302] Using all the collected data, the server uses a diagram generation engine (e.g., D3.js or Matplotlib) to generate the final diagram, which can be adjusted in layout and color based on the user's preferences. For example, a Gantt chart showing each project task and its duration can be generated.

[0303] Illustrated data transmission and display

[0304] The generated diagram data is sent from the server to the terminal. This diagram data is provided in a format that the user can view and edit (e.g. PNG, Excel file, Google Sheets URL). The terminal displays the received diagram data to the user, allowing the user to visually check the contents of the diagram and edit it as necessary.

[0305] Example prompt sentences

[0306] For example, by inputting the following prompts into a generative AI model, it can generate diagrams that meet the user's requirements:

[0307] You want to create a Gantt chart that details the progress of a project. The project starts on October 1, 2023, and the major tasks include research, design, development, testing, and deployment. You want it to reflect the deadlines and progress of each task. Also, your current emotional state is relaxed. Choose a layout and color scheme that reflects this.

[0308] The above is a specific embodiment for carrying out the present invention. This system is capable of quickly and effectively converting user input information into illustrations and presenting the results in an optimal format.

[0309] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0310] Step 1: Enter your information

[0311] Users input information into the device either through text or voice. Text input is done using a keyboard or touchscreen. Voice input is done using a microphone or voice recognition software. The input data can be in text or voice data format. For example, a user might say, "I need a Gantt chart showing the progress of a project."

[0312] Step 2: Preprocess and send data

[0313] The device preprocesses the input data. In the case of voice data, it converts it into text data using voice recognition technology (e.g., Google voice recognition API). The converted text data is then formatted into a standard format (e.g., JSON) and sent to the server. Specifically, the device converts the voice into text, formats the text as JSON, and sends it to the server via an HTTP request.

[0314] Step 3: Emotion Recognition

[0315] The server receives the data sent from the device and analyzes the user's emotions using an emotion recognition engine (e.g., IBM Watson Emotion Analysis). The input data is analyzed based on voice tone and text expressions to determine emotions such as tension, relaxation, and anger. The emotion recognition engine analyzes the input data and returns the user's emotional state (e.g., tension, relaxation, anger) as its output.

[0316] Step 4: Data analysis

[0317] The server receives the emotion recognition results and then analyzes the input text data using natural language processing technology (e.g., OpenAI GPT-4). This analysis engine extracts key phrases and keywords from the text and identifies the subject and intent. For example, it extracts input keywords such as "project progress" and "Gantt chart." The subject and intent are provided as specific keywords as output.

[0318] Step 5: Decide on a graphical format

[0319] The server determines the appropriate diagram format based on the analysis results and emotion recognition results. For example, the analysis results may determine that a "Gantt chart" is appropriate. The layout and color are also adjusted taking into account the emotion recognition results. Specifically, if the user is relaxed, calm colors are selected, and if the user is tense, a color scheme that is less visually demanding is chosen. The output is the diagram format and settings for adjusting the layout.

[0320] Step 6: Obtain additional information

[0321] The server collects the necessary details (tasks, deadlines, progress, etc.) to generate the diagram. It sends a query message to the user for any missing information. For example, it sends specific questions such as "What is the project start date, task details, and deadlines for each task?" and waits for a response from the user. The input is the question from the server, and the output is the answer from the user.

[0322] Step 7: Generate the diagram

[0323] The server uses all the collected data to generate diagrams using a diagram generation engine (e.g., D3.js or Matplotlib). At this time, the diagram is generated with layout and color adjusted based on the results of emotion recognition. Specifically, a Gantt chart showing the duration of each task is generated. The input is the collected detailed information and adjustment settings, and the output is the generated diagram data.

[0324] Step 8: Submitting diagram data

[0325] The generated diagram data is sent from the server to the terminal. This diagram data is provided in a format that the user can view and edit (e.g. PNG, Excel file, Google Sheets URL). Specifically, the generated PNG file is sent to the terminal via an HTTP response. The generated diagram data is used as input, and transmission to the user's terminal is completed as output.

[0326] Step 9: Displaying the diagram

[0327] The terminal displays the diagram data received from the server to the user. The user can check the diagram on the terminal screen and edit it as necessary. For example, the received PNG file can be displayed on the screen to visually check the details and progress of each task. It is also possible to make additional edits and adjustments based on the contents of the diagram. The input is the diagram data received from the server, and the output is the display to the user.

[0328] (Application example 2)

[0329] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0330] Traditional brick-and-mortar stores have faced the challenge of quickly and accurately providing customers with the products and information they are looking for. Particularly in large stores or stores with complex layouts, customers often spend a lot of time trying to find the product they are looking for. Furthermore, providing information without considering the customer's emotions or state can sometimes detract from the customer experience. To solve these problems, a system was needed that could generate and display appropriate illustrations in real time based on the information entered by the customer.

[0331] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0332] In this invention, the server includes a text input means, a voice input means, a transmission means for processing the input data and transmitting it to the server, an analysis means for the server to analyze the data and determine an appropriate illustration format, a means for the server to acquire necessary additional information, a means for transmitting the generated illustration data to the terminal, a display means for displaying the illustration data received by the terminal, a means for a user wearing smart glasses to input store information by voice or text, and an emotion recognition means for recognizing the user's emotions and adjusting the layout and color of the illustrations. This allows customers to easily check the location of products and ask questions in the store, and also allows them to receive information presented according to their emotions and state.

[0333] "Text input means" refers to a device or method for a user to input text, including a keyboard and a touch screen.

[0334] "Voice input means" refers to a device or method that allows a user to input information by voice, and includes a microphone and voice recognition software.

[0335] "Transmission means" refers to a device or method for processing input data and transmitting it to a server, and includes a communication module and a data format conversion means.

[0336] "Analysis means" refers to a method or device for analyzing the data received by the server and determining an appropriate graphical format, including natural language processing techniques and keyword extraction algorithms.

[0337] "Means for obtaining additional information" refers to a device or method for querying a user or collecting additional data required for generating an illustration.

[0338] The "means for transmitting generated graphic data" refers to a device or method for transmitting graphic data generated by the server to the terminal.

[0339] "Display means" refers to a device or method for displaying the graphical data received by the terminal to the user, and includes a display and a VR device.

[0340] "Smart glasses" refers to a wearable device that has the ability to receive and display data.

[0341] "Emotion recognition means" refers to a method or device for analyzing and identifying a user's emotions from vocal tone and facial expressions, and includes emotion analysis engines and machine learning technologies.

[0342] "Illustrative formats" refers to formats that visually display data and include maps, flowcharts, graphs, tables, etc.

[0343] This invention provides a system that uses smart glasses to improve customer experience in brick-and-mortar stores. This system analyzes information entered by customers via voice or text in real time, and generates and displays appropriate illustrations. It also recognizes customer emotions and adjusts the layout and color of the illustrations accordingly.

[0344] Hardware and software used

[0345] Hardware:

[0346] Smart glasses (e.g. general-purpose wearable devices)

[0347] microphone

[0348] Display Device

[0349] software:

[0350] Natural language processing engine (e.g., cloud natural language processing API)

[0351] Speech recognition software (e.g., cloud speech recognition services)

[0352] Sentiment analysis engine (e.g., cloud sentiment analysis service)

[0353] Diagram generation engine (e.g., a web-based diagram generation library)

[0354] Data processing and calculation methods

[0355] 1. Data entry and submission:

[0356] The user inputs information through the smart glasses by voice or text. The input voice data is converted into text data using voice recognition software. The text data is then sent to the server.

[0357] 2. Emotion recognition:

[0358] The server analyzes the received voice data using an emotion analysis engine to identify the user's emotion, such as tension, relaxation, or anger, based on the user's tone of voice.

[0359] 3. Data Analysis:

[0360] The server analyzes the text data using a natural language processing engine, which extracts key phrases and keywords and identifies themes in the input information.

[0361] 4. Decide on the illustration format:

[0362] The server determines the appropriate illustration format based on the analysis results. It also adjusts the layout and color of the illustration based on the user's emotions. For example, if the user uses a relaxed voice tone, a map with calm colors will be generated.

[0363] 5. Obtaining additional information:

[0364] If necessary, the server queries the user for any missing additional information to complete the required data.

[0365] 6. Generating and sending diagrams:

[0366] The server uses an illustration generation engine to generate an appropriate illustration and transmits the data to the smart glasses, which then display the generated illustration on their display.

[0367] Specific examples

[0368] When a customer asks the smart glasses, "Where are the new dresses?", the voice data is sent to the server for analysis. The server extracts the keywords "new" and "dress" and generates an illustration showing the location of the appropriate category. If the server recognizes the customer's voice tone as relaxed, the illustration will be displayed in a calming blue color. The customer can then view the illustration through the smart glasses.

[0369] Prompt Sentence Examples

[0370] It analyzes voice data and illustrates the location of the product the customer is asking about.

[0371] Voice: "Where are the new dresses?"

[0372] Emotion: Relaxed tone of voice

[0373] Analyzed information: "New item", "Dress"

[0374] Generated illustration: store map, location of category "dresses", blue relaxed color scheme

[0375] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0376] Step 1:

[0377] A user wears smart glasses and inputs information by voice or text. The input data is captured from the smart glasses' microphone and keyboard. The input voice data is converted into text using voice recognition software.

[0378] Step 2:

[0379] The device preprocesses the input data and sends it to the cloud server, where it undergoes format conversion and optimization, converting voice data to text, and transmitting it according to the communication protocol. The input information is properly formatted and prepared for analysis on the server side.

[0380] Step 3:

[0381] The server analyzes the received text data using a natural language processing engine, which extracts key phrases and keywords and identifies the subject of the information entered. During this step, multiple filtering and analysis algorithms are applied to identify the category and subject of the information.

[0382] Step 4:

[0383] The server analyzes the received voice data using an emotion analysis engine to identify the user's emotions. The emotion is determined from the tone of the voice, speaking style, emphasis, etc., and emotional states such as relaxation, tension, joy, and anger are identified. The emotional data is used to adjust the layout and color of the illustrations.

[0384] Step 5:

[0385] Based on the analysis results, the server determines the appropriate illustration format, such as a map, flowchart, graph, or table, and then adjusts the layout and color scheme according to the user's emotions. For example, a relaxed user might be assigned a softer color scheme.

[0386] Step 6:

[0387] If the server does not have enough additional information, it will ask the user for a query. For example, if the question is "Where is the new dress?", the server will generate and send a query message to the user to ask for additional data such as the start date and location if more detailed product information is needed.

[0388] Step 7:

[0389] The server sends the generated illustration data to the smart glasses terminal, where it uses an illustration generation engine to generate a visually easy-to-understand illustration according to the selected format, and then the generated illustration data is sent from the cloud server to the smart glasses.

[0390] Step 8:

[0391] The smart glasses display the diagram data received from the server. Users can visually confirm the information through the glasses' display. The layout and color of the diagrams displayed are based on the analysis results and emotions, allowing users to comfortably view the information.

[0392] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0393] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0394] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0395] [Second embodiment]

[0396] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0397] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0398] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0399] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0400] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0401] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0402] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0403] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0404] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0405] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0406] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0407] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0408] The present invention relates to a system that converts information input by text or voice into a diagram format and displays it. This system provides a function in which a user inputs information using text or voice, analyzes the information, and generates and displays an appropriate diagram. Specific embodiments of the present invention are described below.

[0409] 1. Text and voice input methods

[0410] Users input information into the device either by text or voice. Text input is done using a keyboard or touchscreen, while voice input is done using a microphone or voice recognition software. For example, a user can type, "I need a Gantt chart showing the progress of a project."

[0411] 2. Transmission Method

[0412] The device preprocesses the input data and sends it to the server. In the case of voice input, the device uses voice recognition technology to convert the voice data into text data, which is then formatted into a standard format and sent to the server.

[0413] 3. Analysis method

[0414] The server uses natural language processing technology to analyze the text data sent. The analysis engine extracts key phrases and keywords and identifies themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted, and based on this, the text is determined to be information about project management.

[0415] 4. Means of obtaining additional information

[0416] To generate a Gantt chart, the server needs detailed information such as specific tasks, deadlines, and progress. If this additional information is missing, the server queries the user to obtain the necessary data. For example, the server might send a message to the user requesting the "project start date," "task details," and "deadline for each task."

[0417] 5. Illustration Generation Method

[0418] After retrieving the necessary data, the server uses a diagram generation engine to generate the appropriate diagram, which supports formats such as flowcharts, Gantt charts, graphs, tables, etc. For example, a Gantt chart showing each task in a project and its duration can be generated.

[0419] 6. Illustrated Data Transmission Methods

[0420] The generated diagram is sent from the server to the device. The diagram data is sent in a format that the user can view and edit (e.g., image data, Excel file, Google Sheets URL, etc.).

[0421] 7. Display means

[0422] The device receives the diagram data from the server and displays it to the user. The user can visually check the diagram on the device screen and edit it as needed. For example, a Gantt chart showing the progress of a project can be displayed on the screen, allowing the user to check the progress and deadlines of each task at a glance.

[0423] In this way, the system of the present invention can analyze the information entered by the user in real time and convert it into an appropriate diagram format for display, thereby enabling complex information to be visualized concisely and greatly improving communication efficiency.

[0424] The processing flow will be explained below.

[0425] Step 1:

[0426] The user inputs text or voice into the device. For example, the user inputs, "I need a Gantt chart showing the progress of a project."

[0427] Step 2:

[0428] The device receives input or speech. In the case of speech input, the device uses speech recognition software to convert the speech data into text.

[0429] Step 3:

[0430] The terminal preprocesses the text data and formats it into a standard format, then sends the formatted data to the server.

[0431] Step 4:

[0432] The server analyzes the received text data. Natural language processing techniques are used to extract key phrases and keywords and identify themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[0433] Step 5:

[0434] The server then determines the appropriate diagram format based on the analysis results. In this example, a Gantt chart is selected as the appropriate diagram format.

[0435] Step 6:

[0436] The server checks the details needed to generate the Gantt chart. If the required information is missing, the server prompts the user for additional information. For example, "What is the start date of the project?"

[0437] Step 7:

[0438] The user answers with additional information. For example, the user answers, "The project started on October 1, 2023."

[0439] Step 8:

[0440] The server receives additional information and adds it to the data used to generate the Gantt chart, such as project tasks, deadlines, and progress.

[0441] Step 9:

[0442] The server generates a Gantt chart using a diagram generation engine, and the generated Gantt chart is converted into image data or an editable format within the server.

[0443] Step 10:

[0444] The server transmits the generated diagram data to the terminal in a format that the user can view and edit.

[0445] Step 11:

[0446] The terminal displays the received graphical data to the user, for example, a Gantt chart is displayed on the terminal screen, allowing the user to visually check the progress of the project.

[0447] Step 12:

[0448] Users can check the displayed diagrams and edit them as necessary, which allows users to communicate more efficiently.

[0449] Example 1

[0450] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0451] Conventional information input systems have difficulty properly analyzing information entered via text or voice and converting it into a visually easy-to-understand diagram format. In particular, when the user's input information is insufficient, there is a lack of a means to effectively acquire additional information and generate a diagram, which reduces user convenience. Furthermore, the inability to update or edit diagram data in real time makes it difficult to make decisions based on the latest information.

[0452] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0453] In this invention, the server includes a means for converting input data into text using speech recognition technology, a means for analyzing the data using natural language processing technology, and a means for updating the generated illustrations in real time. This allows the information input by the user via text or voice to be properly analyzed, and missing information to be effectively acquired and an illustration to be generated. Furthermore, editing and real-time updating of the generated illustration data allows for efficient decision-making based on the latest information.

[0454] A "text input means" is a device and method that allows a user to input text information using a keyboard or touch screen.

[0455] "Voice input means" refers to a device and method including a microphone and voice recognition software for inputting user voice information and processing it as digital data.

[0456] The "transmitting means" refers to a device and method including a network interface and communication protocol for preprocessing input data and transmitting it to a server.

[0457] "Analysis means" refers to software and algorithms that the server uses to analyze the received data using natural language processing techniques and determine an appropriate graphical format.

[0458] The "acquisition means" refers to a device and method for acquiring additional information required by the server to generate the illustration from the user through a query.

[0459] "Illustration generation means" refers to software and algorithms for generating appropriate illustrations (Gantt charts, flow charts, graphs, tables) based on collected data.

[0460] The "transmitting means" refers to an apparatus and method including a network interface and a communication protocol for the server to transmit the generated illustration data to the terminal.

[0461] "Display means" refers to a display and software for visually displaying the graphical data received by the terminal.

[0462] The "query means" refers to a device and method by which the server queries the user for necessary information when additional information is missing.

[0463] "Speech recognition technology" refers to software and algorithms used to convert voice data into text data.

[0464] "Natural language processing technology" refers to software and algorithms that analyze text data, extract key phrases and keywords, and perform semantic analysis.

[0465] "Real-time update means" refers to a device and method for instantly updating generated graphical data based on the latest information.

[0466] The "editing means" refers to software and an interface that allows the user to edit the received graphic data on the terminal.

[0467] The system of the present invention is a system that can convert information input by a user in text or voice into a diagram format and visually display it. To implement this system, the following hardware and software are used.

[0468] Hardware and Software

[0469] Enter information

[0470] Users can input text using a keyboard or touchscreen, such as "I need a Gantt chart showing the progress of a project." A microphone is also used for voice input, which is then converted into text using speech recognition software, such as the Google Speech-to-Text API.

[0471] Data transmission

[0472] The terminal formats the input text data into a standard format and sends it to the server, for example, in JSON format, using an HTTP POST request.

[0473] Data analysis

[0474] The server analyzes the received text data using natural language processing (NLP) techniques. Analysis engines such as SpaCy and the BERT model are used. This allows the server to extract key phrases and keywords and identify themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[0475] Obtaining additional information

[0476] The server queries the user to obtain additional information needed to generate the diagram. This information is obtained via HTTP requests or message transmission. The user enters the requested information into the terminal and resubmits it. For example, the user enters specific details such as "The project start date is October 1, 2023, Task 1 will be planning until October 5, and Task 2 will be design until October 15."

[0477] Illustration generation

[0478] After the server collects all the necessary data, it uses a diagram generation engine, such as D3.js or Syncfusion, to generate diagrams, such as Gantt charts, that show each project task and its duration.

[0479] Sending diagram data

[0480] The generated diagram data is sent from the server to the terminal via a network interface or communication protocol. The diagram data is often sent in image format or an editable format.

[0481] Data Display

[0482] The device then displays the received diagram data to the user. A display and corresponding software are used as the display method. The user can check the diagram on the device screen and edit it as needed. For example, a Gantt chart showing the project progress can be displayed, allowing the progress and deadlines of each task to be confirmed at a glance.

[0483] Specific examples

[0484] Consider the example where a user types into a terminal, "Show me the agenda for my next meeting in a flowchart." The following steps occur:

[0485] 1. The user enters text into the terminal.

[0486] 2. The terminal converts the input text data into a standard format and sends it to the server.

[0487] 3. The server analyzes the data using natural language processing technology and extracts the keywords "meeting agenda" and "flowchart."

[0488] 4. The server queries the user for the missing information (specific agenda items).

[0489] 5. The user enters the specific agenda item into the terminal and submits it.

[0490] 6. The server generates a flowchart based on the collected information.

[0491] 7. The generated flowchart is sent to the terminal and displayed.

[0492] 8. The user can check the displayed flowchart and edit it if necessary.

[0493] This system allows users to easily visualize complex information and communicate effectively based on it.

[0494] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0495] Step 1:

[0496] The user inputs information into the terminal by text or voice.

[0497] Specifically, the user inputs text information using a keyboard or touch screen, or voice information using a microphone. If the input information is text information such as "I need a Gantt chart showing the progress of the project," the device receives it as is. If the input information is voice information, it is received by the microphone and converted into text using voice recognition technology in the next step.

[0498] Step 2:

[0499] The terminal uses voice recognition software to convert the voice data into text data.

[0500] Specifically, speech recognition software (e.g., Google Speech-to-Text API) is used to convert voice data into text data in real time. This converted text data is used in the next step. The output is sentence text data.

[0501] Step 3:

[0502] The terminal formats the input text data into a standard format (e.g., JSON) and sends it to the server.

[0503] Specifically, the input text data is formatted into JSON format and sent to the server using an HTTP POST request. For example, the formatted JSON data looks like this:

[0504] json

[0505] {

[0506] "request": "I need a Gantt chart to show the progress of my project"

[0507] }

[0508] The output is JSON data that is sent to the server.

[0509] Step 4:

[0510] The server analyzes the received text data using natural language processing (NLP) techniques.

[0511] Specifically, the server uses an NLP engine (e.g., SpaCy or a BERT model) to extract key phrases and keywords from the received text data. For example, keywords such as "project progress" and "Gantt chart" are extracted, and the theme of the text is identified as "project management." The output is the analyzed keywords and theme information.

[0512] Step 5:

[0513] The server queries the user to obtain additional information needed to generate the illustration.

[0514] Specifically, the server identifies the missing information and generates a message to inquire about it to the user. For example, it generates a message asking for the "start date of the project," "details of the tasks," and "deadlines of each task," and sends it to the terminal. The output is a query message.

[0515] Step 6:

[0516] The user receives the inquiry message from the server at the terminal, enters additional information, and resends it.

[0517] Specifically, the user inputs the necessary information according to the inquiry displayed on the terminal. Specific details such as "The project start date is October 1, 2023, Task 1 will be planning until October 5, and Task 2 will be design until October 15" are entered, and the terminal sends this to the server. The output is the additional information entered by the user.

[0518] Step 7:

[0519] The server uses a diagram generation engine to generate diagrams based on all the collected information.

[0520] Specifically, the server runs a diagram generation engine (e.g., D3.js or Syncfusion) to generate an appropriate diagram format (e.g., Gantt chart) based on the collected information. For example, a Gantt chart is generated that visually shows each task and its duration. The output is the generated diagram data.

[0521] Step 8:

[0522] The server transmits the generated illustration data to the terminal.

[0523] Specifically, the generated diagram data is sent to the terminal in an image format (e.g., PNG or JPEG) or in an editable format (e.g., an Excel file or a Google Sheets URL). The output is the diagram data sent to the terminal.

[0524] Step 9:

[0525] The terminal displays the received diagram data to the user and edits it as necessary.

[0526] Specifically, the received diagram data is displayed on the terminal display, and the user can visually check it. For example, a Gantt chart showing the progress of a project is displayed on the screen, and the user can check the progress and deadlines of each task and make edits as necessary. The output is the visually displayed diagram data.

[0527] This series of processing steps allows users to quickly and effectively generate and use visual diagrams based on the information they enter.

[0528] (Application example 1)

[0529] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0530] Conventional factory work process management faces the problem of making it difficult to visually grasp complex processes and vast amounts of data. Effective visualization is essential to efficiently manage work processes and enable workers and managers to quickly understand the processes and issue appropriate instructions. Furthermore, process data exists in different formats, and integrating and visualizing them is time-consuming. To solve these issues, a system is needed that uses voice and text input to automatically analyze work processes and convert them into an appropriate diagram format, such as a flowchart.

[0531] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0532] In this invention, the server includes a text input means, a voice input means, a transmission means for processing the input data and transmitting it to the server, an analysis means for the server to analyze the data and determine an appropriate diagram format, a means for the server to acquire necessary additional information, a means for transmitting the generated diagram data to the terminal, a display means for displaying the diagram data received by the terminal, an input information analysis means for managing the work process in the factory, and a flowchart generation means for visualizing the work process based on the analysis results. This makes it possible to efficiently manage complex work processes in a factory and to grasp the processes in a form that is visually easy for workers and managers to understand.

[0533] "Text input means" refers to a means for a user to input text using a keyboard, touch screen, or the like.

[0534] "Voice input means" refers to a means for a user to input voice using a microphone or voice recognition software.

[0535] The "transmission means" is a means for processing input data and transmitting it to the server.

[0536] "Analysis means" means by which the server analyzes the data and determines the appropriate graphical format.

[0537] The "additional information acquisition means" is a means for the server to acquire the necessary additional information.

[0538] The "graphic data transmission means" is a means for transmitting the generated graphic data to the terminal.

[0539] The "display means" is a means for displaying the graphic data received by the terminal to the user.

[0540] The "input information analysis means" is a means for analyzing input information in order to manage the work process in the factory.

[0541] The "flowchart generation means" is a means for generating a flow chart to visualize the work process based on the analysis results.

[0542] The present invention is a system that analyzes information input from voice or text, and automatically generates and displays visual diagrams such as flowcharts based on that information, in order to facilitate the management of factory work processes. Specific embodiments of the system are described below.

[0543] 1. Text and voice input methods

[0544] The user can input text using a keyboard or touchscreen. They can also input speech through a microphone. Speech input uses speech recognition software to convert the speech data into text. At this stage, the user might input a prompt such as, "First, prepare the materials, then move on to the assembly process, and finally perform quality checks."

[0545] 2. Transmission Method

[0546] The device preprocesses the input data and sends it to the server. In particular, in the case of voice input, the voice data is converted into text data using voice recognition technology. The converted text data is then formatted into a standard format and sent to the server.

[0547] 3. Analysis method

[0548] The server uses natural language processing technology to analyze the text data sent. During this process, key phrases and keywords are extracted using an analysis model using the transformers library. For example, keywords such as "material preparation," "assembly process," and "quality check" are extracted.

[0549] 4. Means of obtaining additional information

[0550] Based on the analysis results, the server queries the user for any missing details. This query is in the form of a message, and can obtain additional information from the user. For example, the server may need to know the detailed work content of each process or the time required for each process.

[0551] 5. Flowchart generation method

[0552] After obtaining the necessary additional information, the server generates a flowchart or Gantt chart. The networkx and matplotlib libraries are used to visualize the information. For example, a flowchart from "material preparation" to "assembly process" and "quality check" is generated.

[0553] 6. Illustrated Data Transmission Methods

[0554] The generated diagrams are sent from the server to the terminal and provided in a format that can be viewed and edited by the user.

[0555] 7. Display means

[0556] The terminal displays the diagram data received from the server to the user. By checking the displayed flowcharts and Gantt charts, the user can easily visually understand the factory work processes. The user can also edit the data as needed.

[0557] In this way, the system of the present invention efficiently manages the work processes in a factory, allowing workers and managers to grasp the processes in a visually easy-to-understand format.

[0558] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0559] Step 1:

[0560] The user inputs information into the terminal by text or voice.

[0561] Input: The user types text using a keyboard or touchscreen, or speaks using a microphone.

[0562] Output: The input text data or audio data.

[0563] Specific operation: The user inputs the prompt sentence: "First, prepare the materials, then move on to the assembly process. Finally, perform quality check."

[0564] Step 2:

[0565] The terminal converts the voice data into text data.

[0566] Input: Audio data.

[0567] Output: Text data.

[0568] Specific operation: The device uses voice recognition technology to convert voice data into text, using the speech_recognition library.

[0569] Step 3:

[0570] The terminal preprocesses the text data and sends it to the server.

[0571] Input: Text data.

[0572] Output: Preprocessed text data.

[0573] Specific operation: The terminal performs formatting processing on the text data, converts it into a standard format, and then sends the data to the server.

[0574] Step 4:

[0575] The server analyzes the data using natural language processing technology.

[0576] Input: Preprocessed text data.

[0577] Output: Keyphrases or keywords.

[0578] Specific operation: The server uses the transformers library to extract key phrases and keywords such as "material preparation," "assembly process," and "quality check" from the text data.

[0579] Step 5:

[0580] The server prompts the user for any additional information that is missing.

[0581] Input: Keywords obtained from the analysis results.

[0582] Output: Additional information from the user.

[0583] Specific operation: The server queries the user for missing information in the form of a message, for example, to confirm "detailed work content for each process" and "time required for each process."

[0584] Step 6:

[0585] The server generates the flowchart.

[0586] Input: Additional information collected from the user and analysis results.

[0587] Output: Flowchart data.

[0588] What it does: The server uses the networkx and matplotlib libraries to generate a flowchart based on the information it collects.

[0589] Step 7:

[0590] The server transmits the generated flowchart data to the terminal.

[0591] Input: Flowchart data.

[0592] Output: Illustrated data sent to the terminal.

[0593] Specific operation: The server sends the flowchart data to the terminal and provides it in a format (e.g., image data) that the user can view and edit.

[0594] Step 8:

[0595] The terminal displays the received flowchart data to the user.

[0596] Input: Illustrated data sent from the server.

[0597] Output: The flowchart that is displayed to the user.

[0598] Specific operation: The terminal displays the received flowchart data on the screen, allowing the user to visually check the factory work process.

[0599] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0600] The present invention relates to a system that converts information input via text or voice into a diagram format and adjusts the diagram by recognizing the user's emotions. This system provides a function that allows a user to input information using text or voice, analyzes the information, generates and displays an appropriate diagram, and also a function that recognizes the user's emotions and adjusts the layout and color of the diagram. Specific embodiments of the present invention are described below.

[0601] 1. Text and voice input methods

[0602] Users input information into the device either by text or voice. Text input is done using a keyboard or touchscreen, while voice input is done using a microphone or voice recognition software. For example, a user can type, "I need a Gantt chart showing the progress of a project."

[0603] 2. Transmission Method

[0604] The device preprocesses the input data and sends it to the server. In the case of voice input, the device uses voice recognition technology to convert the voice data into text data, which is then formatted into a standard format and sent to the server.

[0605] 3. Emotion recognition means

[0606] The server is equipped with an emotion engine for recognizing the user's emotion from the received voice data. This engine uses voice analysis technology to determine the emotion from the user's tone of voice and expressions. For example, it can recognize emotions such as nervousness, relaxation, anger, etc. from the user's tone of voice.

[0607] 4. Analysis method

[0608] After emotion recognition, the server uses natural language processing technology to analyze the text data. The analysis engine extracts key phrases and keywords to identify themes within the text. For example, keywords like "project progress" and "Gantt chart" may be extracted.

[0609] 5. Deciding on the illustration format

[0610] The server then determines the appropriate illustration format based on the analysis results. In this example, a Gantt chart is selected as the appropriate illustration format. At the same time, the layout and color of the illustration are adjusted based on the user's emotions. For example, if the user is relaxed, calm colors are used, and if the user is tense, a visually gentle color scheme is used.

[0611] 6. Means of obtaining additional information

[0612] To generate a Gantt chart, the server needs detailed information such as specific tasks, deadlines, progress, etc. If this additional information is missing, the server queries the user to obtain the necessary data. For example, the server might send a message to the user asking for the project start date, task details, and deadlines for each task.

[0613] 7. Illustration Generation Method

[0614] After retrieving the necessary data, the server uses a diagram generation engine to generate the appropriate diagram, which supports formats such as flowcharts, Gantt charts, graphs, and tables. For example, a Gantt chart showing each project task and its duration can be generated and adjusted according to the user's emotions.

[0615] 8. Illustrated Data Transmission Methods

[0616] The generated diagram is sent from the server to the device. The diagram data is sent in a format that the user can view and edit (e.g., image data, Excel file, Google Sheets URL, etc.).

[0617] 9. Display means

[0618] The device displays the diagram data received from the server to the user. The user can visually check the diagram on the device screen and edit it as needed. For example, a Gantt chart showing the progress of a project can be displayed on the screen, allowing the user to check the progress and deadlines of each task at a glance. In addition, layouts and colors are applied according to emotions, making it easier for users to use the diagrams.

[0619] In this way, the system of the present invention can analyze the information entered by the user in real time, convert it into an appropriate diagram format for display, and adjust the diagram taking into account the user's feelings, thereby enabling complex information to be visualized concisely and greatly improving communication efficiency.

[0620] The processing flow will be explained below.

[0621] Step 1:

[0622] The user inputs text or voice into the device. For example, the user inputs, "I need a Gantt chart showing the progress of a project."

[0623] Step 2:

[0624] The device receives input or speech. In the case of speech input, the device uses speech recognition software to convert the speech data into text.

[0625] Step 3:

[0626] The terminal preprocesses the text data and formats it into a standard format, then sends the formatted data to the server.

[0627] Step 4:

[0628] The server analyzes the received text data. Natural language processing techniques are used to extract key phrases and keywords and identify themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[0629] Step 5:

[0630] The server uses an emotion recognition engine to analyze emotions from the voice data sent by the user, for example, determining whether the user is nervous or relaxed from their voice tone.

[0631] Step 6:

[0632] The server determines the appropriate diagram format based on the analysis results. At the same time, it adjusts the layout and color of the diagram based on the user's emotions. For example, a Gantt chart is selected as the appropriate diagram format, and the color scheme is determined according to the user's emotions.

[0633] Step 7:

[0634] The server checks the details required to generate the Gantt chart. If the required information is missing, the server will ask the user questions such as "Please tell me the start date of the project and details of the tasks."

[0635] Step 8:

[0636] The user answers with additional information. For example, the user might say, "The project started on October 1, 2023."

[0637] Step 9:

[0638] The server receives additional information and adds it to the data used to generate the Gantt chart, such as project tasks, deadlines, and progress.

[0639] Step 10:

[0640] The server generates a Gantt chart using a diagram generation engine. The generated Gantt chart is converted into image data or an editable format within the server, and the layout and color are applied according to the user's emotions.

[0641] Step 11:

[0642] The server transmits the generated diagram data to the terminal in a format that the user can view and edit.

[0643] Step 12:

[0644] The device displays the received graphical data to the user. For example, a Gantt chart is displayed on the device screen, allowing the user to visually check the progress of the project. The layout and color are also applied according to the emotion.

[0645] Example 2

[0646] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0647] Conventional information input systems can properly analyze information entered via text or voice and convert it into diagram format, but they lack the ability to recognize the user's emotions and adjust the layout and color of the diagram, limiting their ability to improve the user experience. Furthermore, they lacked efficiency and flexibility when acquiring additional information and transmitting and displaying diagrams.

[0648] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an emotion recognition means, a means for analyzing data using natural language processing technology, and a means for adjusting the layout and color of the illustration based on the user's emotion. This makes it possible to generate an appropriate illustration format taking the user's emotion into consideration and to efficiently and flexibly obtain additional information the user needs.

[0649] "Text input means" refers to a means by which a user inputs text using a keyboard or a touch screen.

[0650] "Voice input means" refers to means for obtaining the user's voice as input data using a microphone or voice recognition software.

[0651] The "transmission means" is a means for preprocessing input data, converting it into a standard format, and transmitting it to the server.

[0652] "Analysis means" refers to means by which the server analyzes the received data and determines an appropriate graphical format.

[0653] The "emotion recognition means" is a means by which the server determines the user's emotions from voice and text expressions.

[0654] The "additional information acquisition means" is a means for the server to inquire of the user and acquire detailed information necessary for generating an illustration.

[0655] The "illustration generating means" is a means for generating an appropriate illustration using all collected data.

[0656] The "display means" is a means for displaying the graphic data received by the terminal to the user.

[0657] "Speech recognition technology" is a technology that converts input speech into text data.

[0658] "Natural language processing technology" is a technology that analyzes input text data and understands key phrases, keywords, and context.

[0659] "Illustration format" refers to a format that visually presents data, such as a Gantt chart, flow chart, graph, or table.

[0660] The "layout adjustment means" is a means for adjusting the layout of illustrations based on the user's feelings.

[0661] The "color adjustment means" is a means for adjusting the color scheme of the illustration based on the user's feelings.

[0662] The present invention relates to a system that converts information input via text or voice into an illustration format and adjusts the illustration by recognizing the user's emotions. This system is intended to be used by an unspecified number of users and aims to generate appropriate illustrations in real time. The following describes in detail the embodiments of the present invention.

[0663] Text and voice input methods

[0664] Users input information through the device, either through a keyboard or touchscreen for text input or through a microphone and voice recognition software for voice input. For example, a user might type, "I need a Gantt chart showing the progress of a project."

[0665] Sending data

[0666] The device preprocesses the data entered by the user, converts it into a standard format, and sends it to the server. In the case of voice input, the device uses voice recognition technology (e.g., Google voice recognition API) to convert the voice data into text data, formats it into JSON format, and sends it to the server.

[0667] emotion recognition means

[0668] The server processes the received data and recognizes the user's emotions using an emotion engine (e.g., IBM Watson Emotion Analysis), which determines the user's emotions from voice tone and text expressions, for example, identifying whether the user is tense, relaxed, angry, etc.

[0669] Data analysis

[0670] After emotion recognition, the server analyzes the input text data using natural language processing technology (e.g., OpenAI GPT-4). This analysis engine extracts key phrases and keywords from the text and identifies the subject and purpose of the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[0671] Deciding on an illustration format

[0672] The server determines the appropriate illustration format based on the analysis results and emotion recognition results. For example, a Gantt chart is deemed appropriate, and the layout and color of the illustration are also adjusted according to the user's emotional state. If the user is relaxed, calm colors are used, and if the user is tense, a color scheme that is less visually demanding is selected.

[0673] Obtaining additional information

[0674] The server collects detailed information (e.g., tasks, deadlines, and progress) required to generate the diagram. If the required information is missing, it prompts the user for additional information. For example, it sends questions such as, "What is the project start date, task details, and deadlines for each task?"

[0675] Generating illustrations

[0676] Using all the collected data, the server uses a diagram generation engine (e.g., D3.js or Matplotlib) to generate the final diagram, which can be adjusted in layout and color based on the user's preferences. For example, a Gantt chart showing each project task and its duration can be generated.

[0677] Illustrated data transmission and display

[0678] The generated diagram data is sent from the server to the terminal. This diagram data is provided in a format that the user can view and edit (e.g. PNG, Excel file, Google Sheets URL). The terminal displays the received diagram data to the user, allowing the user to visually check the contents of the diagram and edit it as necessary.

[0679] Example prompt sentences

[0680] For example, by inputting the following prompts into a generative AI model, it can generate diagrams that meet the user's requirements:

[0681] You want to create a Gantt chart that details the progress of a project. The project starts on October 1, 2023, and the major tasks include research, design, development, testing, and deployment. You want it to reflect the deadlines and progress of each task. Also, your current emotional state is relaxed. Choose a layout and color scheme that reflects this.

[0682] The above is a specific embodiment for carrying out the present invention. This system is capable of quickly and effectively converting user input information into illustrations and presenting the results in an optimal format.

[0683] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0684] Step 1: Enter your information

[0685] Users input information into the device either through text or voice. Text input is done using a keyboard or touchscreen. Voice input is done using a microphone or voice recognition software. The input data can be in text or voice data format. For example, a user might say, "I need a Gantt chart showing the progress of a project."

[0686] Step 2: Preprocess and send data

[0687] The device preprocesses the input data. In the case of voice data, it converts it into text data using voice recognition technology (e.g., Google voice recognition API). The converted text data is then formatted into a standard format (e.g., JSON) and sent to the server. Specifically, the device converts the voice into text, formats the text as JSON, and sends it to the server via an HTTP request.

[0688] Step 3: Emotion Recognition

[0689] The server receives the data sent from the device and analyzes the user's emotions using an emotion recognition engine (e.g., IBM Watson Emotion Analysis). The input data is analyzed based on voice tone and text expressions to determine emotions such as tension, relaxation, and anger. The emotion recognition engine analyzes the input data and returns the user's emotional state (e.g., tension, relaxation, anger) as its output.

[0690] Step 4: Data analysis

[0691] The server receives the emotion recognition results and then analyzes the input text data using natural language processing technology (e.g., OpenAI GPT-4). This analysis engine extracts key phrases and keywords from the text and identifies the subject and intent. For example, it extracts input keywords such as "project progress" and "Gantt chart." The subject and intent are provided as specific keywords as output.

[0692] Step 5: Decide on a graphical format

[0693] The server determines the appropriate diagram format based on the analysis results and emotion recognition results. For example, the analysis results may determine that a "Gantt chart" is appropriate. The layout and color are also adjusted taking into account the emotion recognition results. Specifically, if the user is relaxed, calm colors are selected, and if the user is tense, a color scheme that is less visually demanding is chosen. The output is the diagram format and settings for adjusting the layout.

[0694] Step 6: Obtain additional information

[0695] The server collects the necessary details (tasks, deadlines, progress, etc.) to generate the diagram. It sends a query message to the user for any missing information. For example, it sends specific questions such as "What is the project start date, task details, and deadlines for each task?" and waits for a response from the user. The input is the question from the server, and the output is the answer from the user.

[0696] Step 7: Generate the diagram

[0697] The server uses all the collected data to generate diagrams using a diagram generation engine (e.g., D3.js or Matplotlib). At this time, the diagram is generated with layout and color adjusted based on the results of emotion recognition. Specifically, a Gantt chart showing the duration of each task is generated. The input is the collected detailed information and adjustment settings, and the output is the generated diagram data.

[0698] Step 8: Submitting diagram data

[0699] The generated diagram data is sent from the server to the terminal. This diagram data is provided in a format that the user can view and edit (e.g. PNG, Excel file, Google Sheets URL). Specifically, the generated PNG file is sent to the terminal via an HTTP response. The generated diagram data is used as input, and transmission to the user's terminal is completed as output.

[0700] Step 9: Displaying the diagram

[0701] The terminal displays the diagram data received from the server to the user. The user can check the diagram on the terminal screen and edit it as necessary. For example, the received PNG file can be displayed on the screen to visually check the details and progress of each task. It is also possible to make additional edits and adjustments based on the contents of the diagram. The input is the diagram data received from the server, and the output is the display to the user.

[0702] (Application example 2)

[0703] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0704] Traditional brick-and-mortar stores have faced the challenge of quickly and accurately providing customers with the products and information they are looking for. Particularly in large stores or stores with complex layouts, customers often spend a lot of time trying to find the product they are looking for. Furthermore, providing information without considering the customer's emotions or state can sometimes detract from the customer experience. To solve these problems, a system was needed that could generate and display appropriate illustrations in real time based on the information entered by the customer.

[0705] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0706] In this invention, the server includes a text input means, a voice input means, a transmission means for processing the input data and transmitting it to the server, an analysis means for the server to analyze the data and determine an appropriate illustration format, a means for the server to acquire necessary additional information, a means for transmitting the generated illustration data to the terminal, a display means for displaying the illustration data received by the terminal, a means for a user wearing smart glasses to input store information by voice or text, and an emotion recognition means for recognizing the user's emotions and adjusting the layout and color of the illustrations. This allows customers to easily check the location of products and ask questions in the store, and also allows them to receive information presented according to their emotions and state.

[0707] "Text input means" refers to a device or method for a user to input text, including a keyboard and a touch screen.

[0708] "Voice input means" refers to a device or method that allows a user to input information by voice, and includes a microphone and voice recognition software.

[0709] "Transmission means" refers to a device or method for processing input data and transmitting it to a server, and includes a communication module and a data format conversion means.

[0710] "Analysis means" refers to a method or device for analyzing the data received by the server and determining an appropriate graphical format, including natural language processing techniques and keyword extraction algorithms.

[0711] "Means for obtaining additional information" refers to a device or method for querying a user or collecting additional data required for generating an illustration.

[0712] The "means for transmitting generated graphic data" refers to a device or method for transmitting graphic data generated by the server to the terminal.

[0713] "Display means" refers to a device or method for displaying the graphical data received by the terminal to the user, and includes a display and a VR device.

[0714] "Smart glasses" refers to a wearable device that has the ability to receive and display data.

[0715] "Emotion recognition means" refers to a method or device for analyzing and identifying a user's emotions from vocal tone and facial expressions, and includes emotion analysis engines and machine learning technologies.

[0716] "Illustrative formats" refers to formats that visually display data and include maps, flowcharts, graphs, tables, etc.

[0717] This invention provides a system that uses smart glasses to improve customer experience in brick-and-mortar stores. This system analyzes information entered by customers via voice or text in real time, and generates and displays appropriate illustrations. It also recognizes customer emotions and adjusts the layout and color of the illustrations accordingly.

[0718] Hardware and software used

[0719] Hardware:

[0720] Smart glasses (e.g. general-purpose wearable devices)

[0721] microphone

[0722] Display Device

[0723] software:

[0724] Natural language processing engine (e.g., cloud natural language processing API)

[0725] Speech recognition software (e.g., cloud speech recognition services)

[0726] Sentiment analysis engine (e.g., cloud sentiment analysis service)

[0727] Diagram generation engine (e.g., a web-based diagram generation library)

[0728] Data processing and calculation methods

[0729] 1. Data entry and submission:

[0730] The user inputs information through the smart glasses by voice or text. The input voice data is converted into text data using voice recognition software. The text data is then sent to the server.

[0731] 2. Emotion recognition:

[0732] The server analyzes the received voice data using an emotion analysis engine to identify the user's emotion, such as tension, relaxation, or anger, based on the user's tone of voice.

[0733] 3. Data Analysis:

[0734] The server analyzes the text data using a natural language processing engine, which extracts key phrases and keywords and identifies themes in the input information.

[0735] 4. Decide on the illustration format:

[0736] The server determines the appropriate illustration format based on the analysis results. It also adjusts the layout and color of the illustration based on the user's emotions. For example, if the user uses a relaxed voice tone, a map with calm colors will be generated.

[0737] 5. Obtaining additional information:

[0738] If necessary, the server queries the user for any missing additional information to complete the required data.

[0739] 6. Generating and sending diagrams:

[0740] The server uses an illustration generation engine to generate an appropriate illustration and transmits the data to the smart glasses, which then display the generated illustration on their display.

[0741] Specific examples

[0742] When a customer asks the smart glasses, "Where are the new dresses?", the voice data is sent to the server for analysis. The server extracts the keywords "new" and "dress" and generates an illustration showing the location of the appropriate category. If the server recognizes the customer's voice tone as relaxed, the illustration will be displayed in a calming blue color. The customer can then view the illustration through the smart glasses.

[0743] Prompt Sentence Examples

[0744] It analyzes voice data and illustrates the location of the product the customer is asking about.

[0745] Voice: "Where are the new dresses?"

[0746] Emotion: Relaxed tone of voice

[0747] Analyzed information: "New item", "Dress"

[0748] Generated illustration: store map, location of category "dresses", blue relaxed color scheme

[0749] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0750] Step 1:

[0751] A user wears smart glasses and inputs information by voice or text. The input data is captured from the smart glasses' microphone and keyboard. The input voice data is converted into text using voice recognition software.

[0752] Step 2:

[0753] The device preprocesses the input data and sends it to the cloud server, where it undergoes format conversion and optimization, converting voice data to text, and transmitting it according to the communication protocol. The input information is properly formatted and prepared for analysis on the server side.

[0754] Step 3:

[0755] The server analyzes the received text data using a natural language processing engine, which extracts key phrases and keywords and identifies the subject of the information entered. During this step, multiple filtering and analysis algorithms are applied to identify the category and subject of the information.

[0756] Step 4:

[0757] The server analyzes the received voice data using an emotion analysis engine to identify the user's emotions. The emotion is determined from the tone of the voice, speaking style, emphasis, etc., and emotional states such as relaxation, tension, joy, and anger are identified. The emotional data is used to adjust the layout and color of the illustrations.

[0758] Step 5:

[0759] Based on the analysis results, the server determines the appropriate illustration format, such as a map, flowchart, graph, or table, and then adjusts the layout and color scheme according to the user's emotions. For example, a relaxed user might be assigned a softer color scheme.

[0760] Step 6:

[0761] If the server does not have enough additional information, it will ask the user for a query. For example, if the question is "Where is the new dress?", the server will generate and send a query message to the user to ask for additional data such as the start date and location if more detailed product information is needed.

[0762] Step 7:

[0763] The server sends the generated illustration data to the smart glasses terminal, where it uses an illustration generation engine to generate a visually easy-to-understand illustration according to the selected format, and then the generated illustration data is sent from the cloud server to the smart glasses.

[0764] Step 8:

[0765] The smart glasses display the diagram data received from the server. Users can visually confirm the information through the glasses' display. The layout and color of the diagrams displayed are based on the analysis results and emotions, allowing users to comfortably view the information.

[0766] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0767] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0768] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0769] [Third embodiment]

[0770] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0771] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0772] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0773] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0774] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0775] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0776] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0777] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0778] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0779] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0780] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0781] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0782] The present invention relates to a system that converts information input by text or voice into a diagram format and displays it. This system provides a function in which a user inputs information using text or voice, analyzes the information, and generates and displays an appropriate diagram. Specific embodiments of the present invention are described below.

[0783] 1. Text and voice input methods

[0784] Users input information into the device either by text or voice. Text input is done using a keyboard or touchscreen, while voice input is done using a microphone or voice recognition software. For example, a user can type, "I need a Gantt chart showing the progress of a project."

[0785] 2. Transmission Method

[0786] The device preprocesses the input data and sends it to the server. In the case of voice input, the device uses voice recognition technology to convert the voice data into text data, which is then formatted into a standard format and sent to the server.

[0787] 3. Analysis method

[0788] The server uses natural language processing technology to analyze the text data sent. The analysis engine extracts key phrases and keywords and identifies themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted, and based on this, the text is determined to be information about project management.

[0789] 4. Means of obtaining additional information

[0790] To generate a Gantt chart, the server needs detailed information such as specific tasks, deadlines, and progress. If this additional information is missing, the server queries the user to obtain the necessary data. For example, the server might send a message to the user requesting the "project start date," "task details," and "deadline for each task."

[0791] 5. Illustration Generation Method

[0792] After retrieving the necessary data, the server uses a diagram generation engine to generate the appropriate diagram, which supports formats such as flowcharts, Gantt charts, graphs, tables, etc. For example, a Gantt chart showing each task in a project and its duration can be generated.

[0793] 6. Illustrated Data Transmission Methods

[0794] The generated diagram is sent from the server to the device. The diagram data is sent in a format that the user can view and edit (e.g., image data, Excel file, Google Sheets URL, etc.).

[0795] 7. Display means

[0796] The device receives the diagram data from the server and displays it to the user. The user can visually check the diagram on the device screen and edit it as needed. For example, a Gantt chart showing the progress of a project can be displayed on the screen, allowing the user to check the progress and deadlines of each task at a glance.

[0797] In this way, the system of the present invention can analyze the information entered by the user in real time and convert it into an appropriate diagram format for display, thereby enabling complex information to be visualized concisely and greatly improving communication efficiency.

[0798] The processing flow will be explained below.

[0799] Step 1:

[0800] The user inputs text or voice into the device. For example, the user inputs, "I need a Gantt chart showing the progress of a project."

[0801] Step 2:

[0802] The device receives input or speech. In the case of speech input, the device uses speech recognition software to convert the speech data into text.

[0803] Step 3:

[0804] The terminal preprocesses the text data and formats it into a standard format, then sends the formatted data to the server.

[0805] Step 4:

[0806] The server analyzes the received text data. Natural language processing techniques are used to extract key phrases and keywords and identify themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[0807] Step 5:

[0808] The server then determines the appropriate diagram format based on the analysis results. In this example, a Gantt chart is selected as the appropriate diagram format.

[0809] Step 6:

[0810] The server checks the details needed to generate the Gantt chart. If the required information is missing, the server prompts the user for additional information. For example, "What is the start date of the project?"

[0811] Step 7:

[0812] The user answers with additional information. For example, the user answers, "The project started on October 1, 2023."

[0813] Step 8:

[0814] The server receives additional information and adds it to the data used to generate the Gantt chart, such as project tasks, deadlines, and progress.

[0815] Step 9:

[0816] The server generates a Gantt chart using a diagram generation engine, and the generated Gantt chart is converted into image data or an editable format within the server.

[0817] Step 10:

[0818] The server transmits the generated diagram data to the terminal in a format that the user can view and edit.

[0819] Step 11:

[0820] The terminal displays the received graphical data to the user, for example, a Gantt chart is displayed on the terminal screen, allowing the user to visually check the progress of the project.

[0821] Step 12:

[0822] Users can check the displayed diagrams and edit them as necessary, which allows users to communicate more efficiently.

[0823] Example 1

[0824] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0825] Conventional information input systems have difficulty properly analyzing information entered via text or voice and converting it into a visually easy-to-understand diagram format. In particular, when the user's input information is insufficient, there is a lack of a means to effectively acquire additional information and generate a diagram, which reduces user convenience. Furthermore, the inability to update or edit diagram data in real time makes it difficult to make decisions based on the latest information.

[0826] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0827] In this invention, the server includes a means for converting input data into text using speech recognition technology, a means for analyzing the data using natural language processing technology, and a means for updating the generated illustrations in real time. This allows the information input by the user via text or voice to be properly analyzed, and missing information to be effectively acquired and an illustration to be generated. Furthermore, editing and real-time updating of the generated illustration data allows for efficient decision-making based on the latest information.

[0828] A "text input means" is a device and method that allows a user to input text information using a keyboard or touch screen.

[0829] "Voice input means" refers to a device and method including a microphone and voice recognition software for inputting user voice information and processing it as digital data.

[0830] The "transmitting means" refers to a device and method including a network interface and communication protocol for preprocessing input data and transmitting it to a server.

[0831] "Analysis means" refers to software and algorithms that the server uses to analyze the received data using natural language processing techniques and determine an appropriate graphical format.

[0832] The "acquisition means" refers to a device and method for acquiring additional information required by the server to generate the illustration from the user through a query.

[0833] "Illustration generation means" refers to software and algorithms for generating appropriate illustrations (Gantt charts, flow charts, graphs, tables) based on collected data.

[0834] The "transmitting means" refers to an apparatus and method including a network interface and a communication protocol for the server to transmit the generated illustration data to the terminal.

[0835] "Display means" refers to a display and software for visually displaying the graphical data received by the terminal.

[0836] The "query means" refers to a device and method by which the server queries the user for necessary information when additional information is missing.

[0837] "Speech recognition technology" refers to software and algorithms used to convert voice data into text data.

[0838] "Natural language processing technology" refers to software and algorithms that analyze text data, extract key phrases and keywords, and perform semantic analysis.

[0839] "Real-time update means" refers to a device and method for instantly updating generated graphical data based on the latest information.

[0840] The "editing means" refers to software and an interface that allows the user to edit the received graphic data on the terminal.

[0841] The system of the present invention is a system that can convert information input by a user in text or voice into a diagram format and visually display it. To implement this system, the following hardware and software are used.

[0842] Hardware and Software

[0843] Enter information

[0844] Users can input text using a keyboard or touchscreen, such as "I need a Gantt chart showing the progress of a project." A microphone is also used for voice input, which is then converted into text using speech recognition software, such as the Google Speech-to-Text API.

[0845] Data transmission

[0846] The terminal formats the input text data into a standard format and sends it to the server, for example, in JSON format, using an HTTP POST request.

[0847] Data analysis

[0848] The server analyzes the received text data using natural language processing (NLP) techniques. Analysis engines such as SpaCy and the BERT model are used. This allows the server to extract key phrases and keywords and identify themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[0849] Obtaining additional information

[0850] The server queries the user to obtain additional information needed to generate the diagram. This information is obtained via HTTP requests or message transmission. The user enters the requested information into the terminal and resubmits it. For example, the user enters specific details such as "The project start date is October 1, 2023, Task 1 will be planning until October 5, and Task 2 will be design until October 15."

[0851] Illustration generation

[0852] After the server collects all the necessary data, it uses a diagram generation engine, such as D3.js or Syncfusion, to generate diagrams, such as Gantt charts, that show each project task and its duration.

[0853] Sending diagram data

[0854] The generated diagram data is sent from the server to the terminal via a network interface or communication protocol. The diagram data is often sent in image format or an editable format.

[0855] Data Display

[0856] The device then displays the received diagram data to the user. A display and corresponding software are used as the display method. The user can check the diagram on the device screen and edit it as needed. For example, a Gantt chart showing the project progress can be displayed, allowing the progress and deadlines of each task to be confirmed at a glance.

[0857] Specific examples

[0858] Consider the example where a user types into a terminal, "Show me the agenda for my next meeting in a flowchart." The following steps occur:

[0859] 1. The user enters text into the terminal.

[0860] 2. The terminal converts the input text data into a standard format and sends it to the server.

[0861] 3. The server analyzes the data using natural language processing technology and extracts the keywords "meeting agenda" and "flowchart."

[0862] 4. The server queries the user for the missing information (specific agenda items).

[0863] 5. The user enters the specific agenda item into the terminal and submits it.

[0864] 6. The server generates a flowchart based on the collected information.

[0865] 7. The generated flowchart is sent to the terminal and displayed.

[0866] 8. The user can check the displayed flowchart and edit it if necessary.

[0867] This system allows users to easily visualize complex information and communicate effectively based on it.

[0868] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0869] Step 1:

[0870] The user inputs information into the terminal by text or voice.

[0871] Specifically, the user inputs text information using a keyboard or touch screen, or voice information using a microphone. If the input information is text information such as "I need a Gantt chart showing the progress of the project," the device receives it as is. If the input information is voice information, it is received by the microphone and converted into text using voice recognition technology in the next step.

[0872] Step 2:

[0873] The terminal uses voice recognition software to convert the voice data into text data.

[0874] Specifically, speech recognition software (e.g., Google Speech-to-Text API) is used to convert voice data into text data in real time. This converted text data is used in the next step. The output is sentence text data.

[0875] Step 3:

[0876] The terminal formats the input text data into a standard format (e.g., JSON) and sends it to the server.

[0877] Specifically, the input text data is formatted into JSON format and sent to the server using an HTTP POST request. For example, the formatted JSON data looks like this:

[0878] json

[0879] {

[0880] "request": "I need a Gantt chart to show the progress of my project"

[0881] }

[0882] The output is JSON data that is sent to the server.

[0883] Step 4:

[0884] The server analyzes the received text data using natural language processing (NLP) techniques.

[0885] Specifically, the server uses an NLP engine (e.g., SpaCy or a BERT model) to extract key phrases and keywords from the received text data. For example, keywords such as "project progress" and "Gantt chart" are extracted, and the theme of the text is identified as "project management." The output is the analyzed keywords and theme information.

[0886] Step 5:

[0887] The server queries the user to obtain additional information needed to generate the illustration.

[0888] Specifically, the server identifies the missing information and generates a message to inquire about it to the user. For example, it generates a message asking for the "start date of the project," "details of the tasks," and "deadlines of each task," and sends it to the terminal. The output is a query message.

[0889] Step 6:

[0890] The user receives the inquiry message from the server at the terminal, enters additional information, and resends it.

[0891] Specifically, the user inputs the necessary information according to the inquiry displayed on the terminal. Specific details such as "The project start date is October 1, 2023, Task 1 will be planning until October 5, and Task 2 will be design until October 15" are entered, and the terminal sends this to the server. The output is the additional information entered by the user.

[0892] Step 7:

[0893] The server uses a diagram generation engine to generate diagrams based on all the collected information.

[0894] Specifically, the server runs a diagram generation engine (e.g., D3.js or Syncfusion) to generate an appropriate diagram format (e.g., Gantt chart) based on the collected information. For example, a Gantt chart is generated that visually shows each task and its duration. The output is the generated diagram data.

[0895] Step 8:

[0896] The server transmits the generated illustration data to the terminal.

[0897] Specifically, the generated diagram data is sent to the terminal in an image format (e.g., PNG or JPEG) or in an editable format (e.g., an Excel file or a Google Sheets URL). The output is the diagram data sent to the terminal.

[0898] Step 9:

[0899] The terminal displays the received diagram data to the user and edits it as necessary.

[0900] Specifically, the received diagram data is displayed on the terminal display, and the user can visually check it. For example, a Gantt chart showing the progress of a project is displayed on the screen, and the user can check the progress and deadlines of each task and make edits as necessary. The output is the visually displayed diagram data.

[0901] This series of processing steps allows users to quickly and effectively generate and use visual diagrams based on the information they enter.

[0902] (Application example 1)

[0903] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0904] Conventional factory work process management faces the problem of making it difficult to visually grasp complex processes and vast amounts of data. Effective visualization is essential to efficiently manage work processes and enable workers and managers to quickly understand the processes and issue appropriate instructions. Furthermore, process data exists in different formats, and integrating and visualizing them is time-consuming. To solve these issues, a system is needed that uses voice and text input to automatically analyze work processes and convert them into an appropriate diagram format, such as a flowchart.

[0905] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0906] In this invention, the server includes a text input means, a voice input means, a transmission means for processing the input data and transmitting it to the server, an analysis means for the server to analyze the data and determine an appropriate diagram format, a means for the server to acquire necessary additional information, a means for transmitting the generated diagram data to the terminal, a display means for displaying the diagram data received by the terminal, an input information analysis means for managing the work process in the factory, and a flowchart generation means for visualizing the work process based on the analysis results. This makes it possible to efficiently manage complex work processes in a factory and to grasp the processes in a form that is visually easy for workers and managers to understand.

[0907] "Text input means" refers to a means for a user to input text using a keyboard, touch screen, or the like.

[0908] "Voice input means" refers to a means for a user to input voice using a microphone or voice recognition software.

[0909] The "transmission means" is a means for processing input data and transmitting it to the server.

[0910] "Analysis means" means by which the server analyzes the data and determines the appropriate graphical format.

[0911] The "additional information acquisition means" is a means for the server to acquire the necessary additional information.

[0912] The "graphic data transmission means" is a means for transmitting the generated graphic data to the terminal.

[0913] The "display means" is a means for displaying the graphic data received by the terminal to the user.

[0914] The "input information analysis means" is a means for analyzing input information in order to manage the work process in the factory.

[0915] The "flowchart generation means" is a means for generating a flow chart to visualize the work process based on the analysis results.

[0916] The present invention is a system that analyzes information input from voice or text, and automatically generates and displays visual diagrams such as flowcharts based on that information, in order to facilitate the management of factory work processes. Specific embodiments of the system are described below.

[0917] 1. Text and voice input methods

[0918] The user can input text using a keyboard or touchscreen. They can also input speech through a microphone. Speech input uses speech recognition software to convert the speech data into text. At this stage, the user might input a prompt such as, "First, prepare the materials, then move on to the assembly process, and finally perform quality checks."

[0919] 2. Transmission Method

[0920] The device preprocesses the input data and sends it to the server. In particular, in the case of voice input, the voice data is converted into text data using voice recognition technology. The converted text data is then formatted into a standard format and sent to the server.

[0921] 3. Analysis method

[0922] The server uses natural language processing technology to analyze the text data sent. During this process, key phrases and keywords are extracted using an analysis model using the transformers library. For example, keywords such as "material preparation," "assembly process," and "quality check" are extracted.

[0923] 4. Means of obtaining additional information

[0924] Based on the analysis results, the server queries the user for any missing details. This query is in the form of a message, and can obtain additional information from the user. For example, the server may need to know the detailed work content of each process or the time required for each process.

[0925] 5. Flowchart generation method

[0926] After obtaining the necessary additional information, the server generates a flowchart or Gantt chart. The networkx and matplotlib libraries are used to visualize the information. For example, a flowchart from "material preparation" to "assembly process" and "quality check" is generated.

[0927] 6. Illustrated Data Transmission Methods

[0928] The generated diagrams are sent from the server to the terminal and provided in a format that can be viewed and edited by the user.

[0929] 7. Display means

[0930] The terminal displays the diagram data received from the server to the user. By checking the displayed flowcharts and Gantt charts, the user can easily visually understand the factory work processes. The user can also edit the data as needed.

[0931] In this way, the system of the present invention efficiently manages the work processes in a factory, allowing workers and managers to grasp the processes in a visually easy-to-understand format.

[0932] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0933] Step 1:

[0934] The user inputs information into the terminal by text or voice.

[0935] Input: The user types text using a keyboard or touchscreen, or speaks using a microphone.

[0936] Output: The input text data or audio data.

[0937] Specific operation: The user inputs the prompt sentence: "First, prepare the materials, then move on to the assembly process. Finally, perform quality check."

[0938] Step 2:

[0939] The terminal converts the voice data into text data.

[0940] Input: Audio data.

[0941] Output: Text data.

[0942] Specific operation: The device uses voice recognition technology to convert voice data into text, using the speech_recognition library.

[0943] Step 3:

[0944] The terminal preprocesses the text data and sends it to the server.

[0945] Input: Text data.

[0946] Output: Preprocessed text data.

[0947] Specific operation: The terminal performs formatting processing on the text data, converts it into a standard format, and then sends the data to the server.

[0948] Step 4:

[0949] The server analyzes the data using natural language processing technology.

[0950] Input: Preprocessed text data.

[0951] Output: Keyphrases or keywords.

[0952] Specific operation: The server uses the transformers library to extract key phrases and keywords such as "material preparation," "assembly process," and "quality check" from the text data.

[0953] Step 5:

[0954] The server prompts the user for any additional information that is missing.

[0955] Input: Keywords obtained from the analysis results.

[0956] Output: Additional information from the user.

[0957] Specific operation: The server queries the user for missing information in the form of a message, for example, to confirm "detailed work content for each process" and "time required for each process."

[0958] Step 6:

[0959] The server generates the flowchart.

[0960] Input: Additional information collected from the user and analysis results.

[0961] Output: Flowchart data.

[0962] What it does: The server uses the networkx and matplotlib libraries to generate a flowchart based on the information it collects.

[0963] Step 7:

[0964] The server transmits the generated flowchart data to the terminal.

[0965] Input: Flowchart data.

[0966] Output: Illustrated data sent to the terminal.

[0967] Specific operation: The server sends the flowchart data to the terminal and provides it in a format (e.g., image data) that the user can view and edit.

[0968] Step 8:

[0969] The terminal displays the received flowchart data to the user.

[0970] Input: Illustrated data sent from the server.

[0971] Output: The flowchart that is displayed to the user.

[0972] Specific operation: The terminal displays the received flowchart data on the screen, allowing the user to visually check the factory work process.

[0973] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0974] The present invention relates to a system that converts information input via text or voice into a diagram format and adjusts the diagram by recognizing the user's emotions. This system provides a function that allows a user to input information using text or voice, analyzes the information, generates and displays an appropriate diagram, and also a function that recognizes the user's emotions and adjusts the layout and color of the diagram. Specific embodiments of the present invention are described below.

[0975] 1. Text and voice input methods

[0976] Users input information into the device either by text or voice. Text input is done using a keyboard or touchscreen, while voice input is done using a microphone or voice recognition software. For example, a user can type, "I need a Gantt chart showing the progress of a project."

[0977] 2. Transmission Method

[0978] The device preprocesses the input data and sends it to the server. In the case of voice input, the device uses voice recognition technology to convert the voice data into text data, which is then formatted into a standard format and sent to the server.

[0979] 3. Emotion recognition means

[0980] The server is equipped with an emotion engine for recognizing the user's emotion from the received voice data. This engine uses voice analysis technology to determine the emotion from the user's tone of voice and expressions. For example, it can recognize emotions such as nervousness, relaxation, anger, etc. from the user's tone of voice.

[0981] 4. Analysis method

[0982] After emotion recognition, the server uses natural language processing technology to analyze the text data. The analysis engine extracts key phrases and keywords to identify themes within the text. For example, keywords like "project progress" and "Gantt chart" may be extracted.

[0983] 5. Deciding on the illustration format

[0984] The server then determines the appropriate illustration format based on the analysis results. In this example, a Gantt chart is selected as the appropriate illustration format. At the same time, the layout and color of the illustration are adjusted based on the user's emotions. For example, if the user is relaxed, calm colors are used, and if the user is tense, a visually gentle color scheme is used.

[0985] 6. Means of obtaining additional information

[0986] To generate a Gantt chart, the server needs detailed information such as specific tasks, deadlines, progress, etc. If this additional information is missing, the server queries the user to obtain the necessary data. For example, the server might send a message to the user asking for the project start date, task details, and deadlines for each task.

[0987] 7. Illustration Generation Method

[0988] After retrieving the necessary data, the server uses a diagram generation engine to generate the appropriate diagram, which supports formats such as flowcharts, Gantt charts, graphs, and tables. For example, a Gantt chart showing each project task and its duration can be generated and adjusted according to the user's emotions.

[0989] 8. Illustrated Data Transmission Methods

[0990] The generated diagram is sent from the server to the device. The diagram data is sent in a format that the user can view and edit (e.g., image data, Excel file, Google Sheets URL, etc.).

[0991] 9. Display means

[0992] The device displays the diagram data received from the server to the user. The user can visually check the diagram on the device screen and edit it as needed. For example, a Gantt chart showing the progress of a project can be displayed on the screen, allowing the user to check the progress and deadlines of each task at a glance. In addition, layouts and colors are applied according to emotions, making it easier for users to use the diagrams.

[0993] In this way, the system of the present invention can analyze the information entered by the user in real time, convert it into an appropriate diagram format for display, and adjust the diagram taking into account the user's feelings, thereby enabling complex information to be visualized concisely and greatly improving communication efficiency.

[0994] The processing flow will be explained below.

[0995] Step 1:

[0996] The user inputs text or voice into the device. For example, the user inputs, "I need a Gantt chart showing the progress of a project."

[0997] Step 2:

[0998] The device receives input or speech. In the case of speech input, the device uses speech recognition software to convert the speech data into text.

[0999] Step 3:

[1000] The terminal preprocesses the text data and formats it into a standard format, then sends the formatted data to the server.

[1001] Step 4:

[1002] The server analyzes the received text data. Natural language processing techniques are used to extract key phrases and keywords and identify themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[1003] Step 5:

[1004] The server uses an emotion recognition engine to analyze emotions from the voice data sent by the user, for example, determining whether the user is nervous or relaxed from their voice tone.

[1005] Step 6:

[1006] The server determines the appropriate diagram format based on the analysis results. At the same time, it adjusts the layout and color of the diagram based on the user's emotions. For example, a Gantt chart is selected as the appropriate diagram format, and the color scheme is determined according to the user's emotions.

[1007] Step 7:

[1008] The server checks the details required to generate the Gantt chart. If the required information is missing, the server will ask the user questions such as "Please tell me the start date of the project and details of the tasks."

[1009] Step 8:

[1010] The user answers with additional information. For example, the user might say, "The project started on October 1, 2023."

[1011] Step 9:

[1012] The server receives additional information and adds it to the data used to generate the Gantt chart, such as project tasks, deadlines, and progress.

[1013] Step 10:

[1014] The server generates a Gantt chart using a diagram generation engine. The generated Gantt chart is converted into image data or an editable format within the server, and the layout and color are applied according to the user's emotions.

[1015] Step 11:

[1016] The server transmits the generated diagram data to the terminal in a format that the user can view and edit.

[1017] Step 12:

[1018] The device displays the received graphical data to the user. For example, a Gantt chart is displayed on the device screen, allowing the user to visually check the progress of the project. The layout and color are also applied according to the emotion.

[1019] Example 2

[1020] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1021] Conventional information input systems can properly analyze information entered via text or voice and convert it into diagram format, but they lack the ability to recognize the user's emotions and adjust the layout and color of the diagram, limiting their ability to improve the user experience. Furthermore, they lacked efficiency and flexibility when acquiring additional information and transmitting and displaying diagrams.

[1022] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an emotion recognition means, a means for analyzing data using natural language processing technology, and a means for adjusting the layout and color of the illustration based on the user's emotion. This makes it possible to generate an appropriate illustration format taking the user's emotion into consideration and to efficiently and flexibly obtain additional information the user needs.

[1023] "Text input means" refers to a means by which a user inputs text using a keyboard or a touch screen.

[1024] "Voice input means" refers to means for obtaining the user's voice as input data using a microphone or voice recognition software.

[1025] The "transmission means" is a means for preprocessing input data, converting it into a standard format, and transmitting it to the server.

[1026] "Analysis means" refers to means by which the server analyzes the received data and determines an appropriate graphical format.

[1027] The "emotion recognition means" is a means by which the server determines the user's emotions from voice and text expressions.

[1028] The "additional information acquisition means" is a means for the server to inquire of the user and acquire detailed information necessary for generating an illustration.

[1029] The "illustration generating means" is a means for generating an appropriate illustration using all collected data.

[1030] The "display means" is a means for displaying the graphic data received by the terminal to the user.

[1031] "Speech recognition technology" is a technology that converts input speech into text data.

[1032] "Natural language processing technology" is a technology that analyzes input text data and understands key phrases, keywords, and context.

[1033] "Illustration format" refers to a format that visually presents data, such as a Gantt chart, flow chart, graph, or table.

[1034] The "layout adjustment means" is a means for adjusting the layout of illustrations based on the user's feelings.

[1035] The "color adjustment means" is a means for adjusting the color scheme of the illustration based on the user's feelings.

[1036] The present invention relates to a system that converts information input via text or voice into an illustration format and adjusts the illustration by recognizing the user's emotions. This system is intended to be used by an unspecified number of users and aims to generate appropriate illustrations in real time. The following describes in detail the embodiments of the present invention.

[1037] Text and voice input methods

[1038] Users input information through the device, either through a keyboard or touchscreen for text input or through a microphone and voice recognition software for voice input. For example, a user might type, "I need a Gantt chart showing the progress of a project."

[1039] Sending data

[1040] The device preprocesses the data entered by the user, converts it into a standard format, and sends it to the server. In the case of voice input, the device uses voice recognition technology (e.g., Google voice recognition API) to convert the voice data into text data, formats it into JSON format, and sends it to the server.

[1041] emotion recognition means

[1042] The server processes the received data and recognizes the user's emotions using an emotion engine (e.g., IBM Watson Emotion Analysis), which determines the user's emotions from voice tone and text expressions, for example, identifying whether the user is tense, relaxed, angry, etc.

[1043] Data analysis

[1044] After emotion recognition, the server analyzes the input text data using natural language processing technology (e.g., OpenAI GPT-4). This analysis engine extracts key phrases and keywords from the text and identifies the subject and purpose of the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[1045] Deciding on an illustration format

[1046] The server determines the appropriate illustration format based on the analysis results and emotion recognition results. For example, a Gantt chart is deemed appropriate, and the layout and color of the illustration are also adjusted according to the user's emotional state. If the user is relaxed, calm colors are used, and if the user is tense, a color scheme that is less visually demanding is selected.

[1047] Obtaining additional information

[1048] The server collects detailed information (e.g., tasks, deadlines, and progress) required to generate the diagram. If the required information is missing, it prompts the user for additional information. For example, it sends questions such as, "What is the project start date, task details, and deadlines for each task?"

[1049] Generating illustrations

[1050] Using all the collected data, the server uses a diagram generation engine (e.g., D3.js or Matplotlib) to generate the final diagram, which can be adjusted in layout and color based on the user's preferences. For example, a Gantt chart showing each project task and its duration can be generated.

[1051] Illustrated data transmission and display

[1052] The generated diagram data is sent from the server to the terminal. This diagram data is provided in a format that the user can view and edit (e.g. PNG, Excel file, Google Sheets URL). The terminal displays the received diagram data to the user, allowing the user to visually check the contents of the diagram and edit it as necessary.

[1053] Example prompt sentences

[1054] For example, by inputting the following prompts into a generative AI model, it can generate diagrams that meet the user's requirements:

[1055] You want to create a Gantt chart that details the progress of a project. The project starts on October 1, 2023, and the major tasks include research, design, development, testing, and deployment. You want it to reflect the deadlines and progress of each task. Also, your current emotional state is relaxed. Choose a layout and color scheme that reflects this.

[1056] The above is a specific embodiment for carrying out the present invention. This system is capable of quickly and effectively converting user input information into illustrations and presenting the results in an optimal format.

[1057] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1058] Step 1: Enter your information

[1059] Users input information into the device either through text or voice. Text input is done using a keyboard or touchscreen. Voice input is done using a microphone or voice recognition software. The input data can be in text or voice data format. For example, a user might say, "I need a Gantt chart showing the progress of a project."

[1060] Step 2: Preprocess and send data

[1061] The device preprocesses the input data. In the case of voice data, it converts it into text data using voice recognition technology (e.g., Google voice recognition API). The converted text data is then formatted into a standard format (e.g., JSON) and sent to the server. Specifically, the device converts the voice into text, formats the text as JSON, and sends it to the server via an HTTP request.

[1062] Step 3: Emotion Recognition

[1063] The server receives the data sent from the device and analyzes the user's emotions using an emotion recognition engine (e.g., IBM Watson Emotion Analysis). The input data is analyzed based on voice tone and text expressions to determine emotions such as tension, relaxation, and anger. The emotion recognition engine analyzes the input data and returns the user's emotional state (e.g., tension, relaxation, anger) as its output.

[1064] Step 4: Data analysis

[1065] The server receives the emotion recognition results and then analyzes the input text data using natural language processing technology (e.g., OpenAI GPT-4). This analysis engine extracts key phrases and keywords from the text and identifies the subject and intent. For example, it extracts input keywords such as "project progress" and "Gantt chart." The subject and intent are provided as specific keywords as output.

[1066] Step 5: Decide on a graphical format

[1067] The server determines the appropriate diagram format based on the analysis results and emotion recognition results. For example, the analysis results may determine that a "Gantt chart" is appropriate. The layout and color are also adjusted taking into account the emotion recognition results. Specifically, if the user is relaxed, calm colors are selected, and if the user is tense, a color scheme that is less visually demanding is chosen. The output is the diagram format and settings for adjusting the layout.

[1068] Step 6: Obtain additional information

[1069] The server collects the necessary details (tasks, deadlines, progress, etc.) to generate the diagram. It sends a query message to the user for any missing information. For example, it sends specific questions such as "What is the project start date, task details, and deadlines for each task?" and waits for a response from the user. The input is the question from the server, and the output is the answer from the user.

[1070] Step 7: Generate the diagram

[1071] The server uses all the collected data to generate diagrams using a diagram generation engine (e.g., D3.js or Matplotlib). At this time, the diagram is generated with layout and color adjusted based on the results of emotion recognition. Specifically, a Gantt chart showing the duration of each task is generated. The input is the collected detailed information and adjustment settings, and the output is the generated diagram data.

[1072] Step 8: Submitting diagram data

[1073] The generated diagram data is sent from the server to the terminal. This diagram data is provided in a format that the user can view and edit (e.g. PNG, Excel file, Google Sheets URL). Specifically, the generated PNG file is sent to the terminal via an HTTP response. The generated diagram data is used as input, and transmission to the user's terminal is completed as output.

[1074] Step 9: Displaying the diagram

[1075] The terminal displays the diagram data received from the server to the user. The user can check the diagram on the terminal screen and edit it as necessary. For example, the received PNG file can be displayed on the screen to visually check the details and progress of each task. It is also possible to make additional edits and adjustments based on the contents of the diagram. The input is the diagram data received from the server, and the output is the display to the user.

[1076] (Application example 2)

[1077] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1078] Traditional brick-and-mortar stores have faced the challenge of quickly and accurately providing customers with the products and information they are looking for. Particularly in large stores or stores with complex layouts, customers often spend a lot of time trying to find the product they are looking for. Furthermore, providing information without considering the customer's emotions or state can sometimes detract from the customer experience. To solve these problems, a system was needed that could generate and display appropriate illustrations in real time based on the information entered by the customer.

[1079] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1080] In this invention, the server includes a text input means, a voice input means, a transmission means for processing the input data and transmitting it to the server, an analysis means for the server to analyze the data and determine an appropriate illustration format, a means for the server to acquire necessary additional information, a means for transmitting the generated illustration data to the terminal, a display means for displaying the illustration data received by the terminal, a means for a user wearing smart glasses to input store information by voice or text, and an emotion recognition means for recognizing the user's emotions and adjusting the layout and color of the illustrations. This allows customers to easily check the location of products and ask questions in the store, and also allows them to receive information presented according to their emotions and state.

[1081] "Text input means" refers to a device or method for a user to input text, including a keyboard and a touch screen.

[1082] "Voice input means" refers to a device or method that allows a user to input information by voice, and includes a microphone and voice recognition software.

[1083] "Transmission means" refers to a device or method for processing input data and transmitting it to a server, and includes a communication module and a data format conversion means.

[1084] "Analysis means" refers to a method or device for analyzing the data received by the server and determining an appropriate graphical format, including natural language processing techniques and keyword extraction algorithms.

[1085] "Means for obtaining additional information" refers to a device or method for querying a user or collecting additional data required for generating an illustration.

[1086] The "means for transmitting generated graphic data" refers to a device or method for transmitting graphic data generated by the server to the terminal.

[1087] "Display means" refers to a device or method for displaying the graphical data received by the terminal to the user, and includes a display and a VR device.

[1088] "Smart glasses" refers to a wearable device that has the ability to receive and display data.

[1089] "Emotion recognition means" refers to a method or device for analyzing and identifying a user's emotions from vocal tone and facial expressions, and includes emotion analysis engines and machine learning technologies.

[1090] "Illustrative formats" refers to formats that visually display data and include maps, flowcharts, graphs, tables, etc.

[1091] This invention provides a system that uses smart glasses to improve customer experience in brick-and-mortar stores. This system analyzes information entered by customers via voice or text in real time, and generates and displays appropriate illustrations. It also recognizes customer emotions and adjusts the layout and color of the illustrations accordingly.

[1092] Hardware and software used

[1093] Hardware:

[1094] Smart glasses (e.g. general-purpose wearable devices)

[1095] microphone

[1096] Display Device

[1097] software:

[1098] Natural language processing engine (e.g., cloud natural language processing API)

[1099] Speech recognition software (e.g., cloud speech recognition services)

[1100] Sentiment analysis engine (e.g., cloud sentiment analysis service)

[1101] Diagram generation engine (e.g., a web-based diagram generation library)

[1102] Data processing and calculation methods

[1103] 1. Data entry and submission:

[1104] The user inputs information through the smart glasses by voice or text. The input voice data is converted into text data using voice recognition software. The text data is then sent to the server.

[1105] 2. Emotion recognition:

[1106] The server analyzes the received voice data using an emotion analysis engine to identify the user's emotion, such as tension, relaxation, or anger, based on the user's tone of voice.

[1107] 3. Data Analysis:

[1108] The server analyzes the text data using a natural language processing engine, which extracts key phrases and keywords and identifies themes in the input information.

[1109] 4. Decide on the illustration format:

[1110] The server determines the appropriate illustration format based on the analysis results. It also adjusts the layout and color of the illustration based on the user's emotions. For example, if the user uses a relaxed voice tone, a map with calm colors will be generated.

[1111] 5. Obtaining additional information:

[1112] If necessary, the server queries the user for any missing additional information to complete the required data.

[1113] 6. Generating and sending diagrams:

[1114] The server uses an illustration generation engine to generate an appropriate illustration and transmits the data to the smart glasses, which then display the generated illustration on their display.

[1115] Specific examples

[1116] When a customer asks the smart glasses, "Where are the new dresses?", the voice data is sent to the server for analysis. The server extracts the keywords "new" and "dress" and generates an illustration showing the location of the appropriate category. If the server recognizes the customer's voice tone as relaxed, the illustration will be displayed in a calming blue color. The customer can then view the illustration through the smart glasses.

[1117] Prompt Sentence Examples

[1118] It analyzes voice data and illustrates the location of the product the customer is asking about.

[1119] Voice: "Where are the new dresses?"

[1120] Emotion: Relaxed tone of voice

[1121] Analyzed information: "New item", "Dress"

[1122] Generated illustration: store map, location of category "dresses", blue relaxed color scheme

[1123] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1124] Step 1:

[1125] A user wears smart glasses and inputs information by voice or text. The input data is captured from the smart glasses' microphone and keyboard. The input voice data is converted into text using voice recognition software.

[1126] Step 2:

[1127] The device preprocesses the input data and sends it to the cloud server, where it undergoes format conversion and optimization, converting voice data to text, and transmitting it according to the communication protocol. The input information is properly formatted and prepared for analysis on the server side.

[1128] Step 3:

[1129] The server analyzes the received text data using a natural language processing engine, which extracts key phrases and keywords and identifies the subject of the information entered. During this step, multiple filtering and analysis algorithms are applied to identify the category and subject of the information.

[1130] Step 4:

[1131] The server analyzes the received voice data using an emotion analysis engine to identify the user's emotions. The emotion is determined from the tone of the voice, speaking style, emphasis, etc., and emotional states such as relaxation, tension, joy, and anger are identified. The emotional data is used to adjust the layout and color of the illustrations.

[1132] Step 5:

[1133] Based on the analysis results, the server determines the appropriate illustration format, such as a map, flowchart, graph, or table, and then adjusts the layout and color scheme according to the user's emotions. For example, a relaxed user might be assigned a softer color scheme.

[1134] Step 6:

[1135] If the server does not have enough additional information, it will ask the user for a query. For example, if the question is "Where is the new dress?", the server will generate and send a query message to the user to ask for additional data such as the start date and location if more detailed product information is needed.

[1136] Step 7:

[1137] The server sends the generated illustration data to the smart glasses terminal, where it uses an illustration generation engine to generate a visually easy-to-understand illustration according to the selected format, and then the generated illustration data is sent from the cloud server to the smart glasses.

[1138] Step 8:

[1139] The smart glasses display the diagram data received from the server. Users can visually confirm the information through the glasses' display. The layout and color of the diagrams displayed are based on the analysis results and emotions, allowing users to comfortably view the information.

[1140] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1141] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1142] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1143] [Fourth embodiment]

[1144] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1145] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1146] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1147] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1148] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1149] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1150] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1151] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1152] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1153] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1154] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1155] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1156] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1157] The present invention relates to a system that converts information input by text or voice into a diagram format and displays it. This system provides a function in which a user inputs information using text or voice, analyzes the information, and generates and displays an appropriate diagram. Specific embodiments of the present invention are described below.

[1158] 1. Text and voice input methods

[1159] Users input information into the device either by text or voice. Text input is done using a keyboard or touchscreen, while voice input is done using a microphone or voice recognition software. For example, a user can type, "I need a Gantt chart showing the progress of a project."

[1160] 2. Transmission Method

[1161] The device preprocesses the input data and sends it to the server. In the case of voice input, the device uses voice recognition technology to convert the voice data into text data, which is then formatted into a standard format and sent to the server.

[1162] 3. Analysis method

[1163] The server uses natural language processing technology to analyze the text data sent. The analysis engine extracts key phrases and keywords and identifies themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted, and based on this, the text is determined to be information about project management.

[1164] 4. Means of obtaining additional information

[1165] To generate a Gantt chart, the server needs detailed information such as specific tasks, deadlines, and progress. If this additional information is missing, the server queries the user to obtain the necessary data. For example, the server might send a message to the user requesting the "project start date," "task details," and "deadline for each task."

[1166] 5. Illustration Generation Method

[1167] After retrieving the necessary data, the server uses a diagram generation engine to generate the appropriate diagram, which supports formats such as flowcharts, Gantt charts, graphs, tables, etc. For example, a Gantt chart showing each task in a project and its duration can be generated.

[1168] 6. Illustrated Data Transmission Methods

[1169] The generated diagram is sent from the server to the device. The diagram data is sent in a format that the user can view and edit (e.g., image data, Excel file, Google Sheets URL, etc.).

[1170] 7. Display means

[1171] The device receives the diagram data from the server and displays it to the user. The user can visually check the diagram on the device screen and edit it as needed. For example, a Gantt chart showing the progress of a project can be displayed on the screen, allowing the user to check the progress and deadlines of each task at a glance.

[1172] In this way, the system of the present invention can analyze the information entered by the user in real time and convert it into an appropriate diagram format for display, thereby enabling complex information to be visualized concisely and greatly improving communication efficiency.

[1173] The processing flow will be explained below.

[1174] Step 1:

[1175] The user inputs text or voice into the device. For example, the user inputs, "I need a Gantt chart showing the progress of a project."

[1176] Step 2:

[1177] The device receives input or speech. In the case of speech input, the device uses speech recognition software to convert the speech data into text.

[1178] Step 3:

[1179] The terminal preprocesses the text data and formats it into a standard format, then sends the formatted data to the server.

[1180] Step 4:

[1181] The server analyzes the received text data. Natural language processing techniques are used to extract key phrases and keywords and identify themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[1182] Step 5:

[1183] The server then determines the appropriate diagram format based on the analysis results. In this example, a Gantt chart is selected as the appropriate diagram format.

[1184] Step 6:

[1185] The server checks the details needed to generate the Gantt chart. If the required information is missing, the server prompts the user for additional information. For example, "What is the start date of the project?"

[1186] Step 7:

[1187] The user answers with additional information. For example, the user answers, "The project started on October 1, 2023."

[1188] Step 8:

[1189] The server receives additional information and adds it to the data used to generate the Gantt chart, such as project tasks, deadlines, and progress.

[1190] Step 9:

[1191] The server generates a Gantt chart using a diagram generation engine, and the generated Gantt chart is converted into image data or an editable format within the server.

[1192] Step 10:

[1193] The server transmits the generated diagram data to the terminal in a format that the user can view and edit.

[1194] Step 11:

[1195] The terminal displays the received graphical data to the user, for example, a Gantt chart is displayed on the terminal screen, allowing the user to visually check the progress of the project.

[1196] Step 12:

[1197] Users can check the displayed diagrams and edit them as necessary, which allows users to communicate more efficiently.

[1198] Example 1

[1199] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1200] Conventional information input systems have difficulty properly analyzing information entered via text or voice and converting it into a visually easy-to-understand diagram format. In particular, when the user's input information is insufficient, there is a lack of a means to effectively acquire additional information and generate a diagram, which reduces user convenience. Furthermore, the inability to update or edit diagram data in real time makes it difficult to make decisions based on the latest information.

[1201] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1202] In this invention, the server includes a means for converting input data into text using speech recognition technology, a means for analyzing the data using natural language processing technology, and a means for updating the generated illustrations in real time. This allows the information input by the user via text or voice to be properly analyzed, and missing information to be effectively acquired and an illustration to be generated. Furthermore, editing and real-time updating of the generated illustration data allows for efficient decision-making based on the latest information.

[1203] A "text input means" is a device and method that allows a user to input text information using a keyboard or touch screen.

[1204] "Voice input means" refers to a device and method including a microphone and voice recognition software for inputting user voice information and processing it as digital data.

[1205] The "transmitting means" refers to a device and method including a network interface and communication protocol for preprocessing input data and transmitting it to a server.

[1206] "Analysis means" refers to software and algorithms that the server uses to analyze the received data using natural language processing techniques and determine an appropriate graphical format.

[1207] The "acquisition means" refers to a device and method for acquiring additional information required by the server to generate the illustration from the user through a query.

[1208] "Illustration generation means" refers to software and algorithms for generating appropriate illustrations (Gantt charts, flow charts, graphs, tables) based on collected data.

[1209] The "transmitting means" refers to an apparatus and method including a network interface and a communication protocol for the server to transmit the generated illustration data to the terminal.

[1210] "Display means" refers to a display and software for visually displaying the graphical data received by the terminal.

[1211] The "query means" refers to a device and method by which the server queries the user for necessary information when additional information is missing.

[1212] "Speech recognition technology" refers to software and algorithms used to convert voice data into text data.

[1213] "Natural language processing technology" refers to software and algorithms that analyze text data, extract key phrases and keywords, and perform semantic analysis.

[1214] "Real-time update means" refers to a device and method for instantly updating generated graphical data based on the latest information.

[1215] The "editing means" refers to software and an interface that allows the user to edit the received graphic data on the terminal.

[1216] The system of the present invention is a system that can convert information input by a user in text or voice into a diagram format and visually display it. To implement this system, the following hardware and software are used.

[1217] Hardware and Software

[1218] Enter information

[1219] Users can input text using a keyboard or touchscreen, such as "I need a Gantt chart showing the progress of a project." A microphone is also used for voice input, which is then converted into text using speech recognition software, such as the Google Speech-to-Text API.

[1220] Data transmission

[1221] The terminal formats the input text data into a standard format and sends it to the server, for example, in JSON format, using an HTTP POST request.

[1222] Data analysis

[1223] The server analyzes the received text data using natural language processing (NLP) techniques. Analysis engines such as SpaCy and the BERT model are used. This allows the server to extract key phrases and keywords and identify themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[1224] Obtaining additional information

[1225] The server queries the user to obtain additional information needed to generate the diagram. This information is obtained via HTTP requests or message transmission. The user enters the requested information into the terminal and resubmits it. For example, the user enters specific details such as "The project start date is October 1, 2023, Task 1 will be planning until October 5, and Task 2 will be design until October 15."

[1226] Illustration generation

[1227] After the server collects all the necessary data, it uses a diagram generation engine, such as D3.js or Syncfusion, to generate diagrams, such as Gantt charts, that show each project task and its duration.

[1228] Sending diagram data

[1229] The generated diagram data is sent from the server to the terminal via a network interface or communication protocol. The diagram data is often sent in image format or an editable format.

[1230] Data Display

[1231] The device then displays the received diagram data to the user. A display and corresponding software are used as the display method. The user can check the diagram on the device screen and edit it as needed. For example, a Gantt chart showing the project progress can be displayed, allowing the progress and deadlines of each task to be confirmed at a glance.

[1232] Specific examples

[1233] Consider the example where a user types into a terminal, "Show me the agenda for my next meeting in a flowchart." The following steps occur:

[1234] 1. The user enters text into the terminal.

[1235] 2. The terminal converts the input text data into a standard format and sends it to the server.

[1236] 3. The server analyzes the data using natural language processing technology and extracts the keywords "meeting agenda" and "flowchart."

[1237] 4. The server queries the user for the missing information (specific agenda items).

[1238] 5. The user enters the specific agenda item into the terminal and submits it.

[1239] 6. The server generates a flowchart based on the collected information.

[1240] 7. The generated flowchart is sent to the terminal and displayed.

[1241] 8. The user can check the displayed flowchart and edit it if necessary.

[1242] This system allows users to easily visualize complex information and communicate effectively based on it.

[1243] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1244] Step 1:

[1245] The user inputs information into the terminal by text or voice.

[1246] Specifically, the user inputs text information using a keyboard or touch screen, or voice information using a microphone. If the input information is text information such as "I need a Gantt chart showing the progress of the project," the device receives it as is. If the input information is voice information, it is received by the microphone and converted into text using voice recognition technology in the next step.

[1247] Step 2:

[1248] The terminal uses voice recognition software to convert the voice data into text data.

[1249] Specifically, speech recognition software (e.g., Google Speech-to-Text API) is used to convert voice data into text data in real time. This converted text data is used in the next step. The output is sentence text data.

[1250] Step 3:

[1251] The terminal formats the input text data into a standard format (e.g., JSON) and sends it to the server.

[1252] Specifically, the input text data is formatted into JSON format and sent to the server using an HTTP POST request. For example, the formatted JSON data looks like this:

[1253] json

[1254] {

[1255] "request": "I need a Gantt chart to show the progress of my project"

[1256] }

[1257] The output is JSON data that is sent to the server.

[1258] Step 4:

[1259] The server analyzes the received text data using natural language processing (NLP) techniques.

[1260] Specifically, the server uses an NLP engine (e.g., SpaCy or a BERT model) to extract key phrases and keywords from the received text data. For example, keywords such as "project progress" and "Gantt chart" are extracted, and the theme of the text is identified as "project management." The output is the analyzed keywords and theme information.

[1261] Step 5:

[1262] The server queries the user to obtain additional information needed to generate the illustration.

[1263] Specifically, the server identifies the missing information and generates a message to inquire about it to the user. For example, it generates a message asking for the "start date of the project," "details of the tasks," and "deadlines of each task," and sends it to the terminal. The output is a query message.

[1264] Step 6:

[1265] The user receives the inquiry message from the server at the terminal, enters additional information, and resends it.

[1266] Specifically, the user inputs the necessary information according to the inquiry displayed on the terminal. Specific details such as "The project start date is October 1, 2023, Task 1 will be planning until October 5, and Task 2 will be design until October 15" are entered, and the terminal sends this to the server. The output is the additional information entered by the user.

[1267] Step 7:

[1268] The server uses a diagram generation engine to generate diagrams based on all the collected information.

[1269] Specifically, the server runs a diagram generation engine (e.g., D3.js or Syncfusion) to generate an appropriate diagram format (e.g., Gantt chart) based on the collected information. For example, a Gantt chart is generated that visually shows each task and its duration. The output is the generated diagram data.

[1270] Step 8:

[1271] The server transmits the generated illustration data to the terminal.

[1272] Specifically, the generated diagram data is sent to the terminal in an image format (e.g., PNG or JPEG) or in an editable format (e.g., an Excel file or a Google Sheets URL). The output is the diagram data sent to the terminal.

[1273] Step 9:

[1274] The terminal displays the received diagram data to the user and edits it as necessary.

[1275] Specifically, the received diagram data is displayed on the terminal display, and the user can visually check it. For example, a Gantt chart showing the progress of a project is displayed on the screen, and the user can check the progress and deadlines of each task and make edits as necessary. The output is the visually displayed diagram data.

[1276] This series of processing steps allows users to quickly and effectively generate and use visual diagrams based on the information they enter.

[1277] (Application example 1)

[1278] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1279] Conventional factory work process management faces the problem of making it difficult to visually grasp complex processes and vast amounts of data. Effective visualization is essential to efficiently manage work processes and enable workers and managers to quickly understand the processes and issue appropriate instructions. Furthermore, process data exists in different formats, and integrating and visualizing them is time-consuming. To solve these issues, a system is needed that uses voice and text input to automatically analyze work processes and convert them into an appropriate diagram format, such as a flowchart.

[1280] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1281] In this invention, the server includes a text input means, a voice input means, a transmission means for processing the input data and transmitting it to the server, an analysis means for the server to analyze the data and determine an appropriate diagram format, a means for the server to acquire necessary additional information, a means for transmitting the generated diagram data to the terminal, a display means for displaying the diagram data received by the terminal, an input information analysis means for managing the work process in the factory, and a flowchart generation means for visualizing the work process based on the analysis results. This makes it possible to efficiently manage complex work processes in a factory and to grasp the processes in a form that is visually easy for workers and managers to understand.

[1282] "Text input means" refers to a means for a user to input text using a keyboard, touch screen, or the like.

[1283] "Voice input means" refers to a means for a user to input voice using a microphone or voice recognition software.

[1284] The "transmission means" is a means for processing input data and transmitting it to the server.

[1285] "Analysis means" means by which the server analyzes the data and determines the appropriate graphical format.

[1286] The "additional information acquisition means" is a means for the server to acquire the necessary additional information.

[1287] The "graphic data transmission means" is a means for transmitting the generated graphic data to the terminal.

[1288] The "display means" is a means for displaying the graphic data received by the terminal to the user.

[1289] The "input information analysis means" is a means for analyzing input information in order to manage the work process in the factory.

[1290] The "flowchart generation means" is a means for generating a flow chart to visualize the work process based on the analysis results.

[1291] The present invention is a system that analyzes information input from voice or text, and automatically generates and displays visual diagrams such as flowcharts based on that information, in order to facilitate the management of factory work processes. Specific embodiments of the system are described below.

[1292] 1. Text and voice input methods

[1293] The user can input text using a keyboard or touchscreen. They can also input speech through a microphone. Speech input uses speech recognition software to convert the speech data into text. At this stage, the user might input a prompt such as, "First, prepare the materials, then move on to the assembly process, and finally perform quality checks."

[1294] 2. Transmission Method

[1295] The device preprocesses the input data and sends it to the server. In particular, in the case of voice input, the voice data is converted into text data using voice recognition technology. The converted text data is then formatted into a standard format and sent to the server.

[1296] 3. Analysis method

[1297] The server uses natural language processing technology to analyze the text data sent. During this process, key phrases and keywords are extracted using an analysis model using the transformers library. For example, keywords such as "material preparation," "assembly process," and "quality check" are extracted.

[1298] 4. Means of obtaining additional information

[1299] Based on the analysis results, the server queries the user for any missing details. This query is in the form of a message, and can obtain additional information from the user. For example, the server may need to know the detailed work content of each process or the time required for each process.

[1300] 5. Flowchart generation method

[1301] After obtaining the necessary additional information, the server generates a flowchart or Gantt chart. The networkx and matplotlib libraries are used to visualize the information. For example, a flowchart from "material preparation" to "assembly process" and "quality check" is generated.

[1302] 6. Illustrated Data Transmission Methods

[1303] The generated diagrams are sent from the server to the terminal and provided in a format that can be viewed and edited by the user.

[1304] 7. Display means

[1305] The terminal displays the diagram data received from the server to the user. By checking the displayed flowcharts and Gantt charts, the user can easily visually understand the factory work processes. The user can also edit the data as needed.

[1306] In this way, the system of the present invention efficiently manages the work processes in a factory, allowing workers and managers to grasp the processes in a visually easy-to-understand format.

[1307] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1308] Step 1:

[1309] The user inputs information into the terminal by text or voice.

[1310] Input: The user types text using a keyboard or touchscreen, or speaks using a microphone.

[1311] Output: The input text data or audio data.

[1312] Specific operation: The user inputs the prompt sentence: "First, prepare the materials, then move on to the assembly process. Finally, perform quality check."

[1313] Step 2:

[1314] The terminal converts the voice data into text data.

[1315] Input: Audio data.

[1316] Output: Text data.

[1317] Specific operation: The device uses voice recognition technology to convert voice data into text, using the speech_recognition library.

[1318] Step 3:

[1319] The terminal preprocesses the text data and sends it to the server.

[1320] Input: Text data.

[1321] Output: Preprocessed text data.

[1322] Specific operation: The terminal performs formatting processing on the text data, converts it into a standard format, and then sends the data to the server.

[1323] Step 4:

[1324] The server analyzes the data using natural language processing technology.

[1325] Input: Preprocessed text data.

[1326] Output: Keyphrases or keywords.

[1327] Specific operation: The server uses the transformers library to extract key phrases and keywords such as "material preparation," "assembly process," and "quality check" from the text data.

[1328] Step 5:

[1329] The server prompts the user for any additional information that is missing.

[1330] Input: Keywords obtained from the analysis results.

[1331] Output: Additional information from the user.

[1332] Specific operation: The server queries the user for missing information in the form of a message, for example, to confirm "detailed work content for each process" and "time required for each process."

[1333] Step 6:

[1334] The server generates the flowchart.

[1335] Input: Additional information collected from the user and analysis results.

[1336] Output: Flowchart data.

[1337] What it does: The server uses the networkx and matplotlib libraries to generate a flowchart based on the information it collects.

[1338] Step 7:

[1339] The server transmits the generated flowchart data to the terminal.

[1340] Input: Flowchart data.

[1341] Output: Illustrated data sent to the terminal.

[1342] Specific operation: The server sends the flowchart data to the terminal and provides it in a format (e.g., image data) that the user can view and edit.

[1343] Step 8:

[1344] The terminal displays the received flowchart data to the user.

[1345] Input: Illustrated data sent from the server.

[1346] Output: The flowchart that is displayed to the user.

[1347] Specific operation: The terminal displays the received flowchart data on the screen, allowing the user to visually check the factory work process.

[1348] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1349] The present invention relates to a system that converts information input via text or voice into a diagram format and adjusts the diagram by recognizing the user's emotions. This system provides a function that allows a user to input information using text or voice, analyzes the information, generates and displays an appropriate diagram, and also a function that recognizes the user's emotions and adjusts the layout and color of the diagram. Specific embodiments of the present invention are described below.

[1350] 1. Text and voice input methods

[1351] Users input information into the device either by text or voice. Text input is done using a keyboard or touchscreen, while voice input is done using a microphone or voice recognition software. For example, a user can type, "I need a Gantt chart showing the progress of a project."

[1352] 2. Transmission Method

[1353] The device preprocesses the input data and sends it to the server. In the case of voice input, the device uses voice recognition technology to convert the voice data into text data, which is then formatted into a standard format and sent to the server.

[1354] 3. Emotion recognition means

[1355] The server is equipped with an emotion engine for recognizing the user's emotion from the received voice data. This engine uses voice analysis technology to determine the emotion from the user's tone of voice and expressions. For example, it can recognize emotions such as nervousness, relaxation, anger, etc. from the user's tone of voice.

[1356] 4. Analysis method

[1357] After emotion recognition, the server uses natural language processing technology to analyze the text data. The analysis engine extracts key phrases and keywords to identify themes within the text. For example, keywords like "project progress" and "Gantt chart" may be extracted.

[1358] 5. Deciding on the illustration format

[1359] The server then determines the appropriate illustration format based on the analysis results. In this example, a Gantt chart is selected as the appropriate illustration format. At the same time, the layout and color of the illustration are adjusted based on the user's emotions. For example, if the user is relaxed, calm colors are used, and if the user is tense, a visually gentle color scheme is used.

[1360] 6. Means of obtaining additional information

[1361] To generate a Gantt chart, the server needs detailed information such as specific tasks, deadlines, progress, etc. If this additional information is missing, the server queries the user to obtain the necessary data. For example, the server might send a message to the user asking for the project start date, task details, and deadlines for each task.

[1362] 7. Illustration Generation Method

[1363] After retrieving the necessary data, the server uses a diagram generation engine to generate the appropriate diagram, which supports formats such as flowcharts, Gantt charts, graphs, and tables. For example, a Gantt chart showing each project task and its duration can be generated and adjusted according to the user's emotions.

[1364] 8. Illustrated Data Transmission Methods

[1365] The generated diagram is sent from the server to the device. The diagram data is sent in a format that the user can view and edit (e.g., image data, Excel file, Google Sheets URL, etc.).

[1366] 9. Display means

[1367] The device displays the diagram data received from the server to the user. The user can visually check the diagram on the device screen and edit it as needed. For example, a Gantt chart showing the progress of a project can be displayed on the screen, allowing the user to check the progress and deadlines of each task at a glance. In addition, layouts and colors are applied according to emotions, making it easier for users to use the diagrams.

[1368] In this way, the system of the present invention can analyze the information entered by the user in real time, convert it into an appropriate diagram format for display, and adjust the diagram taking into account the user's feelings, thereby enabling complex information to be visualized concisely and greatly improving communication efficiency.

[1369] The processing flow will be explained below.

[1370] Step 1:

[1371] The user inputs text or voice into the device. For example, the user inputs, "I need a Gantt chart showing the progress of a project."

[1372] Step 2:

[1373] The device receives input or speech. In the case of speech input, the device uses speech recognition software to convert the speech data into text.

[1374] Step 3:

[1375] The terminal preprocesses the text data and formats it into a standard format, then sends the formatted data to the server.

[1376] Step 4:

[1377] The server analyzes the received text data. Natural language processing techniques are used to extract key phrases and keywords and identify themes within the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[1378] Step 5:

[1379] The server uses an emotion recognition engine to analyze emotions from the voice data sent by the user, for example, determining whether the user is nervous or relaxed from their voice tone.

[1380] Step 6:

[1381] The server determines the appropriate diagram format based on the analysis results. At the same time, it adjusts the layout and color of the diagram based on the user's emotions. For example, a Gantt chart is selected as the appropriate diagram format, and the color scheme is determined according to the user's emotions.

[1382] Step 7:

[1383] The server checks the details required to generate the Gantt chart. If the required information is missing, the server will ask the user questions such as "Please tell me the start date of the project and details of the tasks."

[1384] Step 8:

[1385] The user answers with additional information. For example, the user might say, "The project started on October 1, 2023."

[1386] Step 9:

[1387] The server receives additional information and adds it to the data used to generate the Gantt chart, such as project tasks, deadlines, and progress.

[1388] Step 10:

[1389] The server generates a Gantt chart using a diagram generation engine. The generated Gantt chart is converted into image data or an editable format within the server, and the layout and color are applied according to the user's emotions.

[1390] Step 11:

[1391] The server transmits the generated diagram data to the terminal in a format that the user can view and edit.

[1392] Step 12:

[1393] The device displays the received graphical data to the user. For example, a Gantt chart is displayed on the device screen, allowing the user to visually check the progress of the project. The layout and color are also applied according to the emotion.

[1394] Example 2

[1395] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1396] Conventional information input systems can properly analyze information entered via text or voice and convert it into diagram format, but they lack the ability to recognize the user's emotions and adjust the layout and color of the diagram, limiting their ability to improve the user experience. Furthermore, they lacked efficiency and flexibility when acquiring additional information and transmitting and displaying diagrams.

[1397] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an emotion recognition means, a means for analyzing data using natural language processing technology, and a means for adjusting the layout and color of the illustration based on the user's emotion. This makes it possible to generate an appropriate illustration format taking the user's emotion into consideration and to efficiently and flexibly obtain additional information the user needs.

[1398] "Text input means" refers to a means by which a user inputs text using a keyboard or a touch screen.

[1399] "Voice input means" refers to means for obtaining the user's voice as input data using a microphone or voice recognition software.

[1400] The "transmission means" is a means for preprocessing input data, converting it into a standard format, and transmitting it to the server.

[1401] "Analysis means" refers to means by which the server analyzes the received data and determines an appropriate graphical format.

[1402] The "emotion recognition means" is a means by which the server determines the user's emotions from voice and text expressions.

[1403] The "additional information acquisition means" is a means for the server to inquire of the user and acquire detailed information necessary for generating an illustration.

[1404] The "illustration generating means" is a means for generating an appropriate illustration using all collected data.

[1405] The "display means" is a means for displaying the graphic data received by the terminal to the user.

[1406] "Speech recognition technology" is a technology that converts input speech into text data.

[1407] "Natural language processing technology" is a technology that analyzes input text data and understands key phrases, keywords, and context.

[1408] "Illustration format" refers to a format that visually presents data, such as a Gantt chart, flow chart, graph, or table.

[1409] The "layout adjustment means" is a means for adjusting the layout of illustrations based on the user's feelings.

[1410] The "color adjustment means" is a means for adjusting the color scheme of the illustration based on the user's feelings.

[1411] The present invention relates to a system that converts information input via text or voice into an illustration format and adjusts the illustration by recognizing the user's emotions. This system is intended to be used by an unspecified number of users and aims to generate appropriate illustrations in real time. The following describes in detail the embodiments of the present invention.

[1412] Text and voice input methods

[1413] Users input information through the device, either through a keyboard or touchscreen for text input or through a microphone and voice recognition software for voice input. For example, a user might type, "I need a Gantt chart showing the progress of a project."

[1414] Sending data

[1415] The device preprocesses the data entered by the user, converts it into a standard format, and sends it to the server. In the case of voice input, the device uses voice recognition technology (e.g., Google voice recognition API) to convert the voice data into text data, formats it into JSON format, and sends it to the server.

[1416] emotion recognition means

[1417] The server processes the received data and recognizes the user's emotions using an emotion engine (e.g., IBM Watson Emotion Analysis), which determines the user's emotions from voice tone and text expressions, for example, identifying whether the user is tense, relaxed, angry, etc.

[1418] Data analysis

[1419] After emotion recognition, the server analyzes the input text data using natural language processing technology (e.g., OpenAI GPT-4). This analysis engine extracts key phrases and keywords from the text and identifies the subject and purpose of the text. For example, keywords such as "project progress" and "Gantt chart" are extracted.

[1420] Deciding on an illustration format

[1421] The server determines the appropriate illustration format based on the analysis results and emotion recognition results. For example, a Gantt chart is deemed appropriate, and the layout and color of the illustration are also adjusted according to the user's emotional state. If the user is relaxed, calm colors are used, and if the user is tense, a color scheme that is less visually demanding is selected.

[1422] Obtaining additional information

[1423] The server collects detailed information (e.g., tasks, deadlines, and progress) required to generate the diagram. If the required information is missing, it prompts the user for additional information. For example, it sends questions such as, "What is the project start date, task details, and deadlines for each task?"

[1424] Generating illustrations

[1425] Using all the collected data, the server uses a diagram generation engine (e.g., D3.js or Matplotlib) to generate the final diagram, which can be adjusted in layout and color based on the user's preferences. For example, a Gantt chart showing each project task and its duration can be generated.

[1426] Illustrated data transmission and display

[1427] The generated diagram data is sent from the server to the terminal. This diagram data is provided in a format that the user can view and edit (e.g. PNG, Excel file, Google Sheets URL). The terminal displays the received diagram data to the user, allowing the user to visually check the contents of the diagram and edit it as necessary.

[1428] Example prompt sentences

[1429] For example, by inputting the following prompts into a generative AI model, it can generate diagrams that meet the user's requirements:

[1430] You want to create a Gantt chart that details the progress of a project. The project starts on October 1, 2023, and the major tasks include research, design, development, testing, and deployment. You want it to reflect the deadlines and progress of each task. Also, your current emotional state is relaxed. Choose a layout and color scheme that reflects this.

[1431] The above is a specific embodiment for carrying out the present invention. This system is capable of quickly and effectively converting user input information into illustrations and presenting the results in an optimal format.

[1432] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1433] Step 1: Enter your information

[1434] Users input information into the device either through text or voice. Text input is done using a keyboard or touchscreen. Voice input is done using a microphone or voice recognition software. The input data can be in text or voice data format. For example, a user might say, "I need a Gantt chart showing the progress of a project."

[1435] Step 2: Preprocess and send data

[1436] The device preprocesses the input data. In the case of voice data, it converts it into text data using voice recognition technology (e.g., Google voice recognition API). The converted text data is then formatted into a standard format (e.g., JSON) and sent to the server. Specifically, the device converts the voice into text, formats the text as JSON, and sends it to the server via an HTTP request.

[1437] Step 3: Emotion Recognition

[1438] The server receives the data sent from the device and analyzes the user's emotions using an emotion recognition engine (e.g., IBM Watson Emotion Analysis). The input data is analyzed based on voice tone and text expressions to determine emotions such as tension, relaxation, and anger. The emotion recognition engine analyzes the input data and returns the user's emotional state (e.g., tension, relaxation, anger) as its output.

[1439] Step 4: Data analysis

[1440] The server receives the emotion recognition results and then analyzes the input text data using natural language processing technology (e.g., OpenAI GPT-4). This analysis engine extracts key phrases and keywords from the text and identifies the subject and intent. For example, it extracts input keywords such as "project progress" and "Gantt chart." The subject and intent are provided as specific keywords as output.

[1441] Step 5: Decide on a graphical format

[1442] The server determines the appropriate diagram format based on the analysis results and emotion recognition results. For example, the analysis results may determine that a "Gantt chart" is appropriate. The layout and color are also adjusted taking into account the emotion recognition results. Specifically, if the user is relaxed, calm colors are selected, and if the user is tense, a color scheme that is less visually demanding is chosen. The output is the diagram format and settings for adjusting the layout.

[1443] Step 6: Obtain additional information

[1444] The server collects the necessary details (tasks, deadlines, progress, etc.) to generate the diagram. It sends a query message to the user for any missing information. For example, it sends specific questions such as "What is the project start date, task details, and deadlines for each task?" and waits for a response from the user. The input is the question from the server, and the output is the answer from the user.

[1445] Step 7: Generate the diagram

[1446] The server uses all the collected data to generate diagrams using a diagram generation engine (e.g., D3.js or Matplotlib). At this time, the diagram is generated with layout and color adjusted based on the results of emotion recognition. Specifically, a Gantt chart showing the duration of each task is generated. The input is the collected detailed information and adjustment settings, and the output is the generated diagram data.

[1447] Step 8: Submitting diagram data

[1448] The generated diagram data is sent from the server to the terminal. This diagram data is provided in a format that the user can view and edit (e.g. PNG, Excel file, Google Sheets URL). Specifically, the generated PNG file is sent to the terminal via an HTTP response. The generated diagram data is used as input, and transmission to the user's terminal is completed as output.

[1449] Step 9: Displaying the diagram

[1450] The terminal displays the diagram data received from the server to the user. The user can check the diagram on the terminal screen and edit it as necessary. For example, the received PNG file can be displayed on the screen to visually check the details and progress of each task. It is also possible to make additional edits and adjustments based on the contents of the diagram. The input is the diagram data received from the server, and the output is the display to the user.

[1451] (Application example 2)

[1452] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1453] Traditional brick-and-mortar stores have faced the challenge of quickly and accurately providing customers with the products and information they are looking for. Particularly in large stores or stores with complex layouts, customers often spend a lot of time trying to find the product they are looking for. Furthermore, providing information without considering the customer's emotions or state can sometimes detract from the customer experience. To solve these problems, a system was needed that could generate and display appropriate illustrations in real time based on the information entered by the customer.

[1454] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1455] In this invention, the server includes a text input means, a voice input means, a transmission means for processing the input data and transmitting it to the server, an analysis means for the server to analyze the data and determine an appropriate illustration format, a means for the server to acquire necessary additional information, a means for transmitting the generated illustration data to the terminal, a display means for displaying the illustration data received by the terminal, a means for a user wearing smart glasses to input store information by voice or text, and an emotion recognition means for recognizing the user's emotions and adjusting the layout and color of the illustrations. This allows customers to easily check the location of products and ask questions in the store, and also allows them to receive information presented according to their emotions and state.

[1456] "Text input means" refers to a device or method for a user to input text, including a keyboard and a touch screen.

[1457] "Voice input means" refers to a device or method that allows a user to input information by voice, and includes a microphone and voice recognition software.

[1458] "Transmission means" refers to a device or method for processing input data and transmitting it to a server, and includes a communication module and a data format conversion means.

[1459] "Analysis means" refers to a method or device for analyzing the data received by the server and determining an appropriate graphical format, including natural language processing techniques and keyword extraction algorithms.

[1460] "Means for obtaining additional information" refers to a device or method for querying a user or collecting additional data required for generating an illustration.

[1461] The "means for transmitting generated graphic data" refers to a device or method for transmitting graphic data generated by the server to the terminal.

[1462] "Display means" refers to a device or method for displaying the graphical data received by the terminal to the user, and includes a display and a VR device.

[1463] "Smart glasses" refers to a wearable device that has the ability to receive and display data.

[1464] "Emotion recognition means" refers to a method or device for analyzing and identifying a user's emotions from vocal tone and facial expressions, and includes emotion analysis engines and machine learning technologies.

[1465] "Illustrative formats" refers to formats that visually display data and include maps, flowcharts, graphs, tables, etc.

[1466] This invention provides a system that uses smart glasses to improve customer experience in brick-and-mortar stores. This system analyzes information entered by customers via voice or text in real time, and generates and displays appropriate illustrations. It also recognizes customer emotions and adjusts the layout and color of the illustrations accordingly.

[1467] Hardware and software used

[1468] Hardware:

[1469] Smart glasses (e.g. general-purpose wearable devices)

[1470] microphone

[1471] Display Device

[1472] software:

[1473] Natural language processing engine (e.g., cloud natural language processing API)

[1474] Speech recognition software (e.g., cloud speech recognition services)

[1475] Sentiment analysis engine (e.g., cloud sentiment analysis service)

[1476] Diagram generation engine (e.g., a web-based diagram generation library)

[1477] Data processing and calculation methods

[1478] 1. Data entry and submission:

[1479] The user inputs information through the smart glasses by voice or text. The input voice data is converted into text data using voice recognition software. The text data is then sent to the server.

[1480] 2. Emotion recognition:

[1481] The server analyzes the received voice data using an emotion analysis engine to identify the user's emotion, such as tension, relaxation, or anger, based on the user's tone of voice.

[1482] 3. Data Analysis:

[1483] The server analyzes the text data using a natural language processing engine, which extracts key phrases and keywords and identifies themes in the input information.

[1484] 4. Decide on the illustration format:

[1485] The server determines the appropriate illustration format based on the analysis results. It also adjusts the layout and color of the illustration based on the user's emotions. For example, if the user uses a relaxed voice tone, a map with calm colors will be generated.

[1486] 5. Obtaining additional information:

[1487] If necessary, the server queries the user for any missing additional information to complete the required data.

[1488] 6. Generating and sending diagrams:

[1489] The server uses an illustration generation engine to generate an appropriate illustration and transmits the data to the smart glasses, which then display the generated illustration on their display.

[1490] Specific examples

[1491] When a customer asks the smart glasses, "Where are the new dresses?", the voice data is sent to the server for analysis. The server extracts the keywords "new" and "dress" and generates an illustration showing the location of the appropriate category. If the server recognizes the customer's voice tone as relaxed, the illustration will be displayed in a calming blue color. The customer can then view the illustration through the smart glasses.

[1492] Prompt Sentence Examples

[1493] It analyzes voice data and illustrates the location of the product the customer is asking about.

[1494] Voice: "Where are the new dresses?"

[1495] Emotion: Relaxed tone of voice

[1496] Analyzed information: "New item", "Dress"

[1497] Generated illustration: store map, location of category "dresses", blue relaxed color scheme

[1498] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1499] Step 1:

[1500] A user wears smart glasses and inputs information by voice or text. The input data is captured from the smart glasses' microphone and keyboard. The input voice data is converted into text using voice recognition software.

[1501] Step 2:

[1502] The device preprocesses the input data and sends it to the cloud server, where it undergoes format conversion and optimization, converting voice data to text, and transmitting it according to the communication protocol. The input information is properly formatted and prepared for analysis on the server side.

[1503] Step 3:

[1504] The server analyzes the received text data using a natural language processing engine, which extracts key phrases and keywords and identifies the subject of the information entered. During this step, multiple filtering and analysis algorithms are applied to identify the category and subject of the information.

[1505] Step 4:

[1506] The server analyzes the received voice data using an emotion analysis engine to identify the user's emotions. The emotion is determined from the tone of the voice, speaking style, emphasis, etc., and emotional states such as relaxation, tension, joy, and anger are identified. The emotional data is used to adjust the layout and color of the illustrations.

[1507] Step 5:

[1508] Based on the analysis results, the server determines the appropriate illustration format, such as a map, flowchart, graph, or table, and then adjusts the layout and color scheme according to the user's emotions. For example, a relaxed user might be assigned a softer color scheme.

[1509] Step 6:

[1510] If the server does not have enough additional information, it will ask the user for a query. For example, if the question is "Where is the new dress?", the server will generate and send a query message to the user to ask for additional data such as the start date and location if more detailed product information is needed.

[1511] Step 7:

[1512] The server sends the generated illustration data to the smart glasses terminal, where it uses an illustration generation engine to generate a visually easy-to-understand illustration according to the selected format, and then the generated illustration data is sent from the cloud server to the smart glasses.

[1513] Step 8:

[1514] The smart glasses display the diagram data received from the server. Users can visually confirm the information through the glasses' display. The layout and color of the diagrams displayed are based on the analysis results and emotions, allowing users to comfortably view the information.

[1515] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1516] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1517] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1518] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1519] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1520] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1521] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1522] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1523] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1524] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1525] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1526] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1527] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1528] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1529] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1530] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1531] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1532] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1533] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1534] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1535] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1536] The following is further disclosed regarding the above embodiment.

[1537] (Claim 1)

[1538] A text input means;

[1539] A voice input means;

[1540] a transmitting means for processing the input data and transmitting the processed data to a server;

[1541] an analysis means for the server to analyze the data and determine an appropriate graphical format;

[1542] a means for the server to obtain any additional information it needs;

[1543] means for transmitting the generated diagram data to a terminal;

[1544] a display means for displaying the diagram data received by the terminal;

[1545] A system including:

[1546] (Claim 2)

[1547] A means for converting input data into text using voice recognition technology;

[1548] A means for the server to analyze the data using natural language processing technology;

[1549] 10. The system of claim 1, comprising:

[1550] (Claim 3)

[1551] a means for selecting a graphical format from a Gantt chart, a flowchart, a graph, and a table;

[1552] means for querying and obtaining additional information from the user;

[1553] 10. The system of claim 1, comprising:

[1554] "Example 1"

[1555] (Claim 1)

[1556] A text input means;

[1557] A voice input means;

[1558] a transmitting means for processing the input data and transmitting the processed data to a server;

[1559] an analysis means for the server to analyze the data and determine an appropriate graphical format;

[1560] A means for obtaining additional information required by the server;

[1561] a transmitting means for transmitting the generated diagram data to a terminal;

[1562] a display means for displaying the diagram data received by the terminal;

[1563] a query means for querying the user if additional information is missing;

[1564] A system including:

[1565] (Claim 2)

[1566] A means for converting input data into text using voice recognition technology;

[1567] A means for the server to analyze the data using natural language processing technology;

[1568] A means by which the generated illustrations can be updated in real time;

[1569] 10. The system of claim 1, comprising:

[1570] (Claim 3)

[1571] a means for selecting a graphical format from a Gantt chart, a flowchart, a graph, and a table;

[1572] means for querying and obtaining additional information from the user;

[1573] A means for allowing the received graphical data to be edited;

[1574] 10. The system of claim 1, comprising:

[1575] "Application Example 1"

[1576] (Claim 1)

[1577] A text input means;

[1578] A voice input means;

[1579] a transmitting means for processing the input data and transmitting the processed data to a server;

[1580] an analysis means for the server to analyze the data and determine an appropriate graphical format;

[1581] a means for the server to obtain any additional information it needs;

[1582] means for transmitting the generated diagram data to a terminal;

[1583] a display means for displaying the diagram data received by the terminal;

[1584] input information analysis means for managing the work process of the factory;

[1585] a flowchart generation means for visualizing the work process based on the analysis results;

[1586] A system including:

[1587] (Claim 2)

[1588] A means for converting input data into text using voice recognition technology;

[1589] A means for the server to analyze the data using natural language processing technology;

[1590] 10. The system of claim 1, comprising:

[1591] (Claim 3)

[1592] a means for selecting a graphical format from a Gantt chart, a flowchart, a graph, and a table;

[1593] means for querying and obtaining additional information from the user;

[1594] 10. The system of claim 1, comprising:

[1595] "Example 2: Combining Emotion Engines"

[1596] (Claim 1)

[1597] A text input means;

[1598] A voice input means;

[1599] a transmitting means for processing the input data and transmitting the processed data to a server;

[1600] an analysis means for the server to analyze the data and determine an appropriate graphical format;

[1601] An emotion recognition means for the server to recognize the emotion of the user;

[1602] a means for the server to obtain any additional information it needs;

[1603] means for transmitting the generated diagram data to a terminal;

[1604] a display means for displaying the diagram data received by the terminal;

[1605] A system including:

[1606] (Claim 2)

[1607] A means for converting input data into text using voice recognition technology;

[1608] A means for the server to analyze the data using natural language processing technology;

[1609] 10. The system of claim 1, comprising:

[1610] (Claim 3)

[1611] a means for selecting a graphical format from a Gantt chart, a flowchart, a graph, and a table;

[1612] means for querying and obtaining additional information from the user;

[1613] a means for adjusting the layout and color of the illustration based on the user's emotions;

[1614] 10. The system of claim 1, comprising:

[1615] "Application example 2 when combining emotion engines"

[1616] (Claim 1)

[1617] A text input means;

[1618] A voice input means;

[1619] a transmitting means for processing the input data and transmitting the processed data to a server;

[1620] an analysis means for the server to analyze the data and determine an appropriate graphical format;

[1621] a means for the server to obtain any additional information it needs;

[1622] means for transmitting the generated diagram data to a terminal;

[1623] a display means for displaying the diagram data received by the terminal;

[1624] A means for a user wearing the smart glasses to input in-store information by voice or text;

[1625] an emotion recognition means for recognizing a user's emotion and adjusting the layout and color of the illustration;

[1626] A system including:

[1627] (Claim 2)

[1628] A means for converting input data into text using voice recognition technology;

[1629] A means for the server to analyze the data using natural language processing technology;

[1630] 10. The system of claim 1, comprising:

[1631] (Claim 3)

[1632] means for selecting the graphical format from maps, flow charts, graphs, and tables;

[1633] means for querying and obtaining additional information from the user;

[1634] means for displaying graphical data via the smart glasses;

[1635] 10. The system of claim 1, comprising: [Explanation of symbols]

[1636] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A text input means; A voice input means; a transmitting means for processing the input data and transmitting the processed data to a server; an analysis means for the server to analyze the data and determine an appropriate graphical format; a means for the server to obtain any additional information it needs; means for transmitting the generated diagram data to a terminal; a display means for displaying the diagram data received by the terminal; A system including:

2. A means for converting input data into text using voice recognition technology; A means for the server to analyze the data using natural language processing technology; The system of claim 1 , comprising:

3. a means for selecting a graphical format from a Gantt chart, a flowchart, a graph, and a table; means for querying and obtaining additional information from the user; The system of claim 1 , comprising:

Images