System

Abstract

A system is provided.SOLUTION: A system, comprising: generating means for parsing Japanese directives; applying means for applying numeric expressions generated by said generating means to a spreadsheet; applying means for applying graphs generated by said generating means to a spreadsheet; applying means for applying conditional forms generated by said generating means to a spreadsheet; and applying means for applying data input restrictions generated by said generating means to a spreadsheet.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Spreadsheets present challenges for beginners and intermediate users, particularly in preventing formula errors, editing, and creating graphs. Specifically, formula errors and debugging require significant time, and users often rely on online searches or consultations with experts to properly configure formulas. Furthermore, many users are unfamiliar with automation using Google Apps Script (GAS), and assistance is needed to resolve these issues. This invention is a system developed to address these challenges and enable spreadsheet users to work more efficiently. [Means for solving the problem]

[0005] The present invention is a system that includes a generation means for analyzing Japanese instructions and an application means for applying the generated numerical formulas, graphs, conditional formatting, and data input restrictions to a spreadsheet. This system allows users to automatically perform the necessary spreadsheet operations simply by entering instructions in Japanese. The generation means also includes a function for customizing the spreadsheet's user interface and a function for using an external AI server to analyze the instructions. This makes spreadsheet use more intuitive and efficient, while significantly reducing the occurrence of errors and the effort required for debugging.

[0006] "Instructions in Japanese" are specific operations and requirements written by the user in Japanese.

[0007] A "generation means" is a program or system that has the function of analyzing instructions from a user and automatically generating numerical formulas, graphs, conditional formats, data input restrictions, etc. based on the instructions.

[0008] A "generated numerical formula" is a calculation formula in a spreadsheet that is generated by the generating means based on instructions from the user.

[0009] A "chart" is a visual element, such as a bar graph, line graph, or pie chart, that is used to visually display data in a spreadsheet.

[0010] "Conditional formatting" refers to rules for applying specific formatting depending on the value or state of cells in a spreadsheet.

[0011] "Data input restriction" is a function that restricts the type and format of data that can be entered into a specific range within a spreadsheet.

[0012] The "application means" is a program or system having a function for reflecting the numerical formulas, graphs, conditional formats, and data input restrictions generated by the generation means in a spreadsheet.

[0013] A "spreadsheet" is a tabular document with rows and columns, and software for entering, editing, and analyzing data. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] This invention relates to a system that converts Japanese language instructions into specific operations for a spreadsheet. This system includes a generating means and an applying means, and can automatically generate and apply formulas, graphs, conditional formatting, data input restrictions, etc. in a spreadsheet using Japanese language instructions.

[0036] Example of a system

[0037] System Overview

[0038] The server receives Japanese instructions from the user via a program that runs in conjunction with the spreadsheet. The terminal displays the Japanese instructions entered by the user and sends them to the server for analysis. Based on the analysis results, code for the appropriate operation is generated, sent back to the terminal, and applied to the spreadsheet.

[0039] Functions and Operation Procedures

[0040] 1. User operations

[0041] The user opens a spreadsheet and selects one of the items in the "AI Support Menu." This menu includes items such as "Generate Formulas," "Generate Graphs," "Apply Conditional Formatting," "Set Data Entry Restrictions," and "Enter Instructions." When the user selects "Enter Instructions," a prompt for entering instructions appears on the device.

[0042] 2. Inputting and retrieving Japanese instructions

[0043] The user inputs instructions in Japanese into the prompt, such as "Please set the formula A1+B1 in C1." This input information is sent to the server and used for subsequent processing.

[0044] 3. Parsing the instructions

[0045] The server uses the generation means to analyze the Japanese instructions received from the user. During the analysis, it identifies whether the instruction is a mathematical formula, graph, conditional formatting, or data input restriction. It then generates specific code according to the operation.

[0046] 4. Applying the generated code

[0047] The server generates a specific code based on the analysis and returns it to the device. The device receives this code and performs operations to apply formulas, charts, conditional formatting, and data entry restrictions to the appropriate locations in the spreadsheet. For example, if the input instruction is "Set the formula A1+B1 in C1," the formula =A1+B1 will be set in cell C1.

[0048] 5. Displaying the operation results

[0049] Once applied, users can review the results in the spreadsheet to ensure formulas are entered correctly, charts are generated as expected, and conditional formatting and data entry restrictions are applied.

[0050] Specific use cases

[0051] Specific usage examples are described below.

[0052] Example 1: Generating a formula

[0053] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0054] 2. Terminal: A prompt appears for you to enter instructions.

[0055] 3. User: Enter "Set formula A1+B1 in C1" in the prompt and press OK.

[0056] 4. Server: Analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[0057] 5. Terminal: Receive the generated code and set the formula "=A1+B1" in cell C1.

[0058] 6. User: Check the spreadsheet and verify that the formula is set correctly in cell C1.

[0059] Example 2: Generating a graph

[0060] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0061] 2. Terminal: A prompt appears for you to enter instructions.

[0062] 3. User: Enter "Create a column chart based on the data from A1 to B5" in the prompt and press OK.

[0063] 4. Server: Parses the received instructions and generates the code to generate the required graph.

[0064] 5. Terminal: Receive the generated code and create a column chart based on the data in the specified range.

[0065] 6. User: Review the spreadsheet and verify that the column chart was generated correctly.

[0066] In this way, with the present invention, various spreadsheet operations are automatically performed simply by the user inputting instructions in Japanese, allowing even beginners to work efficiently and accurately.

[0067] The processing flow will be explained below.

[0068] Step 1:

[0069] The server automatically executes the onOpen function when the spreadsheet is opened, which generates a custom menu, the "AI Support Menu," that the user can use and adds it to the user interface.

[0070] Step 2:

[0071] The user opens the spreadsheet and selects "Enter Instructions" from the "AI Support Menu."

[0072] Step 3:

[0073] The terminal displays a prompt for the user to enter instructions in Japanese.

[0074] Step 4:

[0075] The user enters instructions into the prompt (e.g., "Please set the formula A1+B1 in C1") and presses OK.

[0076] Step 5:

[0077] The terminal receives input from the user and transmits the instructions to the server.

[0078] Step 6:

[0079] The server uses a generation means to analyze the received Japanese instructions and transmits the instructions to an external AI server.

[0080] Step 7:

[0081] The server receives the parsed results and generates code for specific operations based on the instructions (e.g., generating the formula "=A1+B1").

[0082] Step 8:

[0083] The server returns the generated specific operation code to the terminal.

[0084] Step 9:

[0085] The terminal executes the received code and applies it to the spreadsheet, for example, by setting the formula =A1+B1 in cell C1.

[0086] Step 10:

[0087] The user reviews the resulting changes in the spreadsheet and ensures that formulas, charts, and conditional formatting are applied correctly.

[0088] This series of steps allows users to efficiently perform spreadsheet operations using only Japanese instructions.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] Conventional spreadsheet software lacks the functionality to automatically generate and apply formulas, graphs, conditional formatting, and data entry constraints based on instructions in Japanese. This makes it difficult for beginner users, in particular, to operate spreadsheets efficiently. Furthermore, the software lacks a flexible interface that allows users to intuitively perform a variety of operations. A system that solves this problem and allows users to operate spreadsheets easily and accurately is needed.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes a generating means for analyzing instructions in Japanese, an applying means for applying a numerical formula generated by the generating means to a spreadsheet software, an applying means for applying a graph generated by the generating means to the spreadsheet software, an applying means for applying a conditional format generated by the generating means to the spreadsheet software, an applying means for applying a data input constraint generated by the generating means to the spreadsheet software, and a means for receiving instructions from a user terminal and transmitting the analysis results to the user terminal, thereby making it possible to automatically perform various operations on a spreadsheet based on instructions in Japanese.

[0094] The "generation means" is a means for analyzing Japanese instructions and converting them into concrete operations such as mathematical formulas, graphs, conditional formatting, and data input constraints.

[0095] The "application means" is a means for applying the formulas, graphs, conditional formats, and data input constraints generated by the generation means to the spreadsheet software.

[0096] A "formula" is an expression that specifies a combination of calculations or values ​​to be entered into a cell in a spreadsheet.

[0097] A "graph" is a diagram used in spreadsheet software to visually represent data.

[0098] "Conditional formatting" is a feature in spreadsheet software that automatically changes the formatting of cells based on specific conditions.

[0099] "Data input constraints" are settings that limit the type and range of data that can be entered into cells in spreadsheet software.

[0100] A "user terminal" is a computer device or equipment used to operate spreadsheet software and input instructions in Japanese.

[0101] "Analysis results" refers to the specific operation content and code obtained by the generation means when it analyzes the Japanese instructions.

[0102] A "spreadsheet" is software for organizing, calculating, analyzing, and visualizing data.

[0103] A "generative AI model" is a model that uses artificial intelligence to analyze Japanese instructions and generate appropriate operation codes.

[0104] "Prompt" refers to the input message or screen that is displayed when the user enters instructions in Japanese.

[0105] This invention relates to a system that analyzes Japanese language instructions and automatically generates and applies formulas, graphs, conditional formatting, and data input constraints to spreadsheet software. This system includes a generating means and an applying means, and can operate spreadsheets based on Japanese language instructions entered by the user. Specific embodiments of the invention are described below.

[0106] Required Hardware and Software

[0107] User device: A device such as a computer, tablet, or smartphone.

[0108] Server: Cloud server or dedicated server.

[0109] Spreadsheet software: Software for organizing, calculating, analyzing, and visualizing data (e.g., Google Sheets, Microsoft Excel).

[0110] Generative AI model: An artificial intelligence model for analyzing Japanese instructions and generating appropriate operation codes.

[0111] System Operation Overview

[0112] The server receives Japanese instructions sent from the user's device and analyzes them using a generative AI model. Based on the analysis results, it generates specific formulas, graphs, conditional formatting, and data input constraints and returns them to the user's device. The user's device then applies the results to the spreadsheet software.

[0113] Usage example

[0114] Example 1: Generating a formula

[0115] 1. The user selects "Enter instructions" from the "AI Support Menu."

[0116] 2. The terminal will prompt you to enter instructions.

[0117] 3. The user enters "Please set the formula A1+B1 in C1" in the prompt and presses the submit button.

[0118] 4. The server analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[0119] 5. The terminal receives the generated code and sets the formula "=A1+B1" in cell C1.

[0120] 6. The user checks the spreadsheet and confirms that the formula "=A1+B1" is correctly set in cell C1.

[0121] Example 2: Generating a graph

[0122] 1. The user selects "Enter instructions" from the "AI Support Menu."

[0123] 2. The terminal will prompt you to enter instructions.

[0124] 3. The user enters "Create a column chart based on the data from A1 to B5" in the prompt and presses the submit button.

[0125] 4. The server parses the received instructions and generates the necessary graph generation code.

[0126] 5. The terminal receives the generated code and creates a column chart based on the data in the specified range.

[0127] 6. The user reviews the spreadsheet and verifies that the column chart was generated correctly.

[0128] This system allows users to automatically perform complex operations in spreadsheet software simply by inputting instructions in Japanese, enabling even beginners to work efficiently and accurately.

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Step 1:

[0131] The user opens a spreadsheet and selects "Enter instructions" from the "AI Support Menu." When the user selects an operation, a prompt for inputting instructions is displayed on the terminal. This prompt is an input interface that allows the user to enter specific instructions in Japanese. An example of input is "Please set the formula A1+B1 in C1." The output is a prompt screen that is displayed on the terminal.

[0132] Step 2:

[0133] The user inputs Japanese instructions into the displayed prompt and presses the send button. For example, the user can input instructions such as "Please set the formula A1+B1 in C1." The input is the Japanese instructions entered by the user, and the output is the instruction data sent from the user terminal to the server. In concrete terms, the prompt screen receives the user's input and sends it to the server via the network.

[0134] Step 3:

[0135] The server analyzes the received Japanese instructions. This analysis is performed using a generative AI model. The generative AI model analyzes the Japanese instructions and generates the appropriate operation code. The input is the Japanese instructions received by the server, and the output is the operation code (e.g., "=A1+B1") that is the result of the analysis. Specifically, the server passes the input instructions to the AI ​​model, which then analyzes the instructions and identifies the appropriate spreadsheet operation.

[0136] Step 4:

[0137] The server generates operation codes based on the analysis results. These operation codes represent the formulas, graphs, conditional formatting, and data input constraints to be applied to the spreadsheet. The input is the instructions analyzed by the AI ​​model, and the output is the specific operation code. As a specific operation, the server generates the specific spreadsheet operation code to be applied based on the analysis results returned by the AI ​​model.

[0138] Step 5:

[0139] The server sends the generated operation code to the terminal. The input is the operation code generated by the server, and the output is the operation code sent to the terminal. The specific operation is to send the code generated by the server back to the user's terminal via the network.

[0140] Step 6:

[0141] The terminal applies the operation code received from the server to the spreadsheet. For example, set the formula "=A1+B1" in cell C1. The input is the operation code received by the terminal, and the output is the formula and graph applied to the spreadsheet. Specifically, the terminal analyzes the operation code and performs the operation on the corresponding cell or range in the spreadsheet based on its contents.

[0142] Step 7:

[0143] The terminal reports the success of the operation to the server. The input is the result of application to the spreadsheet, and the output is a report of the success of the operation to the server. Specifically, the terminal checks the result of application to the spreadsheet, and if the operation was successful, it sends that information to the server.

[0144] Step 8:

[0145] The user checks the spreadsheet to ensure that their instructions have been executed correctly. For example, they check that the formula "=A1+B1" is set correctly in cell C1. The input is the user's visual confirmation, and the output is the visual confirmation result on the spreadsheet. Specifically, the user opens the spreadsheet and checks to see if the specified operations have been correctly reflected.

[0146] (Application example 1)

[0147] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0148] Modern spreadsheet operations are complex and time-consuming, especially when dealing with large amounts of data. It is particularly difficult for non-technical users and beginners to operate them efficiently. Furthermore, current systems often struggle to handle situations where quick on-site response is required, such as when it is difficult to operate while moving around or by voice input. There is a need for a system that can solve these issues and automate spreadsheet operations by simply inputting instructions via voice.

[0149] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0150] In this invention, the server includes a generating means for analyzing instructions in Japanese, an applying means for applying a numerical formula generated by the generating means to a spreadsheet, an applying means for applying a graph generated by the generating means to a spreadsheet, an applying means for applying a conditional format generated by the generating means to a spreadsheet, an applying means for applying a data input restriction generated by the generating means to a spreadsheet, a means for analyzing a user's voice instructions and automating spreadsheet operations, and a means for receiving the voice instructions via a smart device. This allows spreadsheet operations to be automated with simple voice input of instructions, enabling non-technical people and those in the field to respond quickly.

[0151] The "means for generating Japanese language instructions" is a means for analyzing Japanese language instructions input by a user and interpreting their contents.

[0152] The "means for applying the numerical formula generated by the generating means to the spreadsheet" is a means for inputting the numerical formula generated from the analyzed Japanese instructions into an appropriate cell of the spreadsheet.

[0153] The "means for applying the graph generated by the generating means to the spreadsheet" is a means for introducing the graph analyzed and generated from the Japanese instructions into the spreadsheet.

[0154] The "application means for applying the conditional formatting generated by the generation means to the spreadsheet" is a means for applying the conditional formatting generated based on the user's instructions to a specified range of the spreadsheet.

[0155] The "means for applying the data input restrictions generated by the generating means to the spreadsheet" is a means for setting the data input restrictions generated based on the user's Japanese instructions to the spreadsheet.

[0156] The "means for analyzing a user's voice instructions and automating spreadsheet operations" refers to a means for converting a user's voice instructions into text, analyzing it, and automatically operating a spreadsheet.

[0157] The "means for acquiring voice instructions via a smart device" refers to a means for collecting and analyzing voice instructions from a user using a smart device.

[0158] MODE FOR CARRYING OUT THE INVENTION

[0159] The present invention provides a system for automating spreadsheet operations using voice instructions in Japanese on a smart device. Specific embodiments of the invention are described below.

[0160] System configuration that analyzes voice instructions and performs spreadsheet operations

[0161] The system includes the following major components:

[0162] 1. Smart terminal: A device that receives user voice commands (e.g., smart glasses or head-mounted displays).

[0163] 2. Server: Hardware and software (e.g., Google Sheets API, Python) to parse the voice instructions and generate the appropriate spreadsheet operations.

[0164] 3. Generator: A means (e.g., speech_recognition library, AI server) for parsing Japanese instructions and applying numerical formulas, graphs, conditional formatting, and data entry control operations to the spreadsheet.

[0165] Program processing and the hardware and software used

[0166] 1. Smart Devices:

[0167] Hardware: Smart glasses and head-mounted displays.

[0168] Software: Speech recognition library (e.g. speech_recognition).

[0169] Processing: The smart device collects the user's voice commands and converts them into text data.

[0170] 2. Server:

[0171] Hardware: Standard server.

[0172] Software: Python, Google Sheets API, OAuth2 client.

[0173] process:

[0174] The text data generated by speech recognition is analyzed and Japanese instructions are interpreted.

[0175] Generates code for spreadsheet operations depending on the processing content.

[0176] Example of operation and prompt statement

[0177] Examples of operations include:

[0178] Example 1: Inventory check

[0179] 1. User: "Please check the stock of product code 12345."

[0180] 2. Server: Analyzes the instructions and uses the Google Sheets API to retrieve inventory information for the corresponding product code from the spreadsheet.

[0181] 3. Terminal: The acquired inventory information is displayed on the smart terminal.

[0182] Example 2: Inventory Update

[0183] 1. User: "Please update the inventory of product code 67890 to 20."

[0184] 2. Server: Parses the instruction and generates code to update the stock quantity in the appropriate spreadsheet cell to 20.

[0185] 3. Terminal: Perform the update operation and confirm that the inventory has been updated to 20.

[0186] Example prompt sentence:

[0187] "Please check stock availability for product code 12345"

[0188] "Please update the stock of product code 67890 to 20."

[0189] In this manner, the present invention allows a user to efficiently automate spreadsheet operations through voice instructions.

[0190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0191] Step 1:

[0192] The device receives voice instructions from a user via a smart device (e.g., smart glasses). The input is the user's voice instruction, and the output is voice data. This voice data is collected using the microphone of the smart device.

[0193] Step 2:

[0194] The device converts the acquired voice data into text data. Specifically, it uses a voice recognition library (e.g., speech_recognition) to analyze the voice data and generate text data. The input is voice data, and the output is text data.

[0195] Step 3:

[0196] The server receives the text data sent from the device and analyzes the user's instructions. The input is text data, and the output is the analyzed instructions. Natural language processing technology is used for the analysis to interpret the intent of the Japanese instructions.

[0197] Step 4:

[0198] The server generates code to perform the appropriate spreadsheet operations based on the parsed instructions. The input is the parsed instructions, and the output is the code to perform the spreadsheet operations. Specific operations include the use of algorithms to generate and apply formulas, graphs, conditional formatting, and data entry restrictions.

[0199] Step 5:

[0200] The server sends the generated code to the terminal. The input is the spreadsheet operation code, and the output is the transmitted data. The transmission is performed via data communication over a network.

[0201] Step 6:

[0202] The device applies the received spreadsheet operation code to the spreadsheet. The input is the spreadsheet operation code, and the output is the updated content of the spreadsheet. Specifically, the operation is performed on the specified cell or range of the spreadsheet using the Google Sheets API.

[0203] Step 7:

[0204] The device provides feedback to the user on the results of the spreadsheet updates. The input is the updated content of the spreadsheet, and the output is a display of the updated results. Specifically, the results are displayed on the smart device's display, and feedback is provided by voice or text.

[0205] Specific examples

[0206] Example 1: Inventory check

[0207] 1. User: "Please check the stock of product code 12345."

[0208] 2. Terminal: Converts voice instructions into text and sends it to the server.

[0209] 3. Server: Analyzes the instructions and uses the Google Sheets API to retrieve inventory information for the corresponding product code from the spreadsheet.

[0210] 4. Terminal: Display the retrieved inventory information to the user. It displays "There are 10 units of product code 12345 in stock."

[0211] Example 2: Inventory Update

[0212] 1. User: "Please update the inventory of product code 67890 to 20."

[0213] 2. Terminal: Converts voice instructions into text and sends it to the server.

[0214] 3. Server: Parses the instruction and generates code to update the stock quantity in the appropriate spreadsheet cell to 20.

[0215] 4. Terminal: Executes the update operation and displays the results to the user. It displays "The inventory of product code 67890 has been updated to 20."

[0216] Example prompt sentence:

[0217] "Please check stock availability for product code 12345"

[0218] "Please update the stock of product code 67890 to 20."

[0219] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0220] This invention relates to a system that converts Japanese language instructions into specific operations for a spreadsheet and combines it with an emotion engine that recognizes the user's emotions. The system includes a generation means, an application means, and an emotion engine, and automatically generates and applies formulas, graphs, conditional formatting, data input restrictions, etc. in a spreadsheet in response to Japanese language instructions, and also has a customization function based on the user's emotions.

[0221] System configuration

[0222] Example of a system

[0223] 1. User operations

[0224] The user opens a spreadsheet and selects one of the items in the "AI Support Menu." This menu includes items such as "Generate Formulas," "Generate Graphs," "Apply Conditional Formatting," "Set Data Input Restrictions," and "Enter Instructions." When the user selects "Enter Instructions," a prompt for entering instructions appears on the device.

[0225] 2. Inputting and retrieving Japanese instructions

[0226] The user inputs instructions in Japanese into the prompt, such as "Please set the formula A1+B1 in C1." This input information is sent to the server and used for subsequent processing.

[0227] 3. Parsing the instructions

[0228] The server uses the generation means to analyze the Japanese instructions received from the user. During the analysis, it identifies whether the instruction corresponds to a mathematical formula, graph, conditional formatting, or data input restriction. It then generates specific code according to the operation content.

[0229] 4. Emotion Recognition by Emotion Engine

[0230] The server uses an emotion engine to analyze the user's emotions when inputting instructions. For example, if the user is feeling stressed, the server recognizes that emotion and takes measures to reduce the user's workload.

[0231] 5. Applying the generated code

[0232] The server then returns the specific operation code it generated to the device. The device then receives this code and applies formulas, graphs, conditional formatting, and data entry restrictions to the appropriate locations in the spreadsheet. For example, if the input instruction is "Set the formula A1+B1 in C1," the formula =A1+B1 will be set in cell C1. In addition, based on the analysis results of the emotion engine, the device may change error messages to be more friendly, for example.

[0233] 6. Displaying the operation results

[0234] Once applied, users can review the results in their spreadsheet to ensure formulas are entered correctly, charts are generated as expected, conditional formatting and validation rules are applied, and any customizations made by the emotion engine are reflected.

[0235] Specific use cases

[0236] Example 1: Generating mathematical expressions and reflecting emotions

[0237] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0238] 2. Terminal: A prompt appears for you to enter instructions.

[0239] 3. User: Enter "Set formula A1+B1 in C1" in the prompt and press OK.

[0240] 4. Server: Analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[0241] 5. Server: Analyzes the user's voice and facial expressions when inputting and detects emotions.

[0242] 6. Server: If the emotion engine recognizes high stress, it generates program code to change the error message to a gentler expression.

[0243] 7. Terminal: Receive the generated code, set the formula "=A1+B1" in cell C1, and adjust how the error message is displayed.

[0244] 8. User: Check the spreadsheet and confirm that the formula in cell C1 is set correctly and that the customized error message is displayed.

[0245] Example 2: Graph generation and sentiment reflection

[0246] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0247] 2. Terminal: A prompt appears for you to enter instructions.

[0248] 3. User: Enter "Create a column chart based on the data from A1 to B5" in the prompt and press OK.

[0249] 4. Server: Parses the received instructions and generates the code to generate the required graph.

[0250] 5. Server: If the emotion engine recognizes that the user is highly excited, it generates program code to adjust the color and design of the chart.

[0251] 6. Terminal: Receive the generated code, create a column chart based on the specified range of data, and adjust the chart design.

[0252] 7. User: Review the spreadsheet to ensure the column chart was generated correctly and reflects the sentiment customizations.

[0253] In this way, the present invention analyzes the user's emotions and performs customization based on the analysis, thereby improving the efficiency of spreadsheet operations and providing a more comfortable operating environment for the user.

[0254] The processing flow will be explained below.

[0255] Step 1:

[0256] The server automatically executes the onOpen function when the spreadsheet is opened, which adds an "AI Support Menu" to the user interface.

[0257] Step 2:

[0258] The user opens the spreadsheet and selects "Enter Instructions" from the "AI Support Menu."

[0259] Step 3:

[0260] The terminal displays a prompt for the user to enter instructions in Japanese.

[0261] Step 4:

[0262] The user enters instructions into the prompt (e.g., "Set the formula A1+B1 in C1") and presses OK.

[0263] Step 5:

[0264] The terminal receives input from the user and transmits the instructions to the server.

[0265] Step 6:

[0266] The server uses a generation means to analyze the received Japanese instructions and transmits the instructions to an external AI server.

[0267] Step 7:

[0268] The server receives the analysis results from the AI ​​server and generates code for specific operations based on the instructions (e.g., generating the formula "=A1+B1").

[0269] Step 8:

[0270] The server uses an emotion engine to collect emotion data from the user when inputting commands. This emotion data is obtained by means of voice tone analysis, facial expression analysis, etc.

[0271] Step 9:

[0272] The server analyzes the acquired emotional data and identifies the user's emotional state. For example, it determines that the user is feeling stressed.

[0273] Step 10:

[0274] The server generates code to tailor the outcome of the operation based on the emotion, for example by customizing the wording of error messages to be more gentle.

[0275] Step 11:

[0276] The server returns the generated specific operation code and an adjustment code based on the emotion to the terminal.

[0277] Step 12:

[0278] The terminal executes the received code and applies it to the spreadsheet, for example, by setting the formula =A1+B1 in cell C1 and customizing the error message.

[0279] Step 13:

[0280] Users can review the results of their spreadsheet changes, ensuring that formulas, charts, and conditional formatting are applied correctly, and even view customized error messages.

[0281] This series of steps allows users to efficiently operate spreadsheets using only Japanese instructions, and furthermore, customization based on emotions provides a more comfortable operating environment.

[0282] Example 2

[0283] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0284] Existing spreadsheet operations are largely manual, requiring time and effort, especially when it comes to complex formulas, data entry restrictions, and conditional formatting. Furthermore, there are currently no methods to reduce the stress and discomfort users feel while using spreadsheets. This reduces user efficiency and makes operations more cumbersome, resulting in an uncomfortable working environment.

[0285] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0286] In this invention, the server includes a generating means for analyzing Japanese instructions, an applying means for applying the numerical formula generated by the generating means to the spreadsheet, an applying means for applying the graph generated by the generating means to the spreadsheet, an applying means for applying the conditional format generated by the generating means to the spreadsheet, an applying means for applying the numerical input restriction generated by the generating means to the spreadsheet, an emotion engine for analyzing the user's emotions, and a means for customizing the spreadsheet operation based on the analysis result of the emotion engine. This not only enables the user's instructions to be automatically reflected in the spreadsheet, but also enables customization according to the user's emotional state.

[0287] "Instructions in Japanese" refers to operational commands or requests written in Japanese.

[0288] "Generation means" refers to a device or program that has the function of analyzing input Japanese instructions and generating specific operation codes.

[0289] A "numeric formula" is a mathematical formula used in a spreadsheet to perform calculations.

[0290] A "spreadsheet" is an application that allows you to manage data in a tabular format and apply formulas, graphs, conditional formatting, etc.

[0291] The "application means" refers to a device or program that has the function of applying the generated numerical formulas, graphs, conditional formats, data input restrictions, etc. to a spreadsheet.

[0292] A "graph" is a tool used to visually represent data in a spreadsheet.

[0293] Conditional formatting is a feature that changes the display format of cells in a spreadsheet based on their values ​​or conditions.

[0294] "Data input restrictions" refers to a function that restricts the range and format of data entered into a spreadsheet.

[0295] An "emotion engine" is a device or program for analyzing a user's emotional state from their voice, facial expressions, etc.

[0296] "Means for customization" refers to a device or program that has the function of adjusting the operation and display of a spreadsheet based on the analysis results of the emotion engine.

[0297] "External AI Server" means a server equipped with external artificial intelligence used for parsing and generating instructions.

[0298] This invention is a system that converts Japanese instructions into specific operations for a spreadsheet and further customizes the operations by recognizing the user's emotions. This system includes a generation means, an application means, and an emotion engine, and automatically performs operations such as formulas, graphs, conditional formatting, and data input restrictions in a spreadsheet using Japanese instructions, and also has a customization function based on the user's emotions.

[0299] Hardware and software used

[0300] Hardware:

[0301] Device (PC, tablet, etc.)

[0302] server

[0303] software:

[0304] Spreadsheet applications (e.g. Google Sheets)

[0305] Emotion engine (e.g. IBM Watson)

[0306] Instruction analysis engine (natural language processing model, e.g. GPT)

[0307] Example of a system

[0308] 1. User Action:

[0309] The user opens the spreadsheet on their device and selects "Enter instructions" from the "AI Support Menu."

[0310] 2. Entering and retrieving Japanese instructions:

[0311] The user receives a prompt on the spreadsheet for inputting instructions, and enters specific instructions in Japanese, for example, "Please set the formula A1+B1 in C1." This instruction is then sent to the server.

[0312] 3. Parsing the instructions:

[0313] The server uses an instruction analysis engine to analyze the Japanese instructions sent by the user, and based on the analysis results, generates specific operation codes for generating formulas, creating graphs, applying conditional formatting, and restricting data entry.

[0314] 4. Emotion Recognition with Emotion Engine:

[0315] The server uses an emotion engine to analyze the user's emotions during input, for example, detecting stress or excitement from the user's voice and facial expressions.

[0316] 5. Apply the generated code:

[0317] The server sends the generated operation code and the emotion-based customization result to the device, and the device applies the specific operation to the spreadsheet, for example, setting the formula "=A1+B1" in cell C1, and adjusts the error message according to the user's emotion.

[0318] 6. Displaying the operation results:

[0319] Users can see the results right in the spreadsheet, ensuring formulas are entered correctly, charts are generated as expected, and conditional formatting and data entry restrictions are applied.

[0320] Specific use cases

[0321] Example 1: Generating mathematical expressions and reflecting emotions

[0322] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0323] 2. Terminal: A prompt appears for you to enter instructions.

[0324] 3. User: Enter "Set formula A1+B1 in C1" in the prompt and press OK.

[0325] 4. Server: Analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[0326] 5. Server: Analyzes the user's voice and facial expressions when inputting and detects emotions.

[0327] 6. Server: If the emotion engine recognizes high stress, it generates program code to change the error message to a gentler expression.

[0328] 7. Terminal: Receive the generated code, set the formula "=A1+B1" in cell C1, and adjust how the error message is displayed.

[0329] 8. User: Check the spreadsheet and confirm that the formula in cell C1 is set correctly and that the customized error message is displayed.

[0330] Example 2: Graph generation and sentiment reflection

[0331] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0332] 2. Terminal: Displays a prompt for input of instructions.

[0333] 3. User: Enter "Create a column chart based on the data from A1 to B5" in the prompt and press OK.

[0334] 4. Server: Parses the received instructions and creates the code to generate the required graph.

[0335] 5. Server: Create program code to adjust the color and design of the graph when the emotion engine recognizes that the user is highly excited.

[0336] 6. Terminal: Receive the generated code, create a column chart based on the specified range of data, and adjust the design.

[0337] 7. User: Review the spreadsheet and verify that the column chart is generated correctly and reflects the sentiment-based customizations.

[0338] Prompt Sentence Examples

[0339] Example 1 Formula Generation Prompt:

[0340] "Set the formula A1+B1 in C1"

[0341] Example 2 graph generation prompt:

[0342] "Create a column chart based on the data from A1 to B5."

[0343] In this way, the present invention analyzes the user's emotions and performs customization based on the analysis, thereby improving the efficiency of spreadsheet operations and providing a more comfortable operating environment for the user.

[0344] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0345] Program processing flow

[0346] Step 1: User interaction

[0347] explanation:

[0348] The user opens a spreadsheet application and selects "Enter Instructions" from the AI ​​Support menu.

[0349] Specific behavior:

[0350] The user opens the spreadsheet on a device such as a PC or tablet.

[0351] The device displays the "AI Support Menu" and waits for a selection from the user.

[0352] The user selects "Enter Instructions."

[0353] Input and Output:

[0354] Input: The action of a user interacting with a spreadsheet and selecting "Enter Instructions" from a menu.

[0355] Output: A terminal prompt will appear for you to enter instructions.

[0356] Step 2: Enter and retrieve Japanese instructions

[0357] explanation:

[0358] The user enters specific Japanese instructions for the spreadsheet into a prompt, which are then sent to the server.

[0359] Specific behavior:

[0360] The terminal displays prompts to the user for input of instructions.

[0361] The user enters "Please set the formula A1+B1 in C1" in the prompt and presses the submit button.

[0362] When the terminal receives the instruction, it transmits the instruction to the server.

[0363] Input and Output:

[0364] Input: The user enters specific Japanese instructions into the prompt and submits them.

[0365] Output: The user's Japanese instructions are sent to the server.

[0366] Step 3: Parse the instructions

[0367] explanation:

[0368] The server analyzes the Japanese instructions sent by the user and generates the required specific operation codes.

[0369] Specific behavior:

[0370] The server uses a generative AI model (e.g., GPT) to parse the received Japanese instructions.

[0371] The server analyzes the instruction and determines whether it is a formula, a graph, conditional formatting, or data input restrictions.

[0372] The server generates specific code for each operation. For example, if you request "Set the formula A1+B1 in C1," it generates the formula "=A1+B1."

[0373] Input and Output:

[0374] Input: The user's Japanese instructions sent to the server.

[0375] Output: Specific operation code based on the analysis results.

[0376] Step 4: Emotion Recognition with the Emotion Engine

[0377] explanation:

[0378] The server uses an emotion engine to analyze the user's emotions when inputting Japanese commands.

[0379] Specific behavior:

[0380] The server uses an emotion engine (e.g., IBM Watson) to analyze emotions from the user's voice and facial expressions.

[0381] The server recognizes the user's emotional state, such as whether they are stressed or excited.

[0382] Input and Output:

[0383] Input: User's voice and facial expression data when entering Japanese instructions.

[0384] Output: User's emotional state (e.g., stress, excitement).

[0385] Step 5: Apply the generated code

[0386] explanation:

[0387] The server generates specific operation codes, sends them to the device, and applies emotion-based customization to the spreadsheet.

[0388] Specific behavior:

[0389] The server transmits the generated operation code to the terminal.

[0390] The terminal applies the received operation code to the spreadsheet. For example, it sets the formula "=A1+B1" in cell C1.

[0391] The device reflects the results of customization by the emotion engine, for example, by changing error messages to more gentle expressions.

[0392] Input and Output:

[0393] Input: Operation code sent from the server and analysis results of the emotion engine.

[0394] Output: The state in which a specific operation is performed on the spreadsheet.

[0395] Step 6: View the operation results

[0396] explanation:

[0397] The user reviews the results on a spreadsheet.

[0398] Specific behavior:

[0399] The user checks the results of the operation in the spreadsheet.

[0400] The device will show you whether formulas are entered correctly, charts are generated as expected, and conditional formatting and input restrictions are applied.

[0401] The user also checks the results of customization by the emotion engine, for example, checking that the error message has been changed to a more gentle expression.

[0402] Input and Output:

[0403] Input: The result of an operation performed on a spreadsheet.

[0404] Output: A spreadsheet that reflects the results of the operation, and a customized section based on the user's emotions.

[0405] (Application example 2)

[0406] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0407] Conventional spreadsheet operations require users to manually set formulas and graphs, apply conditional formatting, and restrict data input, which is time-consuming and labor-intensive. Furthermore, they fail to take into account the emotional state of the user, which can cause excessive stress. In particular, systems supporting operations such as factory robots require fast and accurate data processing, so systems that are both efficient and considerate of the user's emotions are needed.

[0408] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a generating means for analyzing Japanese instructions, an applying means for applying a numerical formula generated by the generating means to a spreadsheet, an applying means for applying a graph generated by the generating means to a spreadsheet, an applying means for applying a conditional format generated by the generating means to the spreadsheet, an applying means for applying a data input restriction generated by the generating means to the spreadsheet, and a means for analyzing a user's emotions using an emotion analyzing means and customizing according to the results. This makes it possible to improve the efficiency of operations on the spreadsheet and provide a comfortable operating environment that takes user emotions into consideration.

[0409] "Instructions in Japanese" refers to requests from users regarding operations and settings written in Japanese as a natural language.

[0410] "Generation means" refers to equipment or software that has the function of analyzing Japanese instructions and generating numerical formulas, graphs, conditional formatting, and data input restrictions based on them.

[0411] "Application means" refers to equipment or software having a function for applying the formulas, graphs, conditional formatting, and data input restrictions generated by the generation means to a spreadsheet.

[0412] "Emotion analysis means" refers to equipment or software that has the function of recognizing emotions by analyzing a user's facial expressions, voice, actions, etc.

[0413] "Server" refers to a computer system that has the role of integrating the generation means, application means, emotion analysis means, etc., and provides the computational resources to execute these functions.

[0414] A "spreadsheet" is an electronic spreadsheet application that allows you to enter data into units called cells and manage and analyze the data using numerical formulas and graphs.

[0415] "Customization" refers to improving the comfort and efficiency of operation by changing and adjusting the operation screen and feedback content according to the user's emotions and usage situation.

[0416] This invention relates to a system that analyzes instructions in Japanese and automatically performs specific operations on a spreadsheet, and further has the function of analyzing the user's emotions and customizing the operations.

[0417] The system consists of a server, a terminal, a generation means, an application means, and an emotion analysis means. The server processes various data and is responsible for running the generation means, application means, and emotion analysis means. The terminal provides an interface for user instruction input and displaying operation results.

[0418] Generating a Program

[0419] The system program performs the following processing.

[0420] 1. Analysis of Japanese instructions

[0421] The server uses a generation means to analyze the Japanese instructions received from the user, and the generation means uses natural language processing technology to identify the content of the instructions and generate numerical expressions, graphs, conditional formats, and data input restrictions.

[0422] 2. Applying generated content

[0423] Formulas and graphs generated on the server are reflected in the spreadsheet through application means, such as entering a formula in a cell or generating a graph in a specified range.

[0424] 3. Emotion analysis

[0425] The emotion analysis means analyzes the user's voice and facial expressions to recognize their emotional state at that time. The server adjusts the response to operations and the expression of error messages based on the analysis results.

[0426] Hardware and software used

[0427] Hardware

[0428] Factory robots: Responsible for collecting data and inputting it into spreadsheets.

[0429] Facial recognition camera: Used to capture the user's facial expressions in real time.

[0430] PC or Server: Processes data and manages the system.

[0431] software

[0432] Python and related libraries (pandas, openpyxl, etc.): Used for data processing and analysis.

[0433] Natural Language Processing (NLP) models: used to parse Japanese instructions, including generative AI models.

[0434] EmotionRecognizer: A library for analyzing user emotions.

[0435] Specific examples

[0436] For example, the following operations are specifically performed.

[0437] Data Entry Example

[0438] When the user instructs the terminal, "Please input the temperature sensor data into A1," the server analyzes the instruction and inputs the temperature data into the specified cell through the application means.

[0439] Formula generation example

[0440] When the user instructs "Calculate the average value of B5 to B10 in D1," the server generates the formula =AVERAGE(B5:B10) using the generating means and applies it to cell D1 through the applying means.

[0441] Graph Generation Example

[0442] When a user instructs the server to "create a line graph based on the data in columns A and B," the server generates a line graph based on the data in the specified range and reflects it in the spreadsheet.

[0443] Example of customization using sentiment analysis

[0444] If the camera analyzes the user's facial expression as stressed, the server changes the error message to a softer one, such as "Please check again."

[0445] Prompt Sentence Examples

[0446] Below is an example of a prompt sentence entered into the system.

[0447] "Enter the temperature sensor data in A1"

[0448] "Calculate the average of B5 to B10 in D1."

[0449] "Create a line graph based on the data in columns A and B."

[0450] Please change the "An error has occurred" message to a more friendly one.

[0451] In this way, the present invention is a system that can automate spreadsheet operations while taking into consideration the user's feelings.

[0452] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0453] Step 1: Enter Japanese instructions

[0454] ---

[0455] The terminal receives instructions in Japanese from the user. The user inputs instructions in Japanese into the prompts displayed on the terminal. Examples of such input include "Enter the temperature sensor data in A1" and "Calculate the average value of B5 to B10 in D1."

[0456] Input: Japanese instructions from the user

[0457] Output: Japanese instruction data

[0458] Specific operation: The user enters instructions in Japanese into the input field on the terminal and presses the send button.

[0459] Step 2: Send instructions in Japanese

[0460] ---

[0461] The terminal sends the input Japanese instructions to the server, where this data is processed for analysis.

[0462] Input: Japanese instruction data

[0463] Output: Request to server

[0464] Specific operation: The terminal sends the Japanese instructions entered by the user to the server as an HTTP request.

[0465] Step 3: Parse the instructions

[0466] ---

[0467] The server analyzes the received Japanese instructions using a generation means, which uses a natural language processing model to identify the content of the instructions (numeric expressions, graphs, conditional formatting, data input restrictions).

[0468] Input: Japanese instruction data

[0469] Output: Analysis results (formulas, graph generation code, conditional formatting, data entry restrictions)

[0470] Specific Operation: The server parses the instruction using a natural language processing (NLP) model and generates a corresponding operation code.

[0471] Step 4: Sentiment Analysis

[0472] ---

[0473] The server analyzes the user's emotions using an emotion analysis engine, which determines the user's emotional state based on data acquired from a facial recognition camera and a voice input device.

[0474] Input: User's voice data and facial expression data

[0475] Output: Emotion analysis results (stress level, etc.)

[0476] Specific operation: The server uses emotion analysis means to analyze the user's voice and facial expressions to determine their stress level and emotional state.

[0477] Step 5: Apply the generated content

[0478] ---

[0479] The server applies the formulas, graphs, conditional formatting, and data entry restrictions generated based on the analysis results to the spreadsheet, and these contents are reflected in the spreadsheet through the application means.

[0480] Input: Analysis results and sentiment analysis results

[0481] Output: Updated data in the spreadsheet

[0482] Specific operations: The server applies the generated formulas, graphs, conditional formatting, and data entry restrictions to the spreadsheet using the application means, and changes the wording of error messages as necessary. The server also opens an existing spreadsheet file and inserts the corresponding data, formulas, and graphs into the specified cells.

[0483] Step 6: View the operation results

[0484] ---

[0485] The terminal displays the results of the spreadsheet operations based on the updated data received from the server, allowing the user to check the accuracy of the generated formulas and graphs.

[0486] Input: Spreadsheet update data

[0487] Output: Display of the updated spreadsheet

[0488] Specific behavior: The device redraws the spreadsheet based on the updated data received from the server and displays the results for the user to review.

[0489] The above are the processing steps of the system program that realizes the application example.

[0490] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0491] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0492] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0493] [Second embodiment]

[0494] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0495] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0496] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0497] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0498] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0499] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0500] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0501] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0502] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0503] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0504] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0505] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0506] This invention relates to a system that converts Japanese language instructions into specific operations for a spreadsheet. This system includes a generating means and an applying means, and can automatically generate and apply formulas, graphs, conditional formatting, data input restrictions, etc. in a spreadsheet using Japanese language instructions.

[0507] Example of a system

[0508] System Overview

[0509] The server receives Japanese instructions from the user via a program that runs in conjunction with the spreadsheet. The terminal displays the Japanese instructions entered by the user and sends them to the server for analysis. Based on the analysis results, code for the appropriate operation is generated, sent back to the terminal, and applied to the spreadsheet.

[0510] Functions and Operation Procedures

[0511] 1. User operations

[0512] The user opens a spreadsheet and selects one of the items in the "AI Support Menu." This menu includes items such as "Generate Formulas," "Generate Graphs," "Apply Conditional Formatting," "Set Data Entry Restrictions," and "Enter Instructions." When the user selects "Enter Instructions," a prompt for entering instructions appears on the device.

[0513] 2. Inputting and retrieving Japanese instructions

[0514] The user inputs instructions in Japanese into the prompt, such as "Please set the formula A1+B1 in C1." This input information is sent to the server and used for subsequent processing.

[0515] 3. Parsing the instructions

[0516] The server uses the generation means to analyze the Japanese instructions received from the user. During the analysis, it identifies whether the instruction is a mathematical formula, graph, conditional formatting, or data input restriction. It then generates specific code according to the operation.

[0517] 4. Applying the generated code

[0518] The server generates a specific code based on the analysis and returns it to the device. The device receives this code and performs operations to apply formulas, charts, conditional formatting, and data entry restrictions to the appropriate locations in the spreadsheet. For example, if the input instruction is "Set the formula A1+B1 in C1," the formula =A1+B1 will be set in cell C1.

[0519] 5. Displaying the operation results

[0520] Once applied, users can review the results in the spreadsheet to ensure formulas are entered correctly, charts are generated as expected, and conditional formatting and data entry restrictions are applied.

[0521] Specific use cases

[0522] Specific usage examples are described below.

[0523] Example 1: Generating a formula

[0524] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0525] 2. Terminal: A prompt appears for you to enter instructions.

[0526] 3. User: Enter "Set formula A1+B1 in C1" in the prompt and press OK.

[0527] 4. Server: Analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[0528] 5. Terminal: Receive the generated code and set the formula "=A1+B1" in cell C1.

[0529] 6. User: Check the spreadsheet and verify that the formula is set correctly in cell C1.

[0530] Example 2: Generating a graph

[0531] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0532] 2. Terminal: A prompt appears for you to enter instructions.

[0533] 3. User: Enter "Create a column chart based on the data from A1 to B5" in the prompt and press OK.

[0534] 4. Server: Parses the received instructions and generates the code to generate the required graph.

[0535] 5. Terminal: Receive the generated code and create a column chart based on the data in the specified range.

[0536] 6. User: Review the spreadsheet and verify that the column chart was generated correctly.

[0537] In this way, with the present invention, various spreadsheet operations are automatically performed simply by the user inputting instructions in Japanese, allowing even beginners to work efficiently and accurately.

[0538] The processing flow will be explained below.

[0539] Step 1:

[0540] The server automatically executes the onOpen function when the spreadsheet is opened, which generates a custom menu, the "AI Support Menu," that the user can use and adds it to the user interface.

[0541] Step 2:

[0542] The user opens the spreadsheet and selects "Enter Instructions" from the "AI Support Menu."

[0543] Step 3:

[0544] The terminal displays a prompt for the user to enter instructions in Japanese.

[0545] Step 4:

[0546] The user enters instructions into the prompt (e.g., "Please set the formula A1+B1 in C1") and presses OK.

[0547] Step 5:

[0548] The terminal receives input from the user and transmits the instructions to the server.

[0549] Step 6:

[0550] The server uses a generation means to analyze the received Japanese instructions and transmits the instructions to an external AI server.

[0551] Step 7:

[0552] The server receives the parsed results and generates code for specific operations based on the instructions (e.g., generating the formula "=A1+B1").

[0553] Step 8:

[0554] The server returns the generated specific operation code to the terminal.

[0555] Step 9:

[0556] The terminal executes the received code and applies it to the spreadsheet, for example, by setting the formula =A1+B1 in cell C1.

[0557] Step 10:

[0558] The user reviews the resulting changes in the spreadsheet and ensures that formulas, charts, and conditional formatting are applied correctly.

[0559] This series of steps allows users to efficiently perform spreadsheet operations using only Japanese instructions.

[0560] Example 1

[0561] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0562] Conventional spreadsheet software lacks the functionality to automatically generate and apply formulas, graphs, conditional formatting, and data entry constraints based on instructions in Japanese. This makes it difficult for beginner users, in particular, to operate spreadsheets efficiently. Furthermore, the software lacks a flexible interface that allows users to intuitively perform a variety of operations. A system that solves this problem and allows users to operate spreadsheets easily and accurately is needed.

[0563] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0564] In this invention, the server includes a generating means for analyzing instructions in Japanese, an applying means for applying a numerical formula generated by the generating means to a spreadsheet software, an applying means for applying a graph generated by the generating means to the spreadsheet software, an applying means for applying a conditional format generated by the generating means to the spreadsheet software, an applying means for applying a data input constraint generated by the generating means to the spreadsheet software, and a means for receiving instructions from a user terminal and transmitting the analysis results to the user terminal, thereby making it possible to automatically perform various operations on a spreadsheet based on instructions in Japanese.

[0565] The "generation means" is a means for analyzing Japanese instructions and converting them into concrete operations such as mathematical formulas, graphs, conditional formatting, and data input constraints.

[0566] The "application means" is a means for applying the formulas, graphs, conditional formats, and data input constraints generated by the generation means to the spreadsheet software.

[0567] A "formula" is an expression that specifies a combination of calculations or values ​​to be entered into a cell in a spreadsheet.

[0568] A "graph" is a diagram used in spreadsheet software to visually represent data.

[0569] "Conditional formatting" is a feature in spreadsheet software that automatically changes the formatting of cells based on specific conditions.

[0570] "Data input constraints" are settings that limit the type and range of data that can be entered into cells in spreadsheet software.

[0571] A "user terminal" is a computer device or equipment used to operate spreadsheet software and input instructions in Japanese.

[0572] "Analysis results" refers to the specific operation content and code obtained by the generation means when it analyzes the Japanese instructions.

[0573] A "spreadsheet" is software for organizing, calculating, analyzing, and visualizing data.

[0574] A "generative AI model" is a model that uses artificial intelligence to analyze Japanese instructions and generate appropriate operation codes.

[0575] "Prompt" refers to the input message or screen that is displayed when the user enters instructions in Japanese.

[0576] This invention relates to a system that analyzes Japanese language instructions and automatically generates and applies formulas, graphs, conditional formatting, and data input constraints to spreadsheet software. This system includes a generating means and an applying means, and can operate spreadsheets based on Japanese language instructions entered by the user. Specific embodiments of the invention are described below.

[0577] Required Hardware and Software

[0578] User device: A device such as a computer, tablet, or smartphone.

[0579] Server: Cloud server or dedicated server.

[0580] Spreadsheet software: Software for organizing, calculating, analyzing, and visualizing data (e.g., Google Sheets, Microsoft Excel).

[0581] Generative AI model: An artificial intelligence model for analyzing Japanese instructions and generating appropriate operation codes.

[0582] System Operation Overview

[0583] The server receives Japanese instructions sent from the user's device and analyzes them using a generative AI model. Based on the analysis results, it generates specific formulas, graphs, conditional formatting, and data input constraints and returns them to the user's device. The user's device then applies the results to the spreadsheet software.

[0584] Usage example

[0585] Example 1: Generating a formula

[0586] 1. The user selects "Enter instructions" from the "AI Support Menu."

[0587] 2. The terminal will prompt you to enter instructions.

[0588] 3. The user enters "Please set the formula A1+B1 in C1" in the prompt and presses the submit button.

[0589] 4. The server analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[0590] 5. The terminal receives the generated code and sets the formula "=A1+B1" in cell C1.

[0591] 6. The user checks the spreadsheet and confirms that the formula "=A1+B1" is correctly set in cell C1.

[0592] Example 2: Generating a graph

[0593] 1. The user selects "Enter instructions" from the "AI Support Menu."

[0594] 2. The terminal will prompt you to enter instructions.

[0595] 3. The user enters "Create a column chart based on the data from A1 to B5" in the prompt and presses the submit button.

[0596] 4. The server parses the received instructions and generates the necessary graph generation code.

[0597] 5. The terminal receives the generated code and creates a column chart based on the data in the specified range.

[0598] 6. The user reviews the spreadsheet and verifies that the column chart was generated correctly.

[0599] This system allows users to automatically perform complex operations in spreadsheet software simply by inputting instructions in Japanese, enabling even beginners to work efficiently and accurately.

[0600] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0601] Step 1:

[0602] The user opens a spreadsheet and selects "Enter instructions" from the "AI Support Menu." When the user selects an operation, a prompt for inputting instructions is displayed on the terminal. This prompt is an input interface that allows the user to enter specific instructions in Japanese. An example of input is "Please set the formula A1+B1 in C1." The output is a prompt screen that is displayed on the terminal.

[0603] Step 2:

[0604] The user inputs Japanese instructions into the displayed prompt and presses the send button. For example, the user can input instructions such as "Please set the formula A1+B1 in C1." The input is the Japanese instructions entered by the user, and the output is the instruction data sent from the user terminal to the server. In concrete terms, the prompt screen receives the user's input and sends it to the server via the network.

[0605] Step 3:

[0606] The server analyzes the received Japanese instructions. This analysis is performed using a generative AI model. The generative AI model analyzes the Japanese instructions and generates the appropriate operation code. The input is the Japanese instructions received by the server, and the output is the operation code (e.g., "=A1+B1") that is the result of the analysis. Specifically, the server passes the input instructions to the AI ​​model, which then analyzes the instructions and identifies the appropriate spreadsheet operation.

[0607] Step 4:

[0608] The server generates operation codes based on the analysis results. These operation codes represent the formulas, graphs, conditional formatting, and data input constraints to be applied to the spreadsheet. The input is the instructions analyzed by the AI ​​model, and the output is the specific operation code. As a specific operation, the server generates the specific spreadsheet operation code to be applied based on the analysis results returned by the AI ​​model.

[0609] Step 5:

[0610] The server sends the generated operation code to the terminal. The input is the operation code generated by the server, and the output is the operation code sent to the terminal. The specific operation is to send the code generated by the server back to the user's terminal via the network.

[0611] Step 6:

[0612] The terminal applies the operation code received from the server to the spreadsheet. For example, set the formula "=A1+B1" in cell C1. The input is the operation code received by the terminal, and the output is the formula and graph applied to the spreadsheet. Specifically, the terminal analyzes the operation code and performs the operation on the corresponding cell or range in the spreadsheet based on its contents.

[0613] Step 7:

[0614] The terminal reports the success of the operation to the server. The input is the result of application to the spreadsheet, and the output is a report of the success of the operation to the server. Specifically, the terminal checks the result of application to the spreadsheet, and if the operation was successful, it sends that information to the server.

[0615] Step 8:

[0616] The user checks the spreadsheet to ensure that their instructions have been executed correctly. For example, they check that the formula "=A1+B1" is set correctly in cell C1. The input is the user's visual confirmation, and the output is the visual confirmation result on the spreadsheet. Specifically, the user opens the spreadsheet and checks to see if the specified operations have been correctly reflected.

[0617] (Application example 1)

[0618] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0619] Modern spreadsheet operations are complex and time-consuming, especially when dealing with large amounts of data. It is particularly difficult for non-technical users and beginners to operate them efficiently. Furthermore, current systems often struggle to handle situations where quick on-site response is required, such as when it is difficult to operate while moving around or by voice input. There is a need for a system that can solve these issues and automate spreadsheet operations by simply inputting instructions via voice.

[0620] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0621] In this invention, the server includes a generating means for analyzing instructions in Japanese, an applying means for applying a numerical formula generated by the generating means to a spreadsheet, an applying means for applying a graph generated by the generating means to a spreadsheet, an applying means for applying a conditional format generated by the generating means to a spreadsheet, an applying means for applying a data input restriction generated by the generating means to a spreadsheet, a means for analyzing a user's voice instructions and automating spreadsheet operations, and a means for receiving the voice instructions via a smart device. This allows spreadsheet operations to be automated with simple voice input of instructions, enabling non-technical people and those in the field to respond quickly.

[0622] The "means for generating Japanese language instructions" is a means for analyzing Japanese language instructions input by a user and interpreting their contents.

[0623] The "means for applying the numerical formula generated by the generating means to the spreadsheet" is a means for inputting the numerical formula generated from the analyzed Japanese instructions into an appropriate cell of the spreadsheet.

[0624] The "means for applying the graph generated by the generating means to the spreadsheet" is a means for introducing the graph analyzed and generated from the Japanese instructions into the spreadsheet.

[0625] The "application means for applying the conditional formatting generated by the generation means to the spreadsheet" is a means for applying the conditional formatting generated based on the user's instructions to a specified range of the spreadsheet.

[0626] The "means for applying the data input restrictions generated by the generating means to the spreadsheet" is a means for setting the data input restrictions generated based on the user's Japanese instructions to the spreadsheet.

[0627] The "means for analyzing a user's voice instructions and automating spreadsheet operations" refers to a means for converting a user's voice instructions into text, analyzing it, and automatically operating a spreadsheet.

[0628] The "means for acquiring voice instructions via a smart device" refers to a means for collecting and analyzing voice instructions from a user using a smart device.

[0629] MODE FOR CARRYING OUT THE INVENTION

[0630] The present invention provides a system for automating spreadsheet operations using voice instructions in Japanese on a smart device. Specific embodiments of the invention are described below.

[0631] System configuration that analyzes voice instructions and performs spreadsheet operations

[0632] The system includes the following major components:

[0633] 1. Smart terminal: A device that receives user voice commands (e.g., smart glasses or head-mounted displays).

[0634] 2. Server: Hardware and software (e.g., Google Sheets API, Python) to parse the voice instructions and generate the appropriate spreadsheet operations.

[0635] 3. Generator: A means (e.g., speech_recognition library, AI server) for parsing Japanese instructions and applying numerical formulas, graphs, conditional formatting, and data entry control operations to the spreadsheet.

[0636] Program processing and the hardware and software used

[0637] 1. Smart Devices:

[0638] Hardware: Smart glasses and head-mounted displays.

[0639] Software: Speech recognition library (e.g. speech_recognition).

[0640] Processing: The smart device collects the user's voice commands and converts them into text data.

[0641] 2. Server:

[0642] Hardware: Standard server.

[0643] Software: Python, Google Sheets API, OAuth2 client.

[0644] process:

[0645] The text data generated by speech recognition is analyzed and Japanese instructions are interpreted.

[0646] Generates code for spreadsheet operations depending on the processing content.

[0647] Example of operation and prompt statement

[0648] Examples of operations include:

[0649] Example 1: Inventory check

[0650] 1. User: "Please check the stock of product code 12345."

[0651] 2. Server: Analyzes the instructions and uses the Google Sheets API to retrieve inventory information for the corresponding product code from the spreadsheet.

[0652] 3. Terminal: The acquired inventory information is displayed on the smart terminal.

[0653] Example 2: Inventory Update

[0654] 1. User: "Please update the inventory of product code 67890 to 20."

[0655] 2. Server: Parses the instruction and generates code to update the stock quantity in the appropriate spreadsheet cell to 20.

[0656] 3. Terminal: Perform the update operation and confirm that the inventory has been updated to 20.

[0657] Example prompt sentence:

[0658] "Please check stock availability for product code 12345"

[0659] "Please update the stock of product code 67890 to 20."

[0660] In this manner, the present invention allows a user to efficiently automate spreadsheet operations through voice instructions.

[0661] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0662] Step 1:

[0663] The device receives voice instructions from a user via a smart device (e.g., smart glasses). The input is the user's voice instruction, and the output is voice data. This voice data is collected using the microphone of the smart device.

[0664] Step 2:

[0665] The device converts the acquired voice data into text data. Specifically, it uses a voice recognition library (e.g., speech_recognition) to analyze the voice data and generate text data. The input is voice data, and the output is text data.

[0666] Step 3:

[0667] The server receives the text data sent from the device and analyzes the user's instructions. The input is text data, and the output is the analyzed instructions. Natural language processing technology is used for the analysis to interpret the intent of the Japanese instructions.

[0668] Step 4:

[0669] The server generates code to perform the appropriate spreadsheet operations based on the parsed instructions. The input is the parsed instructions, and the output is the code to perform the spreadsheet operations. Specific operations include the use of algorithms to generate and apply formulas, graphs, conditional formatting, and data entry restrictions.

[0670] Step 5:

[0671] The server sends the generated code to the terminal. The input is the spreadsheet operation code, and the output is the transmitted data. The transmission is performed via data communication over a network.

[0672] Step 6:

[0673] The device applies the received spreadsheet operation code to the spreadsheet. The input is the spreadsheet operation code, and the output is the updated content of the spreadsheet. Specifically, the operation is performed on the specified cell or range of the spreadsheet using the Google Sheets API.

[0674] Step 7:

[0675] The device provides feedback to the user on the results of the spreadsheet updates. The input is the updated content of the spreadsheet, and the output is a display of the updated results. Specifically, the results are displayed on the smart device's display, and feedback is provided by voice or text.

[0676] Specific examples

[0677] Example 1: Inventory check

[0678] 1. User: "Please check the stock of product code 12345."

[0679] 2. Terminal: Converts voice instructions into text and sends it to the server.

[0680] 3. Server: Analyzes the instructions and uses the Google Sheets API to retrieve inventory information for the corresponding product code from the spreadsheet.

[0681] 4. Terminal: Display the retrieved inventory information to the user. It displays "There are 10 units of product code 12345 in stock."

[0682] Example 2: Inventory Update

[0683] 1. User: "Please update the inventory of product code 67890 to 20."

[0684] 2. Terminal: Converts voice instructions into text and sends it to the server.

[0685] 3. Server: Parses the instruction and generates code to update the stock quantity in the appropriate spreadsheet cell to 20.

[0686] 4. Terminal: Executes the update operation and displays the results to the user. It displays "The inventory of product code 67890 has been updated to 20."

[0687] Example prompt sentence:

[0688] "Please check stock availability for product code 12345"

[0689] "Please update the stock of product code 67890 to 20."

[0690] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0691] This invention relates to a system that converts Japanese language instructions into specific operations for a spreadsheet and combines it with an emotion engine that recognizes the user's emotions. The system includes a generation means, an application means, and an emotion engine, and automatically generates and applies formulas, graphs, conditional formatting, data input restrictions, etc. in a spreadsheet in response to Japanese language instructions, and also has a customization function based on the user's emotions.

[0692] System configuration

[0693] Example of a system

[0694] 1. User operations

[0695] The user opens a spreadsheet and selects one of the items in the "AI Support Menu." This menu includes items such as "Generate Formulas," "Generate Graphs," "Apply Conditional Formatting," "Set Data Input Restrictions," and "Enter Instructions." When the user selects "Enter Instructions," a prompt for entering instructions appears on the device.

[0696] 2. Inputting and retrieving Japanese instructions

[0697] The user inputs instructions in Japanese into the prompt, such as "Please set the formula A1+B1 in C1." This input information is sent to the server and used for subsequent processing.

[0698] 3. Parsing the instructions

[0699] The server uses the generation means to analyze the Japanese instructions received from the user. During the analysis, it identifies whether the instruction corresponds to a mathematical formula, graph, conditional formatting, or data input restriction. It then generates specific code according to the operation content.

[0700] 4. Emotion Recognition by Emotion Engine

[0701] The server uses an emotion engine to analyze the user's emotions when inputting instructions. For example, if the user is feeling stressed, the server recognizes that emotion and takes measures to reduce the user's workload.

[0702] 5. Applying the generated code

[0703] The server then returns the specific operation code it generated to the device. The device then receives this code and applies formulas, graphs, conditional formatting, and data entry restrictions to the appropriate locations in the spreadsheet. For example, if the input instruction is "Set the formula A1+B1 in C1," the formula =A1+B1 will be set in cell C1. In addition, based on the analysis results of the emotion engine, the device may change error messages to be more friendly, for example.

[0704] 6. Displaying the operation results

[0705] Once applied, users can review the results in their spreadsheet to ensure formulas are entered correctly, charts are generated as expected, conditional formatting and validation rules are applied, and any customizations made by the emotion engine are reflected.

[0706] Specific use cases

[0707] Example 1: Generating mathematical expressions and reflecting emotions

[0708] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0709] 2. Terminal: A prompt appears for you to enter instructions.

[0710] 3. User: Enter "Set formula A1+B1 in C1" in the prompt and press OK.

[0711] 4. Server: Analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[0712] 5. Server: Analyzes the user's voice and facial expressions when inputting and detects emotions.

[0713] 6. Server: If the emotion engine recognizes high stress, it generates program code to change the error message to a gentler expression.

[0714] 7. Terminal: Receive the generated code, set the formula "=A1+B1" in cell C1, and adjust how the error message is displayed.

[0715] 8. User: Check the spreadsheet and confirm that the formula in cell C1 is set correctly and that the customized error message is displayed.

[0716] Example 2: Graph generation and sentiment reflection

[0717] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0718] 2. Terminal: A prompt appears for you to enter instructions.

[0719] 3. User: Enter "Create a column chart based on the data from A1 to B5" in the prompt and press OK.

[0720] 4. Server: Parses the received instructions and generates the code to generate the required graph.

[0721] 5. Server: If the emotion engine recognizes that the user is highly excited, it generates program code to adjust the color and design of the chart.

[0722] 6. Terminal: Receive the generated code, create a column chart based on the specified range of data, and adjust the chart design.

[0723] 7. User: Review the spreadsheet to ensure the column chart was generated correctly and reflects the sentiment customizations.

[0724] In this way, the present invention analyzes the user's emotions and performs customization based on the analysis, thereby improving the efficiency of spreadsheet operations and providing a more comfortable operating environment for the user.

[0725] The processing flow will be explained below.

[0726] Step 1:

[0727] The server automatically executes the onOpen function when the spreadsheet is opened, which adds an "AI Support Menu" to the user interface.

[0728] Step 2:

[0729] The user opens the spreadsheet and selects "Enter Instructions" from the "AI Support Menu."

[0730] Step 3:

[0731] The terminal displays a prompt for the user to enter instructions in Japanese.

[0732] Step 4:

[0733] The user enters instructions into the prompt (e.g., "Set the formula A1+B1 in C1") and presses OK.

[0734] Step 5:

[0735] The terminal receives input from the user and transmits the instructions to the server.

[0736] Step 6:

[0737] The server uses a generation means to analyze the received Japanese instructions and transmits the instructions to an external AI server.

[0738] Step 7:

[0739] The server receives the analysis results from the AI ​​server and generates code for specific operations based on the instructions (e.g., generating the formula "=A1+B1").

[0740] Step 8:

[0741] The server uses an emotion engine to collect emotion data from the user when inputting commands. This emotion data is obtained by means of voice tone analysis, facial expression analysis, etc.

[0742] Step 9:

[0743] The server analyzes the acquired emotional data and identifies the user's emotional state. For example, it determines that the user is feeling stressed.

[0744] Step 10:

[0745] The server generates code to tailor the outcome of the operation based on the emotion, for example by customizing the wording of error messages to be more gentle.

[0746] Step 11:

[0747] The server returns the generated specific operation code and an adjustment code based on the emotion to the terminal.

[0748] Step 12:

[0749] The terminal executes the received code and applies it to the spreadsheet, for example, by setting the formula =A1+B1 in cell C1 and customizing the error message.

[0750] Step 13:

[0751] Users can review the results of their spreadsheet changes, ensuring that formulas, charts, and conditional formatting are applied correctly, and even view customized error messages.

[0752] This series of steps allows users to efficiently operate spreadsheets using only Japanese instructions, and furthermore, customization based on emotions provides a more comfortable operating environment.

[0753] Example 2

[0754] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0755] Existing spreadsheet operations are largely manual, requiring time and effort, especially when it comes to complex formulas, data entry restrictions, and conditional formatting. Furthermore, there are currently no methods to reduce the stress and discomfort users feel while using spreadsheets. This reduces user efficiency and makes operations more cumbersome, resulting in an uncomfortable working environment.

[0756] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0757] In this invention, the server includes a generating means for analyzing Japanese instructions, an applying means for applying the numerical formula generated by the generating means to the spreadsheet, an applying means for applying the graph generated by the generating means to the spreadsheet, an applying means for applying the conditional format generated by the generating means to the spreadsheet, an applying means for applying the numerical input restriction generated by the generating means to the spreadsheet, an emotion engine for analyzing the user's emotions, and a means for customizing the spreadsheet operation based on the analysis result of the emotion engine. This not only enables the user's instructions to be automatically reflected in the spreadsheet, but also enables customization according to the user's emotional state.

[0758] "Instructions in Japanese" refers to operational commands or requests written in Japanese.

[0759] "Generation means" refers to a device or program that has the function of analyzing input Japanese instructions and generating specific operation codes.

[0760] A "numeric formula" is a mathematical formula used in a spreadsheet to perform calculations.

[0761] A "spreadsheet" is an application that allows you to manage data in a tabular format and apply formulas, graphs, conditional formatting, etc.

[0762] The "application means" refers to a device or program that has the function of applying the generated numerical formulas, graphs, conditional formats, data input restrictions, etc. to a spreadsheet.

[0763] A "graph" is a tool used to visually represent data in a spreadsheet.

[0764] Conditional formatting is a feature that changes the display format of cells in a spreadsheet based on their values ​​or conditions.

[0765] "Data input restrictions" refers to a function that restricts the range and format of data entered into a spreadsheet.

[0766] An "emotion engine" is a device or program for analyzing a user's emotional state from their voice, facial expressions, etc.

[0767] "Means for customization" refers to a device or program that has the function of adjusting the operation and display of a spreadsheet based on the analysis results of the emotion engine.

[0768] "External AI Server" means a server equipped with external artificial intelligence used for parsing and generating instructions.

[0769] This invention is a system that converts Japanese instructions into specific operations for a spreadsheet and further customizes the operations by recognizing the user's emotions. This system includes a generation means, an application means, and an emotion engine, and automatically performs operations such as formulas, graphs, conditional formatting, and data input restrictions in a spreadsheet using Japanese instructions, and also has a customization function based on the user's emotions.

[0770] Hardware and software used

[0771] Hardware:

[0772] Device (PC, tablet, etc.)

[0773] server

[0774] software:

[0775] Spreadsheet applications (e.g. Google Sheets)

[0776] Emotion engine (e.g. IBM Watson)

[0777] Instruction analysis engine (natural language processing model, e.g. GPT)

[0778] Example of a system

[0779] 1. User Action:

[0780] The user opens the spreadsheet on their device and selects "Enter instructions" from the "AI Support Menu."

[0781] 2. Entering and retrieving Japanese instructions:

[0782] The user receives a prompt on the spreadsheet for inputting instructions, and enters specific instructions in Japanese, for example, "Please set the formula A1+B1 in C1." This instruction is then sent to the server.

[0783] 3. Parsing the instructions:

[0784] The server uses an instruction analysis engine to analyze the Japanese instructions sent by the user, and based on the analysis results, generates specific operation codes for generating formulas, creating graphs, applying conditional formatting, and restricting data entry.

[0785] 4. Emotion Recognition with Emotion Engine:

[0786] The server uses an emotion engine to analyze the user's emotions during input, for example, detecting stress or excitement from the user's voice and facial expressions.

[0787] 5. Apply the generated code:

[0788] The server sends the generated operation code and the emotion-based customization result to the device, and the device applies the specific operation to the spreadsheet, for example, setting the formula "=A1+B1" in cell C1, and adjusts the error message according to the user's emotion.

[0789] 6. Displaying the operation results:

[0790] Users can see the results right in the spreadsheet, ensuring formulas are entered correctly, charts are generated as expected, and conditional formatting and data entry restrictions are applied.

[0791] Specific use cases

[0792] Example 1: Generating mathematical expressions and reflecting emotions

[0793] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0794] 2. Terminal: A prompt appears for you to enter instructions.

[0795] 3. User: Enter "Set formula A1+B1 in C1" in the prompt and press OK.

[0796] 4. Server: Analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[0797] 5. Server: Analyzes the user's voice and facial expressions when inputting and detects emotions.

[0798] 6. Server: If the emotion engine recognizes high stress, it generates program code to change the error message to a gentler expression.

[0799] 7. Terminal: Receive the generated code, set the formula "=A1+B1" in cell C1, and adjust how the error message is displayed.

[0800] 8. User: Check the spreadsheet and confirm that the formula in cell C1 is set correctly and that the customized error message is displayed.

[0801] Example 2: Graph generation and sentiment reflection

[0802] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0803] 2. Terminal: Displays a prompt for input of instructions.

[0804] 3. User: Enter "Create a column chart based on the data from A1 to B5" in the prompt and press OK.

[0805] 4. Server: Parses the received instructions and creates the code to generate the required graph.

[0806] 5. Server: Create program code to adjust the color and design of the graph when the emotion engine recognizes that the user is highly excited.

[0807] 6. Terminal: Receive the generated code, create a column chart based on the specified range of data, and adjust the design.

[0808] 7. User: Review the spreadsheet and verify that the column chart is generated correctly and reflects the sentiment-based customizations.

[0809] Prompt Sentence Examples

[0810] Example 1 Formula Generation Prompt:

[0811] "Set the formula A1+B1 in C1"

[0812] Example 2 graph generation prompt:

[0813] "Create a column chart based on the data from A1 to B5."

[0814] In this way, the present invention analyzes the user's emotions and performs customization based on the analysis, thereby improving the efficiency of spreadsheet operations and providing a more comfortable operating environment for the user.

[0815] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0816] Program processing flow

[0817] Step 1: User interaction

[0818] explanation:

[0819] The user opens a spreadsheet application and selects "Enter Instructions" from the AI ​​Support menu.

[0820] Specific behavior:

[0821] The user opens the spreadsheet on a device such as a PC or tablet.

[0822] The device displays the "AI Support Menu" and waits for a selection from the user.

[0823] The user selects "Enter Instructions."

[0824] Input and Output:

[0825] Input: The action of a user interacting with a spreadsheet and selecting "Enter Instructions" from a menu.

[0826] Output: A terminal prompt will appear for you to enter instructions.

[0827] Step 2: Enter and retrieve Japanese instructions

[0828] explanation:

[0829] The user enters specific Japanese instructions for the spreadsheet into a prompt, which are then sent to the server.

[0830] Specific behavior:

[0831] The terminal displays prompts to the user for input of instructions.

[0832] The user enters "Please set the formula A1+B1 in C1" in the prompt and presses the submit button.

[0833] When the terminal receives the instruction, it transmits the instruction to the server.

[0834] Input and Output:

[0835] Input: The user enters specific Japanese instructions into the prompt and submits them.

[0836] Output: The user's Japanese instructions are sent to the server.

[0837] Step 3: Parse the instructions

[0838] explanation:

[0839] The server analyzes the Japanese instructions sent by the user and generates the required specific operation codes.

[0840] Specific behavior:

[0841] The server uses a generative AI model (e.g., GPT) to parse the received Japanese instructions.

[0842] The server analyzes the instruction and determines whether it is a formula, a graph, conditional formatting, or data input restrictions.

[0843] The server generates specific code for each operation. For example, if you request "Set the formula A1+B1 in C1," it generates the formula "=A1+B1."

[0844] Input and Output:

[0845] Input: The user's Japanese instructions sent to the server.

[0846] Output: Specific operation code based on the analysis results.

[0847] Step 4: Emotion Recognition with the Emotion Engine

[0848] explanation:

[0849] The server uses an emotion engine to analyze the user's emotions when inputting Japanese commands.

[0850] Specific behavior:

[0851] The server uses an emotion engine (e.g., IBM Watson) to analyze emotions from the user's voice and facial expressions.

[0852] The server recognizes the user's emotional state, such as whether they are stressed or excited.

[0853] Input and Output:

[0854] Input: User's voice and facial expression data when entering Japanese instructions.

[0855] Output: User's emotional state (e.g., stress, excitement).

[0856] Step 5: Apply the generated code

[0857] explanation:

[0858] The server generates specific operation codes, sends them to the device, and applies emotion-based customization to the spreadsheet.

[0859] Specific behavior:

[0860] The server transmits the generated operation code to the terminal.

[0861] The terminal applies the received operation code to the spreadsheet. For example, it sets the formula "=A1+B1" in cell C1.

[0862] The device reflects the results of customization by the emotion engine, for example, by changing error messages to more gentle expressions.

[0863] Input and Output:

[0864] Input: Operation code sent from the server and analysis results of the emotion engine.

[0865] Output: The state in which a specific operation is performed on the spreadsheet.

[0866] Step 6: View the operation results

[0867] explanation:

[0868] The user reviews the results on a spreadsheet.

[0869] Specific behavior:

[0870] The user checks the results of the operation in the spreadsheet.

[0871] The device will show you whether formulas are entered correctly, charts are generated as expected, and conditional formatting and input restrictions are applied.

[0872] The user also checks the results of customization by the emotion engine, for example, checking that the error message has been changed to a more gentle expression.

[0873] Input and Output:

[0874] Input: The result of an operation performed on a spreadsheet.

[0875] Output: A spreadsheet that reflects the results of the operation, and a customized section based on the user's emotions.

[0876] (Application example 2)

[0877] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0878] Conventional spreadsheet operations require users to manually set formulas and graphs, apply conditional formatting, and restrict data input, which is time-consuming and labor-intensive. Furthermore, they fail to take into account the emotional state of the user, which can cause excessive stress. In particular, systems supporting operations such as factory robots require fast and accurate data processing, so systems that are both efficient and considerate of the user's emotions are needed.

[0879] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a generating means for analyzing Japanese instructions, an applying means for applying a numerical formula generated by the generating means to a spreadsheet, an applying means for applying a graph generated by the generating means to a spreadsheet, an applying means for applying a conditional format generated by the generating means to the spreadsheet, an applying means for applying a data input restriction generated by the generating means to the spreadsheet, and a means for analyzing a user's emotions using an emotion analyzing means and customizing according to the results. This makes it possible to improve the efficiency of operations on the spreadsheet and provide a comfortable operating environment that takes user emotions into consideration.

[0880] "Instructions in Japanese" refers to requests from users regarding operations and settings written in Japanese as a natural language.

[0881] "Generation means" refers to equipment or software that has the function of analyzing Japanese instructions and generating numerical formulas, graphs, conditional formatting, and data input restrictions based on them.

[0882] "Application means" refers to equipment or software having a function for applying the formulas, graphs, conditional formatting, and data input restrictions generated by the generation means to a spreadsheet.

[0883] "Emotion analysis means" refers to equipment or software that has the function of recognizing emotions by analyzing a user's facial expressions, voice, actions, etc.

[0884] "Server" refers to a computer system that has the role of integrating the generation means, application means, emotion analysis means, etc., and provides the computational resources to execute these functions.

[0885] A "spreadsheet" is an electronic spreadsheet application that allows you to enter data into units called cells and manage and analyze the data using numerical formulas and graphs.

[0886] "Customization" refers to improving the comfort and efficiency of operation by changing and adjusting the operation screen and feedback content according to the user's emotions and usage situation.

[0887] This invention relates to a system that analyzes instructions in Japanese and automatically performs specific operations on a spreadsheet, and further has the function of analyzing the user's emotions and customizing the operations.

[0888] The system consists of a server, a terminal, a generation means, an application means, and an emotion analysis means. The server processes various data and is responsible for running the generation means, application means, and emotion analysis means. The terminal provides an interface for user instruction input and displaying operation results.

[0889] Generating a Program

[0890] The system program performs the following processing.

[0891] 1. Analysis of Japanese instructions

[0892] The server uses a generation means to analyze the Japanese instructions received from the user, and the generation means uses natural language processing technology to identify the content of the instructions and generate numerical expressions, graphs, conditional formats, and data input restrictions.

[0893] 2. Applying generated content

[0894] Formulas and graphs generated on the server are reflected in the spreadsheet through application means, such as entering a formula in a cell or generating a graph in a specified range.

[0895] 3. Emotion analysis

[0896] The emotion analysis means analyzes the user's voice and facial expressions to recognize their emotional state at that time. The server adjusts the response to operations and the expression of error messages based on the analysis results.

[0897] Hardware and software used

[0898] Hardware

[0899] Factory robots: Responsible for collecting data and inputting it into spreadsheets.

[0900] Facial recognition camera: Used to capture the user's facial expressions in real time.

[0901] PC or Server: Processes data and manages the system.

[0902] software

[0903] Python and related libraries (pandas, openpyxl, etc.): Used for data processing and analysis.

[0904] Natural Language Processing (NLP) models: used to parse Japanese instructions, including generative AI models.

[0905] EmotionRecognizer: A library for analyzing user emotions.

[0906] Specific examples

[0907] For example, the following operations are specifically performed.

[0908] Data Entry Example

[0909] When the user instructs the terminal, "Please input the temperature sensor data into A1," the server analyzes the instruction and inputs the temperature data into the specified cell through the application means.

[0910] Formula generation example

[0911] When the user instructs "Calculate the average value of B5 to B10 in D1," the server generates the formula =AVERAGE(B5:B10) using the generating means and applies it to cell D1 through the applying means.

[0912] Graph Generation Example

[0913] When a user instructs the server to "create a line graph based on the data in columns A and B," the server generates a line graph based on the data in the specified range and reflects it in the spreadsheet.

[0914] Example of customization using sentiment analysis

[0915] If the camera analyzes the user's facial expression as stressed, the server changes the error message to a softer one, such as "Please check again."

[0916] Prompt Sentence Examples

[0917] Below is an example of a prompt sentence entered into the system.

[0918] "Enter the temperature sensor data in A1"

[0919] "Calculate the average of B5 to B10 in D1."

[0920] "Create a line graph based on the data in columns A and B."

[0921] Please change the "An error has occurred" message to a more friendly one.

[0922] In this way, the present invention is a system that can automate spreadsheet operations while taking into consideration the user's feelings.

[0923] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0924] Step 1: Enter Japanese instructions

[0925] ---

[0926] The terminal receives instructions in Japanese from the user. The user inputs instructions in Japanese into the prompts displayed on the terminal. Examples of such input include "Enter the temperature sensor data in A1" and "Calculate the average value of B5 to B10 in D1."

[0927] Input: Japanese instructions from the user

[0928] Output: Japanese instruction data

[0929] Specific operation: The user enters instructions in Japanese into the input field on the terminal and presses the send button.

[0930] Step 2: Send instructions in Japanese

[0931] ---

[0932] The terminal sends the input Japanese instructions to the server, where this data is processed for analysis.

[0933] Input: Japanese instruction data

[0934] Output: Request to server

[0935] Specific operation: The terminal sends the Japanese instructions entered by the user to the server as an HTTP request.

[0936] Step 3: Parse the instructions

[0937] ---

[0938] The server analyzes the received Japanese instructions using a generation means, which uses a natural language processing model to identify the content of the instructions (numeric expressions, graphs, conditional formatting, data input restrictions).

[0939] Input: Japanese instruction data

[0940] Output: Analysis results (formulas, graph generation code, conditional formatting, data entry restrictions)

[0941] Specific Operation: The server parses the instruction using a natural language processing (NLP) model and generates a corresponding operation code.

[0942] Step 4: Sentiment Analysis

[0943] ---

[0944] The server analyzes the user's emotions using an emotion analysis engine, which determines the user's emotional state based on data acquired from a facial recognition camera and a voice input device.

[0945] Input: User's voice data and facial expression data

[0946] Output: Emotion analysis results (stress level, etc.)

[0947] Specific operation: The server uses emotion analysis means to analyze the user's voice and facial expressions to determine their stress level and emotional state.

[0948] Step 5: Apply the generated content

[0949] ---

[0950] The server applies the formulas, graphs, conditional formatting, and data entry restrictions generated based on the analysis results to the spreadsheet, and these contents are reflected in the spreadsheet through the application means.

[0951] Input: Analysis results and sentiment analysis results

[0952] Output: Updated data in the spreadsheet

[0953] Specific operations: The server applies the generated formulas, graphs, conditional formatting, and data entry restrictions to the spreadsheet using the application means, and changes the wording of error messages as necessary. The server also opens an existing spreadsheet file and inserts the corresponding data, formulas, and graphs into the specified cells.

[0954] Step 6: View the operation results

[0955] ---

[0956] The terminal displays the results of the spreadsheet operations based on the updated data received from the server, allowing the user to check the accuracy of the generated formulas and graphs.

[0957] Input: Spreadsheet update data

[0958] Output: Display of the updated spreadsheet

[0959] Specific behavior: The device redraws the spreadsheet based on the updated data received from the server and displays the results for the user to review.

[0960] The above are the processing steps of the system program that realizes the application example.

[0961] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0962] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0963] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0964] [Third embodiment]

[0965] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0966] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0967] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0968] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0969] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0970] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0971] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0972] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0973] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0974] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0975] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0976] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0977] This invention relates to a system that converts Japanese language instructions into specific operations for a spreadsheet. This system includes a generating means and an applying means, and can automatically generate and apply formulas, graphs, conditional formatting, data input restrictions, etc. in a spreadsheet using Japanese language instructions.

[0978] Example of a system

[0979] System Overview

[0980] The server receives Japanese instructions from the user via a program that runs in conjunction with the spreadsheet. The terminal displays the Japanese instructions entered by the user and sends them to the server for analysis. Based on the analysis results, code for the appropriate operation is generated, sent back to the terminal, and applied to the spreadsheet.

[0981] Functions and Operation Procedures

[0982] 1. User operations

[0983] The user opens a spreadsheet and selects one of the items in the "AI Support Menu." This menu includes items such as "Generate Formulas," "Generate Graphs," "Apply Conditional Formatting," "Set Data Entry Restrictions," and "Enter Instructions." When the user selects "Enter Instructions," a prompt for entering instructions appears on the device.

[0984] 2. Inputting and retrieving Japanese instructions

[0985] The user inputs instructions in Japanese into the prompt, such as "Please set the formula A1+B1 in C1." This input information is sent to the server and used for subsequent processing.

[0986] 3. Parsing the instructions

[0987] The server uses the generation means to analyze the Japanese instructions received from the user. During the analysis, it identifies whether the instruction is a mathematical formula, graph, conditional formatting, or data input restriction. It then generates specific code according to the operation.

[0988] 4. Applying the generated code

[0989] The server generates a specific code based on the analysis and returns it to the device. The device receives this code and performs operations to apply formulas, charts, conditional formatting, and data entry restrictions to the appropriate locations in the spreadsheet. For example, if the input instruction is "Set the formula A1+B1 in C1," the formula =A1+B1 will be set in cell C1.

[0990] 5. Displaying the operation results

[0991] Once applied, users can review the results in the spreadsheet to ensure formulas are entered correctly, charts are generated as expected, and conditional formatting and data entry restrictions are applied.

[0992] Specific use cases

[0993] Specific usage examples are described below.

[0994] Example 1: Generating a formula

[0995] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[0996] 2. Terminal: A prompt appears for you to enter instructions.

[0997] 3. User: Enter "Set formula A1+B1 in C1" in the prompt and press OK.

[0998] 4. Server: Analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[0999] 5. Terminal: Receive the generated code and set the formula "=A1+B1" in cell C1.

[1000] 6. User: Check the spreadsheet and verify that the formula is set correctly in cell C1.

[1001] Example 2: Generating a graph

[1002] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[1003] 2. Terminal: A prompt appears for you to enter instructions.

[1004] 3. User: Enter "Create a column chart based on the data from A1 to B5" in the prompt and press OK.

[1005] 4. Server: Parses the received instructions and generates the code to generate the required graph.

[1006] 5. Terminal: Receive the generated code and create a column chart based on the data in the specified range.

[1007] 6. User: Review the spreadsheet and verify that the column chart was generated correctly.

[1008] In this way, with the present invention, various spreadsheet operations are automatically performed simply by the user inputting instructions in Japanese, allowing even beginners to work efficiently and accurately.

[1009] The processing flow will be explained below.

[1010] Step 1:

[1011] The server automatically executes the onOpen function when the spreadsheet is opened, which generates a custom menu, the "AI Support Menu," that the user can use and adds it to the user interface.

[1012] Step 2:

[1013] The user opens the spreadsheet and selects "Enter Instructions" from the "AI Support Menu."

[1014] Step 3:

[1015] The terminal displays a prompt for the user to enter instructions in Japanese.

[1016] Step 4:

[1017] The user enters instructions into the prompt (e.g., "Please set the formula A1+B1 in C1") and presses OK.

[1018] Step 5:

[1019] The terminal receives input from the user and transmits the instructions to the server.

[1020] Step 6:

[1021] The server uses a generation means to analyze the received Japanese instructions and transmits the instructions to an external AI server.

[1022] Step 7:

[1023] The server receives the parsed results and generates code for specific operations based on the instructions (e.g., generating the formula "=A1+B1").

[1024] Step 8:

[1025] The server returns the generated specific operation code to the terminal.

[1026] Step 9:

[1027] The terminal executes the received code and applies it to the spreadsheet, for example, by setting the formula =A1+B1 in cell C1.

[1028] Step 10:

[1029] The user reviews the resulting changes in the spreadsheet and ensures that formulas, charts, and conditional formatting are applied correctly.

[1030] This series of steps allows users to efficiently perform spreadsheet operations using only Japanese instructions.

[1031] Example 1

[1032] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1033] Conventional spreadsheet software lacks the functionality to automatically generate and apply formulas, graphs, conditional formatting, and data entry constraints based on instructions in Japanese. This makes it difficult for beginner users, in particular, to operate spreadsheets efficiently. Furthermore, the software lacks a flexible interface that allows users to intuitively perform a variety of operations. A system that solves this problem and allows users to operate spreadsheets easily and accurately is needed.

[1034] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1035] In this invention, the server includes a generating means for analyzing instructions in Japanese, an applying means for applying a numerical formula generated by the generating means to a spreadsheet software, an applying means for applying a graph generated by the generating means to the spreadsheet software, an applying means for applying a conditional format generated by the generating means to the spreadsheet software, an applying means for applying a data input constraint generated by the generating means to the spreadsheet software, and a means for receiving instructions from a user terminal and transmitting the analysis results to the user terminal, thereby making it possible to automatically perform various operations on a spreadsheet based on instructions in Japanese.

[1036] The "generation means" is a means for analyzing Japanese instructions and converting them into concrete operations such as mathematical formulas, graphs, conditional formatting, and data input constraints.

[1037] The "application means" is a means for applying the formulas, graphs, conditional formats, and data input constraints generated by the generation means to the spreadsheet software.

[1038] A "formula" is an expression that specifies a combination of calculations or values ​​to be entered into a cell in a spreadsheet.

[1039] A "graph" is a diagram used in spreadsheet software to visually represent data.

[1040] "Conditional formatting" is a feature in spreadsheet software that automatically changes the formatting of cells based on specific conditions.

[1041] "Data input constraints" are settings that limit the type and range of data that can be entered into cells in spreadsheet software.

[1042] A "user terminal" is a computer device or equipment used to operate spreadsheet software and input instructions in Japanese.

[1043] "Analysis results" refers to the specific operation content and code obtained by the generation means when it analyzes the Japanese instructions.

[1044] A "spreadsheet" is software for organizing, calculating, analyzing, and visualizing data.

[1045] A "generative AI model" is a model that uses artificial intelligence to analyze Japanese instructions and generate appropriate operation codes.

[1046] "Prompt" refers to the input message or screen that is displayed when the user enters instructions in Japanese.

[1047] This invention relates to a system that analyzes Japanese language instructions and automatically generates and applies formulas, graphs, conditional formatting, and data input constraints to spreadsheet software. This system includes a generating means and an applying means, and can operate spreadsheets based on Japanese language instructions entered by the user. Specific embodiments of the invention are described below.

[1048] Required Hardware and Software

[1049] User device: A device such as a computer, tablet, or smartphone.

[1050] Server: Cloud server or dedicated server.

[1051] Spreadsheet software: Software for organizing, calculating, analyzing, and visualizing data (e.g., Google Sheets, Microsoft Excel).

[1052] Generative AI model: An artificial intelligence model for analyzing Japanese instructions and generating appropriate operation codes.

[1053] System Operation Overview

[1054] The server receives Japanese instructions sent from the user's device and analyzes them using a generative AI model. Based on the analysis results, it generates specific formulas, graphs, conditional formatting, and data input constraints and returns them to the user's device. The user's device then applies the results to the spreadsheet software.

[1055] Usage example

[1056] Example 1: Generating a formula

[1057] 1. The user selects "Enter instructions" from the "AI Support Menu."

[1058] 2. The terminal will prompt you to enter instructions.

[1059] 3. The user enters "Please set the formula A1+B1 in C1" in the prompt and presses the submit button.

[1060] 4. The server analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[1061] 5. The terminal receives the generated code and sets the formula "=A1+B1" in cell C1.

[1062] 6. The user checks the spreadsheet and confirms that the formula "=A1+B1" is correctly set in cell C1.

[1063] Example 2: Generating a graph

[1064] 1. The user selects "Enter instructions" from the "AI Support Menu."

[1065] 2. The terminal will prompt you to enter instructions.

[1066] 3. The user enters "Create a column chart based on the data from A1 to B5" in the prompt and presses the submit button.

[1067] 4. The server parses the received instructions and generates the necessary graph generation code.

[1068] 5. The terminal receives the generated code and creates a column chart based on the data in the specified range.

[1069] 6. The user reviews the spreadsheet and verifies that the column chart was generated correctly.

[1070] This system allows users to automatically perform complex operations in spreadsheet software simply by inputting instructions in Japanese, enabling even beginners to work efficiently and accurately.

[1071] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1072] Step 1:

[1073] The user opens a spreadsheet and selects "Enter instructions" from the "AI Support Menu." When the user selects an operation, a prompt for inputting instructions is displayed on the terminal. This prompt is an input interface that allows the user to enter specific instructions in Japanese. An example of input is "Please set the formula A1+B1 in C1." The output is a prompt screen that is displayed on the terminal.

[1074] Step 2:

[1075] The user inputs Japanese instructions into the displayed prompt and presses the send button. For example, the user can input instructions such as "Please set the formula A1+B1 in C1." The input is the Japanese instructions entered by the user, and the output is the instruction data sent from the user terminal to the server. In concrete terms, the prompt screen receives the user's input and sends it to the server via the network.

[1076] Step 3:

[1077] The server analyzes the received Japanese instructions. This analysis is performed using a generative AI model. The generative AI model analyzes the Japanese instructions and generates the appropriate operation code. The input is the Japanese instructions received by the server, and the output is the operation code (e.g., "=A1+B1") that is the result of the analysis. Specifically, the server passes the input instructions to the AI ​​model, which then analyzes the instructions and identifies the appropriate spreadsheet operation.

[1078] Step 4:

[1079] The server generates operation codes based on the analysis results. These operation codes represent the formulas, graphs, conditional formatting, and data input constraints to be applied to the spreadsheet. The input is the instructions analyzed by the AI ​​model, and the output is the specific operation code. As a specific operation, the server generates the specific spreadsheet operation code to be applied based on the analysis results returned by the AI ​​model.

[1080] Step 5:

[1081] The server sends the generated operation code to the terminal. The input is the operation code generated by the server, and the output is the operation code sent to the terminal. The specific operation is to send the code generated by the server back to the user's terminal via the network.

[1082] Step 6:

[1083] The terminal applies the operation code received from the server to the spreadsheet. For example, set the formula "=A1+B1" in cell C1. The input is the operation code received by the terminal, and the output is the formula and graph applied to the spreadsheet. Specifically, the terminal analyzes the operation code and performs the operation on the corresponding cell or range in the spreadsheet based on its contents.

[1084] Step 7:

[1085] The terminal reports the success of the operation to the server. The input is the result of application to the spreadsheet, and the output is a report of the success of the operation to the server. Specifically, the terminal checks the result of application to the spreadsheet, and if the operation was successful, it sends that information to the server.

[1086] Step 8:

[1087] The user checks the spreadsheet to ensure that their instructions have been executed correctly. For example, they check that the formula "=A1+B1" is set correctly in cell C1. The input is the user's visual confirmation, and the output is the visual confirmation result on the spreadsheet. Specifically, the user opens the spreadsheet and checks to see if the specified operations have been correctly reflected.

[1088] (Application example 1)

[1089] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1090] Modern spreadsheet operations are complex and time-consuming, especially when dealing with large amounts of data. It is particularly difficult for non-technical users and beginners to operate them efficiently. Furthermore, current systems often struggle to handle situations where quick on-site response is required, such as when it is difficult to operate while moving around or by voice input. There is a need for a system that can solve these issues and automate spreadsheet operations by simply inputting instructions via voice.

[1091] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1092] In this invention, the server includes a generating means for analyzing instructions in Japanese, an applying means for applying a numerical formula generated by the generating means to a spreadsheet, an applying means for applying a graph generated by the generating means to a spreadsheet, an applying means for applying a conditional format generated by the generating means to a spreadsheet, an applying means for applying a data input restriction generated by the generating means to a spreadsheet, a means for analyzing a user's voice instructions and automating spreadsheet operations, and a means for receiving the voice instructions via a smart device. This allows spreadsheet operations to be automated with simple voice input of instructions, enabling non-technical people and those in the field to respond quickly.

[1093] The "means for generating Japanese language instructions" is a means for analyzing Japanese language instructions input by a user and interpreting their contents.

[1094] The "means for applying the numerical formula generated by the generating means to the spreadsheet" is a means for inputting the numerical formula generated from the analyzed Japanese instructions into an appropriate cell of the spreadsheet.

[1095] The "means for applying the graph generated by the generating means to the spreadsheet" is a means for introducing the graph analyzed and generated from the Japanese instructions into the spreadsheet.

[1096] The "application means for applying the conditional formatting generated by the generation means to the spreadsheet" is a means for applying the conditional formatting generated based on the user's instructions to a specified range of the spreadsheet.

[1097] The "means for applying the data input restrictions generated by the generating means to the spreadsheet" is a means for setting the data input restrictions generated based on the user's Japanese instructions to the spreadsheet.

[1098] The "means for analyzing a user's voice instructions and automating spreadsheet operations" refers to a means for converting a user's voice instructions into text, analyzing it, and automatically operating a spreadsheet.

[1099] The "means for acquiring voice instructions via a smart device" refers to a means for collecting and analyzing voice instructions from a user using a smart device.

[1100] MODE FOR CARRYING OUT THE INVENTION

[1101] The present invention provides a system for automating spreadsheet operations using voice instructions in Japanese on a smart device. Specific embodiments of the invention are described below.

[1102] System configuration that analyzes voice instructions and performs spreadsheet operations

[1103] The system includes the following major components:

[1104] 1. Smart terminal: A device that receives user voice commands (e.g., smart glasses or head-mounted displays).

[1105] 2. Server: Hardware and software (e.g., Google Sheets API, Python) to parse the voice instructions and generate the appropriate spreadsheet operations.

[1106] 3. Generator: A means (e.g., speech_recognition library, AI server) for parsing Japanese instructions and applying numerical formulas, graphs, conditional formatting, and data entry control operations to the spreadsheet.

[1107] Program processing and the hardware and software used

[1108] 1. Smart Devices:

[1109] Hardware: Smart glasses and head-mounted displays.

[1110] Software: Speech recognition library (e.g. speech_recognition).

[1111] Processing: The smart device collects the user's voice commands and converts them into text data.

[1112] 2. Server:

[1113] Hardware: Standard server.

[1114] Software: Python, Google Sheets API, OAuth2 client.

[1115] process:

[1116] The text data generated by speech recognition is analyzed and Japanese instructions are interpreted.

[1117] Generates code for spreadsheet operations depending on the processing content.

[1118] Example of operation and prompt statement

[1119] Examples of operations include:

[1120] Example 1: Inventory check

[1121] 1. User: "Please check the stock of product code 12345."

[1122] 2. Server: Analyzes the instructions and uses the Google Sheets API to retrieve inventory information for the corresponding product code from the spreadsheet.

[1123] 3. Terminal: The acquired inventory information is displayed on the smart terminal.

[1124] Example 2: Inventory Update

[1125] 1. User: "Please update the inventory of product code 67890 to 20."

[1126] 2. Server: Parses the instruction and generates code to update the stock quantity in the appropriate spreadsheet cell to 20.

[1127] 3. Terminal: Perform the update operation and confirm that the inventory has been updated to 20.

[1128] Example prompt sentence:

[1129] "Please check stock availability for product code 12345"

[1130] "Please update the stock of product code 67890 to 20."

[1131] In this manner, the present invention allows a user to efficiently automate spreadsheet operations through voice instructions.

[1132] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1133] Step 1:

[1134] The device receives voice instructions from a user via a smart device (e.g., smart glasses). The input is the user's voice instruction, and the output is voice data. This voice data is collected using the microphone of the smart device.

[1135] Step 2:

[1136] The device converts the acquired voice data into text data. Specifically, it uses a voice recognition library (e.g., speech_recognition) to analyze the voice data and generate text data. The input is voice data, and the output is text data.

[1137] Step 3:

[1138] The server receives the text data sent from the device and analyzes the user's instructions. The input is text data, and the output is the analyzed instructions. Natural language processing technology is used for the analysis to interpret the intent of the Japanese instructions.

[1139] Step 4:

[1140] The server generates code to perform the appropriate spreadsheet operations based on the parsed instructions. The input is the parsed instructions, and the output is the code to perform the spreadsheet operations. Specific operations include the use of algorithms to generate and apply formulas, graphs, conditional formatting, and data entry restrictions.

[1141] Step 5:

[1142] The server sends the generated code to the terminal. The input is the spreadsheet operation code, and the output is the transmitted data. The transmission is performed via data communication over a network.

[1143] Step 6:

[1144] The device applies the received spreadsheet operation code to the spreadsheet. The input is the spreadsheet operation code, and the output is the updated content of the spreadsheet. Specifically, the operation is performed on the specified cell or range of the spreadsheet using the Google Sheets API.

[1145] Step 7:

[1146] The device provides feedback to the user on the results of the spreadsheet updates. The input is the updated content of the spreadsheet, and the output is a display of the updated results. Specifically, the results are displayed on the smart device's display, and feedback is provided by voice or text.

[1147] Specific examples

[1148] Example 1: Inventory check

[1149] 1. User: "Please check the stock of product code 12345."

[1150] 2. Terminal: Converts voice instructions into text and sends it to the server.

[1151] 3. Server: Analyzes the instructions and uses the Google Sheets API to retrieve inventory information for the corresponding product code from the spreadsheet.

[1152] 4. Terminal: Display the retrieved inventory information to the user. It displays "There are 10 units of product code 12345 in stock."

[1153] Example 2: Inventory Update

[1154] 1. User: "Please update the inventory of product code 67890 to 20."

[1155] 2. Terminal: Converts voice instructions into text and sends it to the server.

[1156] 3. Server: Parses the instruction and generates code to update the stock quantity in the appropriate spreadsheet cell to 20.

[1157] 4. Terminal: Executes the update operation and displays the results to the user. It displays "The inventory of product code 67890 has been updated to 20."

[1158] Example prompt sentence:

[1159] "Please check stock availability for product code 12345"

[1160] "Please update the stock of product code 67890 to 20."

[1161] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1162] This invention relates to a system that converts Japanese language instructions into specific operations for a spreadsheet and combines it with an emotion engine that recognizes the user's emotions. The system includes a generation means, an application means, and an emotion engine, and automatically generates and applies formulas, graphs, conditional formatting, data input restrictions, etc. in a spreadsheet in response to Japanese language instructions, and also has a customization function based on the user's emotions.

[1163] System configuration

[1164] Example of a system

[1165] 1. User operations

[1166] The user opens a spreadsheet and selects one of the items in the "AI Support Menu." This menu includes items such as "Generate Formulas," "Generate Graphs," "Apply Conditional Formatting," "Set Data Input Restrictions," and "Enter Instructions." When the user selects "Enter Instructions," a prompt for entering instructions appears on the device.

[1167] 2. Inputting and retrieving Japanese instructions

[1168] The user inputs instructions in Japanese into the prompt, such as "Please set the formula A1+B1 in C1." This input information is sent to the server and used for subsequent processing.

[1169] 3. Parsing the instructions

[1170] The server uses the generation means to analyze the Japanese instructions received from the user. During the analysis, it identifies whether the instruction corresponds to a mathematical formula, graph, conditional formatting, or data input restriction. It then generates specific code according to the operation content.

[1171] 4. Emotion Recognition by Emotion Engine

[1172] The server uses an emotion engine to analyze the user's emotions when inputting instructions. For example, if the user is feeling stressed, the server recognizes that emotion and takes measures to reduce the user's workload.

[1173] 5. Applying the generated code

[1174] The server then returns the specific operation code it generated to the device. The device then receives this code and applies formulas, graphs, conditional formatting, and data entry restrictions to the appropriate locations in the spreadsheet. For example, if the input instruction is "Set the formula A1+B1 in C1," the formula =A1+B1 will be set in cell C1. In addition, based on the analysis results of the emotion engine, the device may change error messages to be more friendly, for example.

[1175] 6. Displaying the operation results

[1176] Once applied, users can review the results in their spreadsheet to ensure formulas are entered correctly, charts are generated as expected, conditional formatting and validation rules are applied, and any customizations made by the emotion engine are reflected.

[1177] Specific use cases

[1178] Example 1: Generating mathematical expressions and reflecting emotions

[1179] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[1180] 2. Terminal: A prompt appears for you to enter instructions.

[1181] 3. User: Enter "Set formula A1+B1 in C1" in the prompt and press OK.

[1182] 4. Server: Analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[1183] 5. Server: Analyzes the user's voice and facial expressions when inputting and detects emotions.

[1184] 6. Server: If the emotion engine recognizes high stress, it generates program code to change the error message to a gentler expression.

[1185] 7. Terminal: Receive the generated code, set the formula "=A1+B1" in cell C1, and adjust how the error message is displayed.

[1186] 8. User: Check the spreadsheet and confirm that the formula in cell C1 is set correctly and that the customized error message is displayed.

[1187] Example 2: Graph generation and sentiment reflection

[1188] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[1189] 2. Terminal: A prompt appears for you to enter instructions.

[1190] 3. User: Enter "Create a column chart based on the data from A1 to B5" in the prompt and press OK.

[1191] 4. Server: Parses the received instructions and generates the code to generate the required graph.

[1192] 5. Server: If the emotion engine recognizes that the user is highly excited, it generates program code to adjust the color and design of the chart.

[1193] 6. Terminal: Receive the generated code, create a column chart based on the specified range of data, and adjust the chart design.

[1194] 7. User: Review the spreadsheet to ensure the column chart was generated correctly and reflects the sentiment customizations.

[1195] In this way, the present invention analyzes the user's emotions and performs customization based on the analysis, thereby improving the efficiency of spreadsheet operations and providing a more comfortable operating environment for the user.

[1196] The processing flow will be explained below.

[1197] Step 1:

[1198] The server automatically executes the onOpen function when the spreadsheet is opened, which adds an "AI Support Menu" to the user interface.

[1199] Step 2:

[1200] The user opens the spreadsheet and selects "Enter Instructions" from the "AI Support Menu."

[1201] Step 3:

[1202] The terminal displays a prompt for the user to enter instructions in Japanese.

[1203] Step 4:

[1204] The user enters instructions into the prompt (e.g., "Set the formula A1+B1 in C1") and presses OK.

[1205] Step 5:

[1206] The terminal receives input from the user and transmits the instructions to the server.

[1207] Step 6:

[1208] The server uses a generation means to analyze the received Japanese instructions and transmits the instructions to an external AI server.

[1209] Step 7:

[1210] The server receives the analysis results from the AI ​​server and generates code for specific operations based on the instructions (e.g., generating the formula "=A1+B1").

[1211] Step 8:

[1212] The server uses an emotion engine to collect emotion data from the user when inputting commands. This emotion data is obtained by means of voice tone analysis, facial expression analysis, etc.

[1213] Step 9:

[1214] The server analyzes the acquired emotional data and identifies the user's emotional state. For example, it determines that the user is feeling stressed.

[1215] Step 10:

[1216] The server generates code to tailor the outcome of the operation based on the emotion, for example by customizing the wording of error messages to be more gentle.

[1217] Step 11:

[1218] The server returns the generated specific operation code and an adjustment code based on the emotion to the terminal.

[1219] Step 12:

[1220] The terminal executes the received code and applies it to the spreadsheet, for example, by setting the formula =A1+B1 in cell C1 and customizing the error message.

[1221] Step 13:

[1222] Users can review the results of their spreadsheet changes, ensuring that formulas, charts, and conditional formatting are applied correctly, and even view customized error messages.

[1223] This series of steps allows users to efficiently operate spreadsheets using only Japanese instructions, and furthermore, customization based on emotions provides a more comfortable operating environment.

[1224] Example 2

[1225] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1226] Existing spreadsheet operations are largely manual, requiring time and effort, especially when it comes to complex formulas, data entry restrictions, and conditional formatting. Furthermore, there are currently no methods to reduce the stress and discomfort users feel while using spreadsheets. This reduces user efficiency and makes operations more cumbersome, resulting in an uncomfortable working environment.

[1227] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1228] In this invention, the server includes a generating means for analyzing Japanese instructions, an applying means for applying the numerical formula generated by the generating means to the spreadsheet, an applying means for applying the graph generated by the generating means to the spreadsheet, an applying means for applying the conditional format generated by the generating means to the spreadsheet, an applying means for applying the numerical input restriction generated by the generating means to the spreadsheet, an emotion engine for analyzing the user's emotions, and a means for customizing the spreadsheet operation based on the analysis result of the emotion engine. This not only enables the user's instructions to be automatically reflected in the spreadsheet, but also enables customization according to the user's emotional state.

[1229] "Instructions in Japanese" refers to operational commands or requests written in Japanese.

[1230] "Generation means" refers to a device or program that has the function of analyzing input Japanese instructions and generating specific operation codes.

[1231] A "numeric formula" is a mathematical formula used in a spreadsheet to perform calculations.

[1232] A "spreadsheet" is an application that allows you to manage data in a tabular format and apply formulas, graphs, conditional formatting, etc.

[1233] The "application means" refers to a device or program that has the function of applying the generated numerical formulas, graphs, conditional formats, data input restrictions, etc. to a spreadsheet.

[1234] A "graph" is a tool used to visually represent data in a spreadsheet.

[1235] Conditional formatting is a feature that changes the display format of cells in a spreadsheet based on their values ​​or conditions.

[1236] "Data input restrictions" refers to a function that restricts the range and format of data entered into a spreadsheet.

[1237] An "emotion engine" is a device or program for analyzing a user's emotional state from their voice, facial expressions, etc.

[1238] "Means for customization" refers to a device or program that has the function of adjusting the operation and display of a spreadsheet based on the analysis results of the emotion engine.

[1239] "External AI Server" means a server equipped with external artificial intelligence used for parsing and generating instructions.

[1240] This invention is a system that converts Japanese instructions into specific operations for a spreadsheet and further customizes the operations by recognizing the user's emotions. This system includes a generation means, an application means, and an emotion engine, and automatically performs operations such as formulas, graphs, conditional formatting, and data input restrictions in a spreadsheet using Japanese instructions, and also has a customization function based on the user's emotions.

[1241] Hardware and software used

[1242] Hardware:

[1243] Device (PC, tablet, etc.)

[1244] server

[1245] software:

[1246] Spreadsheet applications (e.g. Google Sheets)

[1247] Emotion engine (e.g. IBM Watson)

[1248] Instruction analysis engine (natural language processing model, e.g. GPT)

[1249] Example of a system

[1250] 1. User Action:

[1251] The user opens the spreadsheet on their device and selects "Enter instructions" from the "AI Support Menu."

[1252] 2. Entering and retrieving Japanese instructions:

[1253] The user receives a prompt on the spreadsheet for inputting instructions, and enters specific instructions in Japanese, for example, "Please set the formula A1+B1 in C1." This instruction is then sent to the server.

[1254] 3. Parsing the instructions:

[1255] The server uses an instruction analysis engine to analyze the Japanese instructions sent by the user, and based on the analysis results, generates specific operation codes for generating formulas, creating graphs, applying conditional formatting, and restricting data entry.

[1256] 4. Emotion Recognition with Emotion Engine:

[1257] The server uses an emotion engine to analyze the user's emotions during input, for example, detecting stress or excitement from the user's voice and facial expressions.

[1258] 5. Apply the generated code:

[1259] The server sends the generated operation code and the emotion-based customization result to the device, and the device applies the specific operation to the spreadsheet, for example, setting the formula "=A1+B1" in cell C1, and adjusts the error message according to the user's emotion.

[1260] 6. Displaying the operation results:

[1261] Users can see the results right in the spreadsheet, ensuring formulas are entered correctly, charts are generated as expected, and conditional formatting and data entry restrictions are applied.

[1262] Specific use cases

[1263] Example 1: Generating mathematical expressions and reflecting emotions

[1264] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[1265] 2. Terminal: A prompt appears for you to enter instructions.

[1266] 3. User: Enter "Set formula A1+B1 in C1" in the prompt and press OK.

[1267] 4. Server: Analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[1268] 5. Server: Analyzes the user's voice and facial expressions when inputting and detects emotions.

[1269] 6. Server: If the emotion engine recognizes high stress, it generates program code to change the error message to a gentler expression.

[1270] 7. Terminal: Receive the generated code, set the formula "=A1+B1" in cell C1, and adjust how the error message is displayed.

[1271] 8. User: Check the spreadsheet and confirm that the formula in cell C1 is set correctly and that the customized error message is displayed.

[1272] Example 2: Graph generation and sentiment reflection

[1273] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[1274] 2. Terminal: Displays a prompt for input of instructions.

[1275] 3. User: Enter "Create a column chart based on the data from A1 to B5" in the prompt and press OK.

[1276] 4. Server: Parses the received instructions and creates the code to generate the required graph.

[1277] 5. Server: Create program code to adjust the color and design of the graph when the emotion engine recognizes that the user is highly excited.

[1278] 6. Terminal: Receive the generated code, create a column chart based on the specified range of data, and adjust the design.

[1279] 7. User: Review the spreadsheet and verify that the column chart is generated correctly and reflects the sentiment-based customizations.

[1280] Prompt Sentence Examples

[1281] Example 1 Formula Generation Prompt:

[1282] "Set the formula A1+B1 in C1"

[1283] Example 2 graph generation prompt:

[1284] "Create a column chart based on the data from A1 to B5."

[1285] In this way, the present invention analyzes the user's emotions and performs customization based on the analysis, thereby improving the efficiency of spreadsheet operations and providing a more comfortable operating environment for the user.

[1286] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1287] Program processing flow

[1288] Step 1: User interaction

[1289] explanation:

[1290] The user opens a spreadsheet application and selects "Enter Instructions" from the AI ​​Support menu.

[1291] Specific behavior:

[1292] The user opens the spreadsheet on a device such as a PC or tablet.

[1293] The device displays the "AI Support Menu" and waits for a selection from the user.

[1294] The user selects "Enter Instructions."

[1295] Input and Output:

[1296] Input: The action of a user interacting with a spreadsheet and selecting "Enter Instructions" from a menu.

[1297] Output: A terminal prompt will appear for you to enter instructions.

[1298] Step 2: Enter and retrieve Japanese instructions

[1299] explanation:

[1300] The user enters specific Japanese instructions for the spreadsheet into a prompt, which are then sent to the server.

[1301] Specific behavior:

[1302] The terminal displays prompts to the user for input of instructions.

[1303] The user enters "Please set the formula A1+B1 in C1" in the prompt and presses the submit button.

[1304] When the terminal receives the instruction, it transmits the instruction to the server.

[1305] Input and Output:

[1306] Input: The user enters specific Japanese instructions into the prompt and submits them.

[1307] Output: The user's Japanese instructions are sent to the server.

[1308] Step 3: Parse the instructions

[1309] explanation:

[1310] The server analyzes the Japanese instructions sent by the user and generates the required specific operation codes.

[1311] Specific behavior:

[1312] The server uses a generative AI model (e.g., GPT) to parse the received Japanese instructions.

[1313] The server analyzes the instruction and determines whether it is a formula, a graph, conditional formatting, or data input restrictions.

[1314] The server generates specific code for each operation. For example, if you request "Set the formula A1+B1 in C1," it generates the formula "=A1+B1."

[1315] Input and Output:

[1316] Input: The user's Japanese instructions sent to the server.

[1317] Output: Specific operation code based on the analysis results.

[1318] Step 4: Emotion Recognition with the Emotion Engine

[1319] explanation:

[1320] The server uses an emotion engine to analyze the user's emotions when inputting Japanese commands.

[1321] Specific behavior:

[1322] The server uses an emotion engine (e.g., IBM Watson) to analyze emotions from the user's voice and facial expressions.

[1323] The server recognizes the user's emotional state, such as whether they are stressed or excited.

[1324] Input and Output:

[1325] Input: User's voice and facial expression data when entering Japanese instructions.

[1326] Output: User's emotional state (e.g., stress, excitement).

[1327] Step 5: Apply the generated code

[1328] explanation:

[1329] The server generates specific operation codes, sends them to the device, and applies emotion-based customization to the spreadsheet.

[1330] Specific behavior:

[1331] The server transmits the generated operation code to the terminal.

[1332] The terminal applies the received operation code to the spreadsheet. For example, it sets the formula "=A1+B1" in cell C1.

[1333] The device reflects the results of customization by the emotion engine, for example, by changing error messages to more gentle expressions.

[1334] Input and Output:

[1335] Input: Operation code sent from the server and analysis results of the emotion engine.

[1336] Output: The state in which a specific operation is performed on the spreadsheet.

[1337] Step 6: View the operation results

[1338] explanation:

[1339] The user reviews the results on a spreadsheet.

[1340] Specific behavior:

[1341] The user checks the results of the operation in the spreadsheet.

[1342] The device will show you whether formulas are entered correctly, charts are generated as expected, and conditional formatting and input restrictions are applied.

[1343] The user also checks the results of customization by the emotion engine, for example, checking that the error message has been changed to a more gentle expression.

[1344] Input and Output:

[1345] Input: The result of an operation performed on a spreadsheet.

[1346] Output: A spreadsheet that reflects the results of the operation, and a customized section based on the user's emotions.

[1347] (Application example 2)

[1348] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1349] Conventional spreadsheet operations require users to manually set formulas and graphs, apply conditional formatting, and restrict data input, which is time-consuming and labor-intensive. Furthermore, they fail to take into account the emotional state of the user, which can cause excessive stress. In particular, systems supporting operations such as factory robots require fast and accurate data processing, so systems that are both efficient and considerate of the user's emotions are needed.

[1350] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a generating means for analyzing Japanese instructions, an applying means for applying a numerical formula generated by the generating means to a spreadsheet, an applying means for applying a graph generated by the generating means to a spreadsheet, an applying means for applying a conditional format generated by the generating means to the spreadsheet, an applying means for applying a data input restriction generated by the generating means to the spreadsheet, and a means for analyzing a user's emotions using an emotion analyzing means and customizing according to the results. This makes it possible to improve the efficiency of operations on the spreadsheet and provide a comfortable operating environment that takes user emotions into consideration.

[1351] "Instructions in Japanese" refers to requests from users regarding operations and settings written in Japanese as a natural language.

[1352] "Generation means" refers to equipment or software that has the function of analyzing Japanese instructions and generating numerical formulas, graphs, conditional formatting, and data input restrictions based on them.

[1353] "Application means" refers to equipment or software having a function for applying the formulas, graphs, conditional formatting, and data input restrictions generated by the generation means to a spreadsheet.

[1354] "Emotion analysis means" refers to equipment or software that has the function of recognizing emotions by analyzing a user's facial expressions, voice, actions, etc.

[1355] "Server" refers to a computer system that has the role of integrating the generation means, application means, emotion analysis means, etc., and provides the computational resources to execute these functions.

[1356] A "spreadsheet" is an electronic spreadsheet application that allows you to enter data into units called cells and manage and analyze the data using numerical formulas and graphs.

[1357] "Customization" refers to improving the comfort and efficiency of operation by changing and adjusting the operation screen and feedback content according to the user's emotions and usage situation.

[1358] This invention relates to a system that analyzes instructions in Japanese and automatically performs specific operations on a spreadsheet, and further has the function of analyzing the user's emotions and customizing the operations.

[1359] The system consists of a server, a terminal, a generation means, an application means, and an emotion analysis means. The server processes various data and is responsible for running the generation means, application means, and emotion analysis means. The terminal provides an interface for user instruction input and displaying operation results.

[1360] Generating a Program

[1361] The system program performs the following processing.

[1362] 1. Analysis of Japanese instructions

[1363] The server uses a generation means to analyze the Japanese instructions received from the user, and the generation means uses natural language processing technology to identify the content of the instructions and generate numerical expressions, graphs, conditional formats, and data input restrictions.

[1364] 2. Applying generated content

[1365] Formulas and graphs generated on the server are reflected in the spreadsheet through application means, such as entering a formula in a cell or generating a graph in a specified range.

[1366] 3. Emotion analysis

[1367] The emotion analysis means analyzes the user's voice and facial expressions to recognize their emotional state at that time. The server adjusts the response to operations and the expression of error messages based on the analysis results.

[1368] Hardware and software used

[1369] Hardware

[1370] Factory robots: Responsible for collecting data and inputting it into spreadsheets.

[1371] Facial recognition camera: Used to capture the user's facial expressions in real time.

[1372] PC or Server: Processes data and manages the system.

[1373] software

[1374] Python and related libraries (pandas, openpyxl, etc.): Used for data processing and analysis.

[1375] Natural Language Processing (NLP) models: used to parse Japanese instructions, including generative AI models.

[1376] EmotionRecognizer: A library for analyzing user emotions.

[1377] Specific examples

[1378] For example, the following operations are specifically performed.

[1379] Data Entry Example

[1380] When the user instructs the terminal, "Please input the temperature sensor data into A1," the server analyzes the instruction and inputs the temperature data into the specified cell through the application means.

[1381] Formula generation example

[1382] When the user instructs "Calculate the average value of B5 to B10 in D1," the server generates the formula =AVERAGE(B5:B10) using the generating means and applies it to cell D1 through the applying means.

[1383] Graph Generation Example

[1384] When a user instructs the server to "create a line graph based on the data in columns A and B," the server generates a line graph based on the data in the specified range and reflects it in the spreadsheet.

[1385] Example of customization using sentiment analysis

[1386] If the camera analyzes the user's facial expression as stressed, the server changes the error message to a softer one, such as "Please check again."

[1387] Prompt Sentence Examples

[1388] Below is an example of a prompt sentence entered into the system.

[1389] "Enter the temperature sensor data in A1"

[1390] "Calculate the average of B5 to B10 in D1."

[1391] "Create a line graph based on the data in columns A and B."

[1392] Please change the "An error has occurred" message to a more friendly one.

[1393] In this way, the present invention is a system that can automate spreadsheet operations while taking into consideration the user's feelings.

[1394] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1395] Step 1: Enter Japanese instructions

[1396] ---

[1397] The terminal receives instructions in Japanese from the user. The user inputs instructions in Japanese into the prompts displayed on the terminal. Examples of such input include "Enter the temperature sensor data in A1" and "Calculate the average value of B5 to B10 in D1."

[1398] Input: Japanese instructions from the user

[1399] Output: Japanese instruction data

[1400] Specific operation: The user enters instructions in Japanese into the input field on the terminal and presses the send button.

[1401] Step 2: Send instructions in Japanese

[1402] ---

[1403] The terminal sends the input Japanese instructions to the server, where this data is processed for analysis.

[1404] Input: Japanese instruction data

[1405] Output: Request to server

[1406] Specific operation: The terminal sends the Japanese instructions entered by the user to the server as an HTTP request.

[1407] Step 3: Parse the instructions

[1408] ---

[1409] The server analyzes the received Japanese instructions using a generation means, which uses a natural language processing model to identify the content of the instructions (numeric expressions, graphs, conditional formatting, data input restrictions).

[1410] Input: Japanese instruction data

[1411] Output: Analysis results (formulas, graph generation code, conditional formatting, data entry restrictions)

[1412] Specific Operation: The server parses the instruction using a natural language processing (NLP) model and generates a corresponding operation code.

[1413] Step 4: Sentiment Analysis

[1414] ---

[1415] The server analyzes the user's emotions using an emotion analysis engine, which determines the user's emotional state based on data acquired from a facial recognition camera and a voice input device.

[1416] Input: User's voice data and facial expression data

[1417] Output: Emotion analysis results (stress level, etc.)

[1418] Specific operation: The server uses emotion analysis means to analyze the user's voice and facial expressions to determine their stress level and emotional state.

[1419] Step 5: Apply the generated content

[1420] ---

[1421] The server applies the formulas, graphs, conditional formatting, and data entry restrictions generated based on the analysis results to the spreadsheet, and these contents are reflected in the spreadsheet through the application means.

[1422] Input: Analysis results and sentiment analysis results

[1423] Output: Updated data in the spreadsheet

[1424] Specific operations: The server applies the generated formulas, graphs, conditional formatting, and data entry restrictions to the spreadsheet using the application means, and changes the wording of error messages as necessary. The server also opens an existing spreadsheet file and inserts the corresponding data, formulas, and graphs into the specified cells.

[1425] Step 6: View the operation results

[1426] ---

[1427] The terminal displays the results of the spreadsheet operations based on the updated data received from the server, allowing the user to check the accuracy of the generated formulas and graphs.

[1428] Input: Spreadsheet update data

[1429] Output: Display of the updated spreadsheet

[1430] Specific behavior: The device redraws the spreadsheet based on the updated data received from the server and displays the results for the user to review.

[1431] The above are the processing steps of the system program that realizes the application example.

[1432] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1433] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1434] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1435] [Fourth embodiment]

[1436] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1437] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1438] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1439] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1440] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1441] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1442] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1443] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1444] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1445] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1446] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1447] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1448] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1449] This invention relates to a system that converts Japanese language instructions into specific operations for a spreadsheet. This system includes a generating means and an applying means, and can automatically generate and apply formulas, graphs, conditional formatting, data input restrictions, etc. in a spreadsheet using Japanese language instructions.

[1450] Example of a system

[1451] System Overview

[1452] The server receives Japanese instructions from the user via a program that runs in conjunction with the spreadsheet. The terminal displays the Japanese instructions entered by the user and sends them to the server for analysis. Based on the analysis results, code for the appropriate operation is generated, sent back to the terminal, and applied to the spreadsheet.

[1453] Functions and Operation Procedures

[1454] 1. User operations

[1455] The user opens a spreadsheet and selects one of the items in the "AI Support Menu." This menu includes items such as "Generate Formulas," "Generate Graphs," "Apply Conditional Formatting," "Set Data Entry Restrictions," and "Enter Instructions." When the user selects "Enter Instructions," a prompt for entering instructions appears on the device.

[1456] 2. Inputting and retrieving Japanese instructions

[1457] The user inputs instructions in Japanese into the prompt, such as "Please set the formula A1+B1 in C1." This input information is sent to the server and used for subsequent processing.

[1458] 3. Parsing the instructions

[1459] The server uses the generation means to analyze the Japanese instructions received from the user. During the analysis, it identifies whether the instruction is a mathematical formula, graph, conditional formatting, or data input restriction. It then generates specific code according to the operation.

[1460] 4. Applying the generated code

[1461] The server generates a specific code based on the analysis and returns it to the device. The device receives this code and performs operations to apply formulas, charts, conditional formatting, and data entry restrictions to the appropriate locations in the spreadsheet. For example, if the input instruction is "Set the formula A1+B1 in C1," the formula =A1+B1 will be set in cell C1.

[1462] 5. Displaying the operation results

[1463] Once applied, users can review the results in the spreadsheet to ensure formulas are entered correctly, charts are generated as expected, and conditional formatting and data entry restrictions are applied.

[1464] Specific use cases

[1465] Specific usage examples are described below.

[1466] Example 1: Generating a formula

[1467] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[1468] 2. Terminal: A prompt appears for you to enter instructions.

[1469] 3. User: Enter "Set formula A1+B1 in C1" in the prompt and press OK.

[1470] 4. Server: Analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[1471] 5. Terminal: Receive the generated code and set the formula "=A1+B1" in cell C1.

[1472] 6. User: Check the spreadsheet and verify that the formula is set correctly in cell C1.

[1473] Example 2: Generating a graph

[1474] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[1475] 2. Terminal: A prompt appears for you to enter instructions.

[1476] 3. User: Enter "Create a column chart based on the data from A1 to B5" in the prompt and press OK.

[1477] 4. Server: Parses the received instructions and generates the code to generate the required graph.

[1478] 5. Terminal: Receive the generated code and create a column chart based on the data in the specified range.

[1479] 6. User: Review the spreadsheet and verify that the column chart was generated correctly.

[1480] In this way, with the present invention, various spreadsheet operations are automatically performed simply by the user inputting instructions in Japanese, allowing even beginners to work efficiently and accurately.

[1481] The processing flow will be explained below.

[1482] Step 1:

[1483] The server automatically executes the onOpen function when the spreadsheet is opened, which generates a custom menu, the "AI Support Menu," that the user can use and adds it to the user interface.

[1484] Step 2:

[1485] The user opens the spreadsheet and selects "Enter Instructions" from the "AI Support Menu."

[1486] Step 3:

[1487] The terminal displays a prompt for the user to enter instructions in Japanese.

[1488] Step 4:

[1489] The user enters instructions into the prompt (e.g., "Please set the formula A1+B1 in C1") and presses OK.

[1490] Step 5:

[1491] The terminal receives input from the user and transmits the instructions to the server.

[1492] Step 6:

[1493] The server uses a generation means to analyze the received Japanese instructions and transmits the instructions to an external AI server.

[1494] Step 7:

[1495] The server receives the parsed results and generates code for specific operations based on the instructions (e.g., generating the formula "=A1+B1").

[1496] Step 8:

[1497] The server returns the generated specific operation code to the terminal.

[1498] Step 9:

[1499] The terminal executes the received code and applies it to the spreadsheet, for example, by setting the formula =A1+B1 in cell C1.

[1500] Step 10:

[1501] The user reviews the resulting changes in the spreadsheet and ensures that formulas, charts, and conditional formatting are applied correctly.

[1502] This series of steps allows users to efficiently perform spreadsheet operations using only Japanese instructions.

[1503] Example 1

[1504] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1505] Conventional spreadsheet software lacks the functionality to automatically generate and apply formulas, graphs, conditional formatting, and data entry constraints based on instructions in Japanese. This makes it difficult for beginner users, in particular, to operate spreadsheets efficiently. Furthermore, the software lacks a flexible interface that allows users to intuitively perform a variety of operations. A system that solves this problem and allows users to operate spreadsheets easily and accurately is needed.

[1506] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1507] In this invention, the server includes a generating means for analyzing instructions in Japanese, an applying means for applying a numerical formula generated by the generating means to a spreadsheet software, an applying means for applying a graph generated by the generating means to the spreadsheet software, an applying means for applying a conditional format generated by the generating means to the spreadsheet software, an applying means for applying a data input constraint generated by the generating means to the spreadsheet software, and a means for receiving instructions from a user terminal and transmitting the analysis results to the user terminal, thereby making it possible to automatically perform various operations on a spreadsheet based on instructions in Japanese.

[1508] The "generation means" is a means for analyzing Japanese instructions and converting them into concrete operations such as mathematical formulas, graphs, conditional formatting, and data input constraints.

[1509] The "application means" is a means for applying the formulas, graphs, conditional formats, and data input constraints generated by the generation means to the spreadsheet software.

[1510] A "formula" is an expression that specifies a combination of calculations or values ​​to be entered into a cell in a spreadsheet.

[1511] A "graph" is a diagram used in spreadsheet software to visually represent data.

[1512] "Conditional formatting" is a feature in spreadsheet software that automatically changes the formatting of cells based on specific conditions.

[1513] "Data input constraints" are settings that limit the type and range of data that can be entered into cells in spreadsheet software.

[1514] A "user terminal" is a computer device or equipment used to operate spreadsheet software and input instructions in Japanese.

[1515] "Analysis results" refers to the specific operation content and code obtained by the generation means when it analyzes the Japanese instructions.

[1516] A "spreadsheet" is software for organizing, calculating, analyzing, and visualizing data.

[1517] A "generative AI model" is a model that uses artificial intelligence to analyze Japanese instructions and generate appropriate operation codes.

[1518] "Prompt" refers to the input message or screen that is displayed when the user enters instructions in Japanese.

[1519] This invention relates to a system that analyzes Japanese language instructions and automatically generates and applies formulas, graphs, conditional formatting, and data input constraints to spreadsheet software. This system includes a generating means and an applying means, and can operate spreadsheets based on Japanese language instructions entered by the user. Specific embodiments of the invention are described below.

[1520] Required Hardware and Software

[1521] User device: A device such as a computer, tablet, or smartphone.

[1522] Server: Cloud server or dedicated server.

[1523] Spreadsheet software: Software for organizing, calculating, analyzing, and visualizing data (e.g., Google Sheets, Microsoft Excel).

[1524] Generative AI model: An artificial intelligence model for analyzing Japanese instructions and generating appropriate operation codes.

[1525] System Operation Overview

[1526] The server receives Japanese instructions sent from the user's device and analyzes them using a generative AI model. Based on the analysis results, it generates specific formulas, graphs, conditional formatting, and data input constraints and returns them to the user's device. The user's device then applies the results to the spreadsheet software.

[1527] Usage example

[1528] Example 1: Generating a formula

[1529] 1. The user selects "Enter instructions" from the "AI Support Menu."

[1530] 2. The terminal will prompt you to enter instructions.

[1531] 3. The user enters "Please set the formula A1+B1 in C1" in the prompt and presses the submit button.

[1532] 4. The server analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[1533] 5. The terminal receives the generated code and sets the formula "=A1+B1" in cell C1.

[1534] 6. The user checks the spreadsheet and confirms that the formula "=A1+B1" is correctly set in cell C1.

[1535] Example 2: Generating a graph

[1536] 1. The user selects "Enter instructions" from the "AI Support Menu."

[1537] 2. The terminal will prompt you to enter instructions.

[1538] 3. The user enters "Create a column chart based on the data from A1 to B5" in the prompt and presses the submit button.

[1539] 4. The server parses the received instructions and generates the necessary graph generation code.

[1540] 5. The terminal receives the generated code and creates a column chart based on the data in the specified range.

[1541] 6. The user reviews the spreadsheet and verifies that the column chart was generated correctly.

[1542] This system allows users to automatically perform complex operations in spreadsheet software simply by inputting instructions in Japanese, enabling even beginners to work efficiently and accurately.

[1543] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1544] Step 1:

[1545] The user opens a spreadsheet and selects "Enter instructions" from the "AI Support Menu." When the user selects an operation, a prompt for inputting instructions is displayed on the terminal. This prompt is an input interface that allows the user to enter specific instructions in Japanese. An example of input is "Please set the formula A1+B1 in C1." The output is a prompt screen that is displayed on the terminal.

[1546] Step 2:

[1547] The user inputs Japanese instructions into the displayed prompt and presses the send button. For example, the user can input instructions such as "Please set the formula A1+B1 in C1." The input is the Japanese instructions entered by the user, and the output is the instruction data sent from the user terminal to the server. In concrete terms, the prompt screen receives the user's input and sends it to the server via the network.

[1548] Step 3:

[1549] The server analyzes the received Japanese instructions. This analysis is performed using a generative AI model. The generative AI model analyzes the Japanese instructions and generates the appropriate operation code. The input is the Japanese instructions received by the server, and the output is the operation code (e.g., "=A1+B1") that is the result of the analysis. Specifically, the server passes the input instructions to the AI ​​model, which then analyzes the instructions and identifies the appropriate spreadsheet operation.

[1550] Step 4:

[1551] The server generates operation codes based on the analysis results. These operation codes represent the formulas, graphs, conditional formatting, and data input constraints to be applied to the spreadsheet. The input is the instructions analyzed by the AI ​​model, and the output is the specific operation code. As a specific operation, the server generates the specific spreadsheet operation code to be applied based on the analysis results returned by the AI ​​model.

[1552] Step 5:

[1553] The server sends the generated operation code to the terminal. The input is the operation code generated by the server, and the output is the operation code sent to the terminal. The specific operation is to send the code generated by the server back to the user's terminal via the network.

[1554] Step 6:

[1555] The terminal applies the operation code received from the server to the spreadsheet. For example, set the formula "=A1+B1" in cell C1. The input is the operation code received by the terminal, and the output is the formula and graph applied to the spreadsheet. Specifically, the terminal analyzes the operation code and performs the operation on the corresponding cell or range in the spreadsheet based on its contents.

[1556] Step 7:

[1557] The terminal reports the success of the operation to the server. The input is the result of application to the spreadsheet, and the output is a report of the success of the operation to the server. Specifically, the terminal checks the result of application to the spreadsheet, and if the operation was successful, it sends that information to the server.

[1558] Step 8:

[1559] The user checks the spreadsheet to ensure that their instructions have been executed correctly. For example, they check that the formula "=A1+B1" is set correctly in cell C1. The input is the user's visual confirmation, and the output is the visual confirmation result on the spreadsheet. Specifically, the user opens the spreadsheet and checks to see if the specified operations have been correctly reflected.

[1560] (Application example 1)

[1561] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1562] Modern spreadsheet operations are complex and time-consuming, especially when dealing with large amounts of data. It is particularly difficult for non-technical users and beginners to operate them efficiently. Furthermore, current systems often struggle to handle situations where quick on-site response is required, such as when it is difficult to operate while moving around or by voice input. There is a need for a system that can solve these issues and automate spreadsheet operations by simply inputting instructions via voice.

[1563] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1564] In this invention, the server includes a generating means for analyzing instructions in Japanese, an applying means for applying a numerical formula generated by the generating means to a spreadsheet, an applying means for applying a graph generated by the generating means to a spreadsheet, an applying means for applying a conditional format generated by the generating means to a spreadsheet, an applying means for applying a data input restriction generated by the generating means to a spreadsheet, a means for analyzing a user's voice instructions and automating spreadsheet operations, and a means for receiving the voice instructions via a smart device. This allows spreadsheet operations to be automated with simple voice input of instructions, enabling non-technical people and those in the field to respond quickly.

[1565] The "means for generating Japanese language instructions" is a means for analyzing Japanese language instructions input by a user and interpreting their contents.

[1566] The "means for applying the numerical formula generated by the generating means to the spreadsheet" is a means for inputting the numerical formula generated from the analyzed Japanese instructions into an appropriate cell of the spreadsheet.

[1567] The "means for applying the graph generated by the generating means to the spreadsheet" is a means for introducing the graph analyzed and generated from the Japanese instructions into the spreadsheet.

[1568] The "application means for applying the conditional formatting generated by the generation means to the spreadsheet" is a means for applying the conditional formatting generated based on the user's instructions to a specified range of the spreadsheet.

[1569] The "means for applying the data input restrictions generated by the generating means to the spreadsheet" is a means for setting the data input restrictions generated based on the user's Japanese instructions to the spreadsheet.

[1570] The "means for analyzing a user's voice instructions and automating spreadsheet operations" refers to a means for converting a user's voice instructions into text, analyzing it, and automatically operating a spreadsheet.

[1571] The "means for acquiring voice instructions via a smart device" refers to a means for collecting and analyzing voice instructions from a user using a smart device.

[1572] MODE FOR CARRYING OUT THE INVENTION

[1573] The present invention provides a system for automating spreadsheet operations using voice instructions in Japanese on a smart device. Specific embodiments of the invention are described below.

[1574] System configuration that analyzes voice instructions and performs spreadsheet operations

[1575] The system includes the following major components:

[1576] 1. Smart terminal: A device that receives user voice commands (e.g., smart glasses or head-mounted displays).

[1577] 2. Server: Hardware and software (e.g., Google Sheets API, Python) to parse the voice instructions and generate the appropriate spreadsheet operations.

[1578] 3. Generator: A means (e.g., speech_recognition library, AI server) for parsing Japanese instructions and applying numerical formulas, graphs, conditional formatting, and data entry control operations to the spreadsheet.

[1579] Program processing and the hardware and software used

[1580] 1. Smart Devices:

[1581] Hardware: Smart glasses and head-mounted displays.

[1582] Software: Speech recognition library (e.g. speech_recognition).

[1583] Processing: The smart device collects the user's voice commands and converts them into text data.

[1584] 2. Server:

[1585] Hardware: Standard server.

[1586] Software: Python, Google Sheets API, OAuth2 client.

[1587] process:

[1588] The text data generated by speech recognition is analyzed and Japanese instructions are interpreted.

[1589] Generates code for spreadsheet operations depending on the processing content.

[1590] Example of operation and prompt statement

[1591] Examples of operations include:

[1592] Example 1: Inventory check

[1593] 1. User: "Please check the stock of product code 12345."

[1594] 2. Server: Analyzes the instructions and uses the Google Sheets API to retrieve inventory information for the corresponding product code from the spreadsheet.

[1595] 3. Terminal: The acquired inventory information is displayed on the smart terminal.

[1596] Example 2: Inventory Update

[1597] 1. User: "Please update the inventory of product code 67890 to 20."

[1598] 2. Server: Parses the instruction and generates code to update the stock quantity in the appropriate spreadsheet cell to 20.

[1599] 3. Terminal: Perform the update operation and confirm that the inventory has been updated to 20.

[1600] Example prompt sentence:

[1601] "Please check stock availability for product code 12345"

[1602] "Please update the stock of product code 67890 to 20."

[1603] In this manner, the present invention allows a user to efficiently automate spreadsheet operations through voice instructions.

[1604] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1605] Step 1:

[1606] The device receives voice instructions from a user via a smart device (e.g., smart glasses). The input is the user's voice instruction, and the output is voice data. This voice data is collected using the microphone of the smart device.

[1607] Step 2:

[1608] The device converts the acquired voice data into text data. Specifically, it uses a voice recognition library (e.g., speech_recognition) to analyze the voice data and generate text data. The input is voice data, and the output is text data.

[1609] Step 3:

[1610] The server receives the text data sent from the device and analyzes the user's instructions. The input is text data, and the output is the analyzed instructions. Natural language processing technology is used for the analysis to interpret the intent of the Japanese instructions.

[1611] Step 4:

[1612] The server generates code to perform the appropriate spreadsheet operations based on the parsed instructions. The input is the parsed instructions, and the output is the code to perform the spreadsheet operations. Specific operations include the use of algorithms to generate and apply formulas, graphs, conditional formatting, and data entry restrictions.

[1613] Step 5:

[1614] The server sends the generated code to the terminal. The input is the spreadsheet operation code, and the output is the transmitted data. The transmission is performed via data communication over a network.

[1615] Step 6:

[1616] The device applies the received spreadsheet operation code to the spreadsheet. The input is the spreadsheet operation code, and the output is the updated content of the spreadsheet. Specifically, the operation is performed on the specified cell or range of the spreadsheet using the Google Sheets API.

[1617] Step 7:

[1618] The device provides feedback to the user on the results of the spreadsheet updates. The input is the updated content of the spreadsheet, and the output is a display of the updated results. Specifically, the results are displayed on the smart device's display, and feedback is provided by voice or text.

[1619] Specific examples

[1620] Example 1: Inventory check

[1621] 1. User: "Please check the stock of product code 12345."

[1622] 2. Terminal: Converts voice instructions into text and sends it to the server.

[1623] 3. Server: Analyzes the instructions and uses the Google Sheets API to retrieve inventory information for the corresponding product code from the spreadsheet.

[1624] 4. Terminal: Display the retrieved inventory information to the user. It displays "There are 10 units of product code 12345 in stock."

[1625] Example 2: Inventory Update

[1626] 1. User: "Please update the inventory of product code 67890 to 20."

[1627] 2. Terminal: Converts voice instructions into text and sends it to the server.

[1628] 3. Server: Parses the instruction and generates code to update the stock quantity in the appropriate spreadsheet cell to 20.

[1629] 4. Terminal: Executes the update operation and displays the results to the user. It displays "The inventory of product code 67890 has been updated to 20."

[1630] Example prompt sentence:

[1631] "Please check stock availability for product code 12345"

[1632] "Please update the stock of product code 67890 to 20."

[1633] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1634] This invention relates to a system that converts Japanese language instructions into specific operations for a spreadsheet and combines it with an emotion engine that recognizes the user's emotions. The system includes a generation means, an application means, and an emotion engine, and automatically generates and applies formulas, graphs, conditional formatting, data input restrictions, etc. in a spreadsheet in response to Japanese language instructions, and also has a customization function based on the user's emotions.

[1635] System configuration

[1636] Example of a system

[1637] 1. User operations

[1638] The user opens a spreadsheet and selects one of the items in the "AI Support Menu." This menu includes items such as "Generate Formulas," "Generate Graphs," "Apply Conditional Formatting," "Set Data Input Restrictions," and "Enter Instructions." When the user selects "Enter Instructions," a prompt for entering instructions appears on the device.

[1639] 2. Inputting and retrieving Japanese instructions

[1640] The user inputs instructions in Japanese into the prompt, such as "Please set the formula A1+B1 in C1." This input information is sent to the server and used for subsequent processing.

[1641] 3. Parsing the instructions

[1642] The server uses the generation means to analyze the Japanese instructions received from the user. During the analysis, it identifies whether the instruction corresponds to a mathematical formula, graph, conditional formatting, or data input restriction. It then generates specific code according to the operation content.

[1643] 4. Emotion Recognition by Emotion Engine

[1644] The server uses an emotion engine to analyze the user's emotions when inputting instructions. For example, if the user is feeling stressed, the server recognizes that emotion and takes measures to reduce the user's workload.

[1645] 5. Applying the generated code

[1646] The server then returns the specific operation code it generated to the device. The device then receives this code and applies formulas, graphs, conditional formatting, and data entry restrictions to the appropriate locations in the spreadsheet. For example, if the input instruction is "Set the formula A1+B1 in C1," the formula =A1+B1 will be set in cell C1. In addition, based on the analysis results of the emotion engine, the device may change error messages to be more friendly, for example.

[1647] 6. Displaying the operation results

[1648] Once applied, users can review the results in their spreadsheet to ensure formulas are entered correctly, charts are generated as expected, conditional formatting and validation rules are applied, and any customizations made by the emotion engine are reflected.

[1649] Specific use cases

[1650] Example 1: Generating mathematical expressions and reflecting emotions

[1651] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[1652] 2. Terminal: A prompt appears for you to enter instructions.

[1653] 3. User: Enter "Set formula A1+B1 in C1" in the prompt and press OK.

[1654] 4. Server: Analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[1655] 5. Server: Analyzes the user's voice and facial expressions when inputting and detects emotions.

[1656] 6. Server: If the emotion engine recognizes high stress, it generates program code to change the error message to a gentler expression.

[1657] 7. Terminal: Receive the generated code, set the formula "=A1+B1" in cell C1, and adjust how the error message is displayed.

[1658] 8. User: Check the spreadsheet and confirm that the formula in cell C1 is set correctly and that the customized error message is displayed.

[1659] Example 2: Graph generation and sentiment reflection

[1660] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[1661] 2. Terminal: A prompt appears for you to enter instructions.

[1662] 3. User: Enter "Create a column chart based on the data from A1 to B5" in the prompt and press OK.

[1663] 4. Server: Parses the received instructions and generates the code to generate the required graph.

[1664] 5. Server: If the emotion engine recognizes that the user is highly excited, it generates program code to adjust the color and design of the chart.

[1665] 6. Terminal: Receive the generated code, create a column chart based on the specified range of data, and adjust the chart design.

[1666] 7. User: Review the spreadsheet to ensure the column chart was generated correctly and reflects the sentiment customizations.

[1667] In this way, the present invention analyzes the user's emotions and performs customization based on the analysis, thereby improving the efficiency of spreadsheet operations and providing a more comfortable operating environment for the user.

[1668] The processing flow will be explained below.

[1669] Step 1:

[1670] The server automatically executes the onOpen function when the spreadsheet is opened, which adds an "AI Support Menu" to the user interface.

[1671] Step 2:

[1672] The user opens the spreadsheet and selects "Enter Instructions" from the "AI Support Menu."

[1673] Step 3:

[1674] The terminal displays a prompt for the user to enter instructions in Japanese.

[1675] Step 4:

[1676] The user enters instructions into the prompt (e.g., "Set the formula A1+B1 in C1") and presses OK.

[1677] Step 5:

[1678] The terminal receives input from the user and transmits the instructions to the server.

[1679] Step 6:

[1680] The server uses a generation means to analyze the received Japanese instructions and transmits the instructions to an external AI server.

[1681] Step 7:

[1682] The server receives the analysis results from the AI ​​server and generates code for specific operations based on the instructions (e.g., generating the formula "=A1+B1").

[1683] Step 8:

[1684] The server uses an emotion engine to collect emotion data from the user when inputting commands. This emotion data is obtained by means of voice tone analysis, facial expression analysis, etc.

[1685] Step 9:

[1686] The server analyzes the acquired emotional data and identifies the user's emotional state. For example, it determines that the user is feeling stressed.

[1687] Step 10:

[1688] The server generates code to tailor the outcome of the operation based on the emotion, for example by customizing the wording of error messages to be more gentle.

[1689] Step 11:

[1690] The server returns the generated specific operation code and an adjustment code based on the emotion to the terminal.

[1691] Step 12:

[1692] The terminal executes the received code and applies it to the spreadsheet, for example, by setting the formula =A1+B1 in cell C1 and customizing the error message.

[1693] Step 13:

[1694] Users can review the results of their spreadsheet changes, ensuring that formulas, charts, and conditional formatting are applied correctly, and even view customized error messages.

[1695] This series of steps allows users to efficiently operate spreadsheets using only Japanese instructions, and furthermore, customization based on emotions provides a more comfortable operating environment.

[1696] Example 2

[1697] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1698] Existing spreadsheet operations are largely manual, requiring time and effort, especially when it comes to complex formulas, data entry restrictions, and conditional formatting. Furthermore, there are currently no methods to reduce the stress and discomfort users feel while using spreadsheets. This reduces user efficiency and makes operations more cumbersome, resulting in an uncomfortable working environment.

[1699] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1700] In this invention, the server includes a generating means for analyzing Japanese instructions, an applying means for applying the numerical formula generated by the generating means to the spreadsheet, an applying means for applying the graph generated by the generating means to the spreadsheet, an applying means for applying the conditional format generated by the generating means to the spreadsheet, an applying means for applying the numerical input restriction generated by the generating means to the spreadsheet, an emotion engine for analyzing the user's emotions, and a means for customizing the spreadsheet operation based on the analysis result of the emotion engine. This not only enables the user's instructions to be automatically reflected in the spreadsheet, but also enables customization according to the user's emotional state.

[1701] "Instructions in Japanese" refers to operational commands or requests written in Japanese.

[1702] "Generation means" refers to a device or program that has the function of analyzing input Japanese instructions and generating specific operation codes.

[1703] A "numeric formula" is a mathematical formula used in a spreadsheet to perform calculations.

[1704] A "spreadsheet" is an application that allows you to manage data in a tabular format and apply formulas, graphs, conditional formatting, etc.

[1705] The "application means" refers to a device or program that has the function of applying the generated numerical formulas, graphs, conditional formats, data input restrictions, etc. to a spreadsheet.

[1706] A "graph" is a tool used to visually represent data in a spreadsheet.

[1707] Conditional formatting is a feature that changes the display format of cells in a spreadsheet based on their values ​​or conditions.

[1708] "Data input restrictions" refers to a function that restricts the range and format of data entered into a spreadsheet.

[1709] An "emotion engine" is a device or program for analyzing a user's emotional state from their voice, facial expressions, etc.

[1710] "Means for customization" refers to a device or program that has the function of adjusting the operation and display of a spreadsheet based on the analysis results of the emotion engine.

[1711] "External AI Server" means a server equipped with external artificial intelligence used for parsing and generating instructions.

[1712] This invention is a system that converts Japanese instructions into specific operations for a spreadsheet and further customizes the operations by recognizing the user's emotions. This system includes a generation means, an application means, and an emotion engine, and automatically performs operations such as formulas, graphs, conditional formatting, and data input restrictions in a spreadsheet using Japanese instructions, and also has a customization function based on the user's emotions.

[1713] Hardware and software used

[1714] Hardware:

[1715] Device (PC, tablet, etc.)

[1716] server

[1717] software:

[1718] Spreadsheet applications (e.g. Google Sheets)

[1719] Emotion engine (e.g. IBM Watson)

[1720] Instruction analysis engine (natural language processing model, e.g. GPT)

[1721] Example of a system

[1722] 1. User Action:

[1723] The user opens the spreadsheet on their device and selects "Enter instructions" from the "AI Support Menu."

[1724] 2. Entering and retrieving Japanese instructions:

[1725] The user receives a prompt on the spreadsheet for inputting instructions, and enters specific instructions in Japanese, for example, "Please set the formula A1+B1 in C1." This instruction is then sent to the server.

[1726] 3. Parsing the instructions:

[1727] The server uses an instruction analysis engine to analyze the Japanese instructions sent by the user, and based on the analysis results, generates specific operation codes for generating formulas, creating graphs, applying conditional formatting, and restricting data entry.

[1728] 4. Emotion Recognition with Emotion Engine:

[1729] The server uses an emotion engine to analyze the user's emotions during input, for example, detecting stress or excitement from the user's voice and facial expressions.

[1730] 5. Apply the generated code:

[1731] The server sends the generated operation code and the emotion-based customization result to the device, and the device applies the specific operation to the spreadsheet, for example, setting the formula "=A1+B1" in cell C1, and adjusts the error message according to the user's emotion.

[1732] 6. Displaying the operation results:

[1733] Users can see the results right in the spreadsheet, ensuring formulas are entered correctly, charts are generated as expected, and conditional formatting and data entry restrictions are applied.

[1734] Specific use cases

[1735] Example 1: Generating mathematical expressions and reflecting emotions

[1736] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[1737] 2. Terminal: A prompt appears for you to enter instructions.

[1738] 3. User: Enter "Set formula A1+B1 in C1" in the prompt and press OK.

[1739] 4. Server: Analyzes the received instructions and generates the code to generate the formula "=A1+B1".

[1740] 5. Server: Analyzes the user's voice and facial expressions when inputting and detects emotions.

[1741] 6. Server: If the emotion engine recognizes high stress, it generates program code to change the error message to a gentler expression.

[1742] 7. Terminal: Receive the generated code, set the formula "=A1+B1" in cell C1, and adjust how the error message is displayed.

[1743] 8. User: Check the spreadsheet and confirm that the formula in cell C1 is set correctly and that the customized error message is displayed.

[1744] Example 2: Graph generation and sentiment reflection

[1745] 1. User: Select "Enter Instructions" from the "AI Support Menu".

[1746] 2. Terminal: Displays a prompt for input of instructions.

[1747] 3. User: Enter "Create a column chart based on the data from A1 to B5" in the prompt and press OK.

[1748] 4. Server: Parses the received instructions and creates the code to generate the required graphs.

[1749] 5. Server: Create program code to adjust the color and design of the graph when the emotion engine recognizes that the user is highly excited.

[1750] 6. Terminal: Receive the generated code, create a column chart based on the specified range of data, and adjust the design.

[1751] 7. User: Review the spreadsheet and verify that the column chart is generated correctly and reflects the sentiment-based customizations.

[1752] Prompt Sentence Examples

[1753] Example 1 Formula Generation Prompt:

[1754] "Set the formula A1+B1 in C1"

[1755] Example 2 graph generation prompt:

[1756] "Create a column chart based on the data from A1 to B5."

[1757] In this way, the present invention analyzes the user's emotions and performs customization based on the analysis, thereby improving the efficiency of spreadsheet operations and providing a more comfortable operating environment for the user.

[1758] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1759] Program processing flow

[1760] Step 1: User interaction

[1761] explanation:

[1762] The user opens a spreadsheet application and selects "Enter Instructions" from the AI ​​Support menu.

[1763] Specific behavior:

[1764] The user opens the spreadsheet on a device such as a PC or tablet.

[1765] The device displays the "AI Support Menu" and waits for a selection from the user.

[1766] The user selects "Enter Instructions."

[1767] Input and Output:

[1768] Input: The action of a user interacting with a spreadsheet and selecting "Enter Instructions" from a menu.

[1769] Output: A terminal prompt will appear for you to enter instructions.

[1770] Step 2: Enter and retrieve Japanese instructions

[1771] explanation:

[1772] The user enters specific Japanese instructions for the spreadsheet into a prompt, which are then sent to the server.

[1773] Specific behavior:

[1774] The terminal displays prompts to the user for input of instructions.

[1775] The user enters "Please set the formula A1+B1 in C1" in the prompt and presses the submit button.

[1776] When the terminal receives the instruction, it transmits the instruction to the server.

[1777] Input and Output:

[1778] Input: The user enters specific Japanese instructions into the prompt and submits them.

[1779] Output: The user's Japanese instructions are sent to the server.

[1780] Step 3: Parse the instructions

[1781] explanation:

[1782] The server analyzes the Japanese instructions sent by the user and generates the required specific operation codes.

[1783] Specific behavior:

[1784] The server uses a generative AI model (e.g., GPT) to parse the received Japanese instructions.

[1785] The server analyzes the instruction and determines whether it is a formula, a graph, conditional formatting, or data input restrictions.

[1786] The server generates specific code for each operation. For example, if you request "Set the formula A1+B1 in C1," it generates the formula "=A1+B1."

[1787] Input and Output:

[1788] Input: The user's Japanese instructions sent to the server.

[1789] Output: Specific operation code based on the analysis results.

[1790] Step 4: Emotion Recognition with the Emotion Engine

[1791] explanation:

[1792] The server uses an emotion engine to analyze the user's emotions when inputting Japanese commands.

[1793] Specific behavior:

[1794] The server uses an emotion engine (e.g., IBM Watson) to analyze emotions from the user's voice and facial expressions.

[1795] The server recognizes the user's emotional state, such as whether they are stressed or excited.

[1796] Input and Output:

[1797] Input: User's voice and facial expression data when entering Japanese instructions.

[1798] Output: User's emotional state (e.g., stress, excitement).

[1799] Step 5: Apply the generated code

[1800] explanation:

[1801] The server generates specific operation codes, sends them to the device, and applies emotion-based customization to the spreadsheet.

[1802] Specific behavior:

[1803] The server transmits the generated operation code to the terminal.

[1804] The terminal applies the received operation code to the spreadsheet. For example, it sets the formula "=A1+B1" in cell C1.

[1805] The device reflects the results of customization by the emotion engine, for example, by changing error messages to more gentle expressions.

[1806] Input and Output:

[1807] Input: Operation code sent from the server and analysis results of the emotion engine.

[1808] Output: The state in which a specific operation is performed on the spreadsheet.

[1809] Step 6: View the operation results

[1810] explanation:

[1811] The user reviews the results on a spreadsheet.

[1812] Specific behavior:

[1813] The user checks the results of the operation in the spreadsheet.

[1814] The device will show you whether formulas are entered correctly, charts are generated as expected, and conditional formatting and input restrictions are applied.

[1815] The user also checks the results of customization by the emotion engine, for example, checking that the error message has been changed to a more gentle expression.

[1816] Input and Output:

[1817] Input: The result of an operation performed on a spreadsheet.

[1818] Output: A spreadsheet that reflects the results of the operation, and a customized section based on the user's emotions.

[1819] (Application example 2)

[1820] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1821] Conventional spreadsheet operations require users to manually set formulas and graphs, apply conditional formatting, and restrict data input, which is time-consuming and labor-intensive. Furthermore, they fail to take into account the emotional state of the user, which can cause excessive stress. In particular, systems supporting operations such as factory robots require fast and accurate data processing, so systems that are both efficient and considerate of the user's emotions are needed.

[1822] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a generating means for analyzing Japanese instructions, an applying means for applying a numerical formula generated by the generating means to a spreadsheet, an applying means for applying a graph generated by the generating means to a spreadsheet, an applying means for applying a conditional format generated by the generating means to the spreadsheet, an applying means for applying a data input restriction generated by the generating means to the spreadsheet, and a means for analyzing a user's emotions using an emotion analyzing means and customizing according to the results. This makes it possible to improve the efficiency of operations on the spreadsheet and provide a comfortable operating environment that takes user emotions into consideration.

[1823] "Instructions in Japanese" refers to requests from users regarding operations and settings written in Japanese as a natural language.

[1824] "Generation means" refers to equipment or software that has the function of analyzing Japanese instructions and generating numerical formulas, graphs, conditional formatting, and data input restrictions based on them.

[1825] "Application means" refers to equipment or software having a function for applying the formulas, graphs, conditional formatting, and data input restrictions generated by the generation means to a spreadsheet.

[1826] "Emotion analysis means" refers to equipment or software that has the function of recognizing emotions by analyzing a user's facial expressions, voice, actions, etc.

[1827] "Server" refers to a computer system that has the role of integrating the generation means, application means, emotion analysis means, etc., and provides the computational resources to execute these functions.

[1828] A "spreadsheet" is an electronic spreadsheet application that allows you to enter data into units called cells and manage and analyze the data using numerical formulas and graphs.

[1829] "Customization" refers to improving the comfort and efficiency of operation by changing and adjusting the operation screen and feedback content according to the user's emotions and usage situation.

[1830] This invention relates to a system that analyzes instructions in Japanese and automatically performs specific operations on a spreadsheet, and further has the function of analyzing the user's emotions and customizing the operations.

[1831] The system consists of a server, a terminal, a generation means, an application means, and an emotion analysis means. The server processes various data and is responsible for running the generation means, application means, and emotion analysis means. The terminal provides an interface for user instruction input and displaying operation results.

[1832] Generating a Program

[1833] The system program performs the following processing.

[1834] 1. Analysis of Japanese instructions

[1835] The server uses a generation means to analyze the Japanese instructions received from the user, and the generation means uses natural language processing technology to identify the content of the instructions and generate numerical expressions, graphs, conditional formats, and data input restrictions.

[1836] 2. Applying generated content

[1837] Formulas and graphs generated on the server are reflected in the spreadsheet through application means, such as entering a formula in a cell or generating a graph in a specified range.

[1838] 3. Emotion analysis

[1839] The emotion analysis means analyzes the user's voice and facial expressions to recognize their emotional state at that time. The server adjusts the response to operations and the expression of error messages based on the analysis results.

[1840] Hardware and software used

[1841] Hardware

[1842] Factory robots: Responsible for collecting data and inputting it into spreadsheets.

[1843] Facial recognition camera: Used to capture the user's facial expressions in real time.

[1844] PC or Server: Processes data and manages the system.

[1845] software

[1846] Python and related libraries (pandas, openpyxl, etc.): Used for data processing and analysis.

[1847] Natural Language Processing (NLP) models: used to parse Japanese instructions, including generative AI models.

[1848] EmotionRecognizer: A library for analyzing user emotions.

[1849] Specific examples

[1850] For example, the following operations are specifically performed.

[1851] Data Entry Example

[1852] When the user instructs the terminal, "Please input the temperature sensor data into A1," the server analyzes the instruction and inputs the temperature data into the specified cell through the application means.

[1853] Formula generation example

[1854] When the user instructs "Calculate the average value of B5 to B10 in D1," the server generates the formula =AVERAGE(B5:B10) using the generating means and applies it to cell D1 through the applying means.

[1855] Graph Generation Example

[1856] When a user instructs the server to "create a line graph based on the data in columns A and B," the server generates a line graph based on the data in the specified range and reflects it in the spreadsheet.

[1857] Example of customization using sentiment analysis

[1858] If the camera analyzes the user's facial expression as stressed, the server changes the error message to a softer one, such as "Please check again."

[1859] Prompt Sentence Examples

[1860] Below is an example of a prompt sentence entered into the system.

[1861] "Enter the temperature sensor data in A1"

[1862] "Calculate the average of B5 to B10 in D1."

[1863] "Create a line graph based on the data in columns A and B."

[1864] Please change the "An error has occurred" message to a more friendly one.

[1865] In this way, the present invention is a system that can automate spreadsheet operations while taking into consideration the user's feelings.

[1866] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1867] Step 1: Enter Japanese instructions

[1868] ---

[1869] The terminal receives instructions in Japanese from the user. The user inputs instructions in Japanese into the prompts displayed on the terminal. Examples of such input include "Enter the temperature sensor data in A1" and "Calculate the average value of B5 to B10 in D1."

[1870] Input: Japanese instructions from the user

[1871] Output: Japanese instruction data

[1872] Specific operation: The user enters instructions in Japanese into the input field on the terminal and presses the send button.

[1873] Step 2: Send instructions in Japanese

[1874] ---

[1875] The terminal sends the input Japanese instructions to the server, where this data is processed for analysis.

[1876] Input: Japanese instruction data

[1877] Output: Request to server

[1878] Specific operation: The terminal sends the Japanese instructions entered by the user to the server as an HTTP request.

[1879] Step 3: Parse the instructions

[1880] ---

[1881] The server analyzes the received Japanese instructions using a generation means, which uses a natural language processing model to identify the content of the instructions (numeric expressions, graphs, conditional formatting, data input restrictions).

[1882] Input: Japanese instruction data

[1883] Output: Analysis results (formulas, graph generation code, conditional formatting, data entry restrictions)

[1884] Specific Operation: The server parses the instruction using a natural language processing (NLP) model and generates a corresponding operation code.

[1885] Step 4: Sentiment Analysis

[1886] ---

[1887] The server analyzes the user's emotions using an emotion analysis engine, which determines the user's emotional state based on data acquired from a facial recognition camera and a voice input device.

[1888] Input: User's voice data and facial expression data

[1889] Output: Emotion analysis results (stress level, etc.)

[1890] Specific operation: The server uses emotion analysis means to analyze the user's voice and facial expressions to determine their stress level and emotional state.

[1891] Step 5: Apply the generated content

[1892] ---

[1893] The server applies the formulas, graphs, conditional formatting, and data entry restrictions generated based on the analysis results to the spreadsheet, and these contents are reflected in the spreadsheet through the application means.

[1894] Input: Analysis results and sentiment analysis results

[1895] Output: Updated data in the spreadsheet

[1896] Specific operations: The server applies the generated formulas, graphs, conditional formatting, and data entry restrictions to the spreadsheet using the application means, and changes the wording of error messages as necessary. The server also opens an existing spreadsheet file and inserts the corresponding data, formulas, and graphs into the specified cells.

[1897] Step 6: View the operation results

[1898] ---

[1899] The terminal displays the results of the spreadsheet operations based on the updated data received from the server, allowing the user to check the accuracy of the generated formulas and graphs.

[1900] Input: Spreadsheet update data

[1901] Output: Display of the updated spreadsheet

[1902] Specific behavior: The device redraws the spreadsheet based on the updated data received from the server and displays the results for the user to review.

[1903] The above are the processing steps of the system program that realizes the application example.

[1904] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1905] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1906] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1907] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1908] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1909] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1910] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1911] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1912] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1913] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1914] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1915] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1916] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1917] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1918] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1919] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1920] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1921] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1922] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1923] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1924] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1925] The following is further disclosed regarding the above embodiment.

[1926] (Claim 1)

[1927] generating means for parsing Japanese instructions;

[1928] applying means for applying the numerical formula generated by said generating means to a spreadsheet;

[1929] applying means for applying the graph generated by the generating means to a spreadsheet;

[1930] applying means for applying the conditional formatting generated by said generating means to a spreadsheet;

[1931] an application means for applying the data input restriction generated by the generation means to a spreadsheet;

[1932] A system including:

[1933] (Claim 2)

[1934] 10. The system of claim 1, wherein the generating means further comprises means for customizing a user interface of the spreadsheet.

[1935] (Claim 3)

[1936] 10. The system of claim 1, wherein the generating means further comprises means for receiving the instructions and using an external AI server to analyze them.

[1937] "Example 1"

[1938] (Claim 1)

[1939] generating means for parsing Japanese instructions;

[1940] application means for applying the numerical formula generated by the generation means to a spreadsheet software;

[1941] an application means for applying the graph generated by the generation means to a spreadsheet software;

[1942] an application means for applying the conditional format generated by the generation means to a spreadsheet software;

[1943] an application means for applying the data input constraints generated by the generation means to a spreadsheet software;

[1944] means for receiving instructions from a user terminal and transmitting analysis results to the user terminal;

[1945] A system including:

[1946] (Claim 2)

[1947] 10. The system of claim 1, wherein the generating means further comprises means for customizing a user interface of the spreadsheet.

[1948] (Claim 3)

[1949] 10. The system of claim 1, wherein the generating means further comprises means for receiving the instruction and using an external generative AI model to analyze it.

[1950] "Application Example 1"

[1951] (Claim 1)

[1952] generating means for parsing Japanese instructions;

[1953] applying means for applying the numerical formula generated by said generating means to a spreadsheet;

[1954] applying means for applying the graph generated by the generating means to a spreadsheet;

[1955] applying means for applying the conditional formatting generated by said generating means to a spreadsheet;

[1956] an application means for applying the data input restriction generated by the generation means to a spreadsheet;

[1957] A means for analyzing a user's voice instructions and automating spreadsheet operations;

[1958] means for receiving voice instructions via the smart device;

[1959] A system including:

[1960] (Claim 2)

[1961] 10. The system of claim 1, wherein the generating means further comprises means for customizing a user interface of the spreadsheet.

[1962] (Claim 3)

[1963] 10. The system of claim 1, wherein the generating means further comprises means for receiving the instructions and using an external artificial intelligence server to analyze them.

[1964] "Example 2: Combining Emotion Engines"

[1965] (Claim 1)

[1966] generating means for parsing Japanese instructions;

[1967] applying means for applying the numerical formula generated by said generating means to a spreadsheet;

[1968] applying means for applying the graph generated by the generating means to a spreadsheet;

[1969] applying means for applying the conditional formatting generated by said generating means to a spreadsheet;

[1970] an application means for applying the input restriction of the numerical values ​​generated by the generation means to a spreadsheet;

[1971] an emotion engine that analyzes the user's emotions;

[1972] means for customizing spreadsheet operations based on the analysis results of the emotion engine;

[1973] A system including:

[1974] (Claim 2)

[1975] 10. The system of claim 1, wherein the generating means further comprises means for customizing a user interface of the spreadsheet.

[1976] (Claim 3)

[1977] 10. The system of claim 1, wherein the generating means further comprises means for receiving the instructions and using an external AI server to analyze them.

[1978] "Application example 2 when combining emotion engines"

[1979] (Claim 1)

[1980] generating means for parsing Japanese instructions;

[1981] applying means for applying the numerical formula generated by said generating means to a spreadsheet;

[1982] applying means for applying the graph generated by the generating means to a spreadsheet;

[1983] applying means for applying the conditional formatting generated by said generating means to a spreadsheet;

[1984] an application means for applying the data input restriction generated by the generation means to a spreadsheet;

[1985] A means for analyzing user emotions using emotion analysis means and customizing the service according to the results;

[1986] A system including:

[1987] (Claim 2)

[1988] 10. The system of claim 1, wherein the generating means further comprises means for customizing a user interface of the spreadsheet.

[1989] (Claim 3)

[1990] 10. The system of claim 1, wherein the generating means further comprises means for receiving the instructions and using an external artificial intelligence server to analyze them. [Explanation of symbols]

[1991] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. generating means for parsing Japanese instructions; applying means for applying the numerical formula generated by said generating means to a spreadsheet; applying means for applying the graph generated by the generating means to a spreadsheet; applying means for applying the conditional formatting generated by said generating means to a spreadsheet; an application means for applying the data input restriction generated by the generation means to a spreadsheet; A system including:

2. 10. The system of claim 1, wherein the generating means further comprises means for customizing a user interface of the spreadsheet.

3. 10. The system of claim 1, wherein the generating means further comprises means for receiving the instructions and using an external AI server to analyze them.

Images