System

Abstract

A system is provided.SOLUTION: A system comprising: an input unit for inputting a task desired to be automated by a user in a natural language; a transmission unit for transmitting contents of the automation task acquired from the input unit to a server; a reception unit for receiving the contents of the automation task transmitted by the transmission unit; a generation unit for generating an appropriate code and setting based on the contents of the automation task received by the reception unit; and a return unit for returning the code and setting generated by the generation unit to a terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Using existing automation tools and APIs requires advanced programming skills and specialized knowledge, which is a major barrier for many users. Also, the configuration and code creation required to efficiently combine and use multiple APIs is complicated, time-consuming, and time-consuming, so there is a need for a system that is easy for general users to use. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including an input means for a user to input a task to be automated in natural language, a transmission means for transmitting the content of the automation task acquired from the input means to a server, a reception means for receiving the content of the automation task transmitted by the transmission means, a generation means for generating appropriate code and settings based on the content of the automation task received by the reception means, and a return means for returning the code and settings generated by the generation means to a terminal. This allows users without advanced programming skills to easily achieve complex automation simply by inputting the task in natural language. Furthermore, the generation means automatically selects and uses multiple published APIs, thereby supporting efficient automation.

[0006] "User" refers to an end user who intends to use the system to perform automated tasks.

[0007] "Natural language" refers to languages ​​that humans use on a daily basis (e.g., Japanese, English, etc.), rather than programming languages.

[0008] "Input means" refers to an interface (e.g., a text box, voice input, etc.) that allows a user to input the task they want to automate in natural language.

[0009] "Transmission means" refers to a communication means, communication protocol (e.g., HTTP request), etc. for transmitting the contents of the automation task obtained from the input means to the server.

[0010] "Server" refers to a computer system for processing received data and returning generated code.

[0011] "Receiving means" refers to a means (e.g., an API endpoint, a database, etc.) for receiving the content of the automation task sent by the sending means on the server side.

[0012] "Generating means" refers to an algorithm or program (e.g., a generative AI model) for generating appropriate code and settings based on the content of the automation task received by the receiving means.

[0013] "Returning means" refers to a communication means (e.g., HTTP response) for returning the code and settings generated by the generating means to the user's terminal.

[0014] "Code" refers to textual data containing program instructions necessary to perform an automated task.

[0015] "Configuration" refers to data that includes information such as various parameters and environment variables that enable code to run correctly.

[0016] "Published API" means an application programming interface that is publicly available and available for use. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The system of the present invention provides a means for generating and automating appropriate code based on a user's natural language input, making it easy to accomplish automation tasks.

[0039] The system provides an input method on the terminal for users to input the tasks they want to automate in natural language. Using this interface, users can input the details of the tasks they want to automate. For example, they can input a task such as "notify me of the weather forecast by email every day at 9:00 AM."

[0040] The input task content is transmitted to the server by the transmitting means, and the server has a receiving means for receiving the transmitted task content and analyzing the content.

[0041] Next, the server generates appropriate code and settings based on the received task content. The generation method uses a generation AI (e.g., a model using natural language processing). The server inputs the task content into the generation AI, selects the necessary APIs, and generates code that integrates them. For example, code is generated that retrieves data from a weather forecast API and uses an email sending API.

[0042] The generated code is returned to the terminal through the return means. The terminal displays the code received from the server or places it in an execution environment to automate the task. The user can check and implement the generated code.

[0043] As a concrete example, if a user inputs "Receive a weather forecast email every day at 9 AM," the system will operate as follows: First, the user's input is received and sent to the server. The server analyzes the natural language task content and generates code using appropriate APIs (for example, a weather forecast acquisition API and an email sending API). The generated code is then returned to the user's device. The user can then use the returned code to automate the desired task.

[0044] This system allows users to automate complex tasks through natural language input without requiring specialized programming knowledge, which is expected to improve work efficiency and save time.

[0045] The processing flow will be explained below.

[0046] Step 1:

[0047] The user uses the input method on the device to input the task they want to automate in natural language. For example, they might input "Notify me of the weather forecast by email every day at 9:00 AM."

[0048] Step 2:

[0049] The terminal transmits the input task content to the server using the transmission means. Specifically, the input text data in natural language is sent to the server in the form of an HTTP request.

[0050] Step 3:

[0051] The server receives the task content through the receiving means. The server-side API endpoint receives this data and prepares it for analysis.

[0052] Step 4:

[0053] The server passes data to a generation AI based on the received task content, and automatically generates appropriate code and settings. A natural language processing model is used as the generation method, and specific APIs (for example, a weather forecast API and an email sending API) are selected according to the user's request, and the code is constructed.

[0054] Step 5:

[0055] The code and settings generated by the generating means are returned to the terminal by the returning means. The server transmits the generated code to the terminal in the form of an HTTP response.

[0056] Step 6:

[0057] The device receives the returned code and displays it for the user to review. The user can then review the generated code, deploy it in their environment, and run it.

[0058] Step 7:

[0059] The user runs the generated code, for example, by setting up a scheduled task to retrieve the weather forecast every day at 9:00 AM and send an email notification.

[0060] Example 1

[0061] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0062] Conventional automation task configuration requires advanced programming knowledge, making it difficult for general users to use. Furthermore, when multiple APIs are used in combination, selecting and integrating them can be time-consuming, making the construction of automation tasks cumbersome. To solve these problems, there is a demand for a system that can easily configure automation tasks and automatically select and integrate multiple APIs.

[0063] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0064] In this invention, the server includes input means for a user to input a task they wish to automate in natural language, transmission means for transmitting the content of the automation task acquired from the input means to the server, reception means for receiving the content of the automation task transmitted by the transmission means, analysis means for analyzing the content of the automation task received by the reception means and selecting necessary APIs and their combinations, generation means for generating appropriate code and settings using a generative AI model based on the APIs and their combinations selected by the analysis means, return means for returning the code and settings generated by the generation means to the terminal, and execution means for providing an execution environment for executing the code and settings returned by the return means. This enables a user to set an automation task and automatically select and integrate multiple APIs through natural language input without requiring advanced programming knowledge.

[0065] An "input means" is a device or mechanism that allows a user to input the task they want to automate in natural language.

[0066] The "transmission means" is a device or mechanism for transmitting the content of the automation task acquired from the input means to the server.

[0067] The "receiving means" is a device or mechanism for receiving the content of the automation task transmitted by the transmitting means.

[0068] The "analysis means" is a device or mechanism for analyzing the content of the automation task received by the reception means and selecting the necessary APIs and their combinations.

[0069] The "generation means" refers to a device or mechanism for generating appropriate code and settings using a generative AI model based on the APIs and their combinations selected by the analysis means.

[0070] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to generate appropriate code for a given task.

[0071] The "returning means" is a device or mechanism for returning the code and settings generated by the generating means to the terminal.

[0072] The "execution means" is a device or mechanism that provides an environment for executing the code and settings returned by the return means.

[0073] "User interface" refers to the interface that a user operates directly on a terminal to input and send the details of an automated task.

[0074] "API" stands for Application Program Interface, a mechanism for sharing functions between multiple pieces of software.

[0075] The system of the present invention provides a means for a user to input a task to be automated in natural language, enabling the user to accomplish the task without specialized programming knowledge. Specifically, the process of generating appropriate code from the user's input and automating it is described in detail below.

[0076] Hardware and Software Configuration

[0077] The system includes the following major hardware and software components:

[0078] Device: A device used by a user, such as a computer, smartphone, or tablet.

[0079] Server: A remote computer system for data processing and code generation.

[0080] User Interface: The interface through which users input automation tasks. This can be a web application or a dedicated desktop application.

[0081] Generative AI model: An artificial intelligence model that uses natural language processing techniques to generate appropriate code from user input. For example, GPT-3 plays this role.

[0082] Data processing and calculation

[0083] 1. User task input:

[0084] Users input the task they want to automate in natural language into the user interface on their device, for example, "Receive a weather forecast email every day at 9 a.m."

[0085] 2. Submit your input:

[0086] The device sends the task details entered by the user to the server using an HTTP request, encrypted with SSL / TLS to ensure security.

[0087] 3. Task analysis on the server:

[0088] The server analyzes the received task content. A natural language processing (NLP) module is used for the analysis, and the necessary APIs and processing procedures are extracted from the text. For example, "spaCy" is used as an NLP module.

[0089] 4. Generate code:

[0090] The server analyzes the received task content, inputs prompts to the generative AI model, and generates code using appropriate APIs and their combinations. The generated code is set up to automate the user's task.

[0091] 5. Returning the code:

[0092] The generated code is returned from the server to the device using an HTTP response, with the response data sent in JSON format.

[0093] 6. Code Execution and Automation:

[0094] The terminal can display the returned code to the user or place it in an execution environment to automate tasks. After the user confirms it, the code will be automatically executed at the set time to automate the task.

[0095] Specific examples

[0096] For example, if a user enters "Receive a weather forecast email every day at 9:00 AM," the system will work as follows:

[0097] 1. The user enters the task details into the corresponding input form and clicks the submit button.

[0098] 2. The terminal sends this input to the server.

[0099] 3. The server analyzes the input and determines that a weather forecast API and an email sending API are required.

[0100] 4. Enter the following prompts into the generative AI model to generate the appropriate code:

[0101] Generate a code that will get the weather forecast every day at 9am and notify a specified email address. Integrate your preferred API.

[0102] 5. The generated code is returned from the server to the device.

[0103] 6. The device will either display the returned code or run it automatically at a set time, automating the task of emailing the weather forecast every day at 9:00 AM.

[0104] As described above, users can easily set up and execute automation tasks without advanced programming knowledge.

[0105] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0106] Step 1:

[0107] The user inputs the task they want to automate into the device's user interface in natural language. For example, the user might input "Notify me of the weather forecast by email every day at 9:00 AM." This input is passed to the device as input data.

[0108] Specific behavior:

[0109] The user opens a browser, accesses the specified URL, enters the task details in the displayed input form, and clicks the submit button. The input data is saved in the device's memory, and the device proceeds to the next step.

[0110] Step 2:

[0111] The device acquires the input task content and sends it to the server. The task content as input data is converted to JSON format and sent via an HTTP POST request. This transmission passes the task content to the server.

[0112] Specific behavior:

[0113] The terminal converts the input content into JSON format and sends it to "https: / / example.com / api / submitTask" using a POST request. At this time, SSL / TLS encryption is used.

[0114] Step 3:

[0115] The server receives the task content sent. It receives the task content as received data and prepares it for analysis.

[0116] Specific behavior:

[0117] The server receives the HTTP POST request and extracts the submitted JSON data for parsing, which is stored in memory and passed to the next parsing step.

[0118] Step 4:

[0119] The server analyzes the received task content. Using a natural language processing (NLP) module, it extracts the necessary APIs and their combinations from the task content. It analyzes the task content as input data and generates API selection data.

[0120] Specific behavior:

[0121] The server runs an NLP module (e.g., "spaCy") and extracts the "weather forecast API" and "email sending API" from the task content. The analysis results are stored in a database and passed to the next step.

[0122] Step 5:

[0123] The server inputs a prompt sentence into a generative AI model (e.g., GPT-3) based on the analysis results, and generates appropriate code. The API selection data is used as input data, and the generated code is obtained as output data.

[0124] Specific behavior:

[0125] The server inputs the following prompt to the generated AI model:

[0126] Generate a code that will get the weather forecast every day at 9am and notify a specified email address. Integrate your preferred API.

[0127] The code returned by the generative AI model is stored in memory and passed to the next step.

[0128] Step 6:

[0129] The server returns the generated code to the device. The generated output data is converted to JSON format and sent to the device as an HTTP response. This sends the code to the device.

[0130] Specific behavior:

[0131] The server converts the generated code into JSON format and sends it to the device as an HTTP response, using information stored in a directory or database to send it to the appropriate endpoint.

[0132] Step 7:

[0133] The device receives the returned code, displays it to the user, and, if necessary, places it in the execution environment to automate tasks. The code is then provided to the user and the execution environment as output data.

[0134] Specific behavior:

[0135] The terminal will display the returned code in an editor for the user to confirm, and after the user confirms, they can register the code as an execution script in a scheduler (for example, a cron job) and set it to run every day at 9:00 AM.

[0136] Through the above steps, users can set up and execute automation tasks through natural language input without any programming knowledge.

[0137] (Application example 1)

[0138] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0139] Operating and maintaining conventional factory robots requires advanced specialized knowledge and programming skills, and often takes a lot of time and effort. Automating tasks in particular requires specialized programming, which has been an obstacle to improving productivity and work efficiency.

[0140] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0141] In this invention, the server includes an input means for allowing a user to input a task they wish to automate in natural language, a transmission means for transmitting the content of the automation task acquired from the input means to the server, a receiving means for receiving the content of the automation task transmitted by the transmission means, a generation means for generating appropriate code and settings based on the content of the automation task received by the receiving means, a generative AI model used in the generation means and including a function for automatically selecting an API and generating code based on a prompt statement, and a return means for returning the code and settings generated by the generation means to the terminal. This makes it possible to easily generate code for automating the operation and maintenance of factory robots based on the task input by the user in natural language, thereby enabling efficient task automation without specialized knowledge.

[0142] A "user" is a person or business entity that utilizes the system to configure and operate automated tasks.

[0143] An "input means" is a device or software that provides an interface for a user to input an automated task in natural language.

[0144] The "transmission means" is a communication device or software for transmitting the content of the automation task acquired from the input means to the server.

[0145] The "receiving means" is a communication device or software for receiving the content of the automated task transmitted by the transmitting means at the server.

[0146] "Generating means" means a device or software that generates appropriate code and settings based on the content of the automation task received by the receiving means, including a generative AI model.

[0147] A "generative AI model" is an artificial intelligence model that uses natural language processing to analyze the user's task content and generate the necessary code and settings.

[0148] "API" stands for Application Programming Interface, an interface used for communication and data exchange between different software programs.

[0149] A "prompt sentence" is a sentence that represents the task content and is input to the generative AI model, and is used as a trigger for code generation.

[0150] The "returning means" is a communication device or software for returning the code and settings generated by the generating means to the user's terminal.

[0151] A "terminal" is an electronic device such as a computer or smartphone that is directly operated by a user.

[0152] This invention relates to a system for automating the operation and maintenance tasks of factory robots. The system allows a user to input a task in natural language, and generates appropriate code and settings based on the input, achieving automation.

[0153] The system uses the following hardware and software: The hardware uses terminals such as factory robot control panels and smartphones, while the software uses OpenAI's GPT-3 model, an internet connection for communication, and the Python programming language.

[0154] System flow

[0155] 1. User interface input

[0156] Users use the input interface on their devices to input tasks they want to automate in natural language, such as "Check the robot's status every day at 3:00 PM and notify me if there is an error."

[0157] 2. Sending task details

[0158] The terminal transmits the content of the automation task acquired from the input means to the server using the transmission means.

[0159] 3. Receiving and analyzing task content

[0160] The server receives the task content sent by the sending means using the receiving means and analyzes it. The receiving means has a function for interpreting data in JSON format or text format.

[0161] 4. Code Generation

[0162] Based on the received task content, a generative AI model (GPT-3) is used as a generator to generate appropriate code and settings. This generator includes the ability to automatically select an API and generate code using prompt statements.

[0163] 5. Returning the Code

[0164] The generated code and settings are returned to the terminal using the return means, and the user can check the generated code and execute it as appropriate to automate the task.

[0165] Specific examples

[0166] Specific task input example:

[0167] The user inputs "restart all systems of the robot every Monday, and then send the logs of each module." This information is sent to the server and analyzed. The server uses a generative AI model to generate the code necessary for the two tasks of "restarting the robot" and "sending the logs," and returns it.

[0168] Example prompt sentence:

[0169] Task: Restart all systems of the robot every Monday and then send the logs of each module.

[0170] code:

[0171] In this way, the operation and maintenance of factory robots can be efficiently automated based on tasks entered by the user in natural language, making it possible for even users without specialized programming knowledge to easily achieve advanced automation.

[0172] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0173] Step 1:

[0174] The user uses the input interface on the terminal to input the task to be automated in natural language.

[0175] Input: A natural language task description entered by the user (e.g., "Check the robot's status every day at 3 PM and notify me if there is an error").

[0176] Output: The task content is displayed in the input interface.

[0177] Specific actions: The user types or speaks the task details into the input screen of their smartphone or computer.

[0178] Step 2:

[0179] The terminal transmits the task content acquired using the input means to the server via the transmission means.

[0180] Input: Task content obtained from the input interface

[0181] Output: The data format in which the task content is sent to the server (e.g., JSON or text format)

[0182] Specific operation: When you click the "Send" button on the device, the task content is sent to the server via the Internet.

[0183] Step 3:

[0184] The server receives the task contents transmitted from the transmitting means using the receiving means.

[0185] Input: Data of the submitted task

[0186] Output: The received task data is saved on the server.

[0187] Specific operation: The server's receiving module records the task details in the database.

[0188] Step 4:

[0189] The server analyzes the received task content and generates the appropriate code and configuration using a generative AI model.

[0190] Input: Received task content data

[0191] Output: Generated code and configuration (e.g. Python scripts, API calls)

[0192] Specific operation: A generative AI model (such as GPT-3) analyzes the task content and generates appropriate APIs and code based on the prompt text.

[0193] Step 5:

[0194] The server returns the code and settings generated by the generating means to the terminal using the returning means.

[0195] Input: Generated code and settings

[0196] Output: Code and configuration data returned to the device

[0197] What it does: The server converts the code and settings into packets and sends them over the Internet to the device.

[0198] Step 6:

[0199] The device receives the code and settings returned from the server and executes it as an automated task.

[0200] Input: Code and settings returned from the server

[0201] Output: Executed automation task (e.g. robot status check, notification execution)

[0202] What happens: A script is launched to execute code on the device, performing robot operations and notification tasks.

[0203] Through these steps, the operation and maintenance of factory robots are automated based on tasks entered by the user in natural language, making it possible to automate tasks efficiently even without specialized knowledge.

[0204] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0205] The system of the present invention effectively realizes automated tasks by combining the user's input of automated tasks in natural language with an emotion engine that recognizes and analyzes the user's emotions.

[0206] This system provides an input method on the terminal for users to input the tasks they want to automate in natural language. Using this interface, users can input the tasks they want to automate. For example, they can input "Notify me of the weather forecast by email every day at 9:00 AM."

[0207] The input task content is transmitted to the server by the transmitting means. The server receives the transmitted task content through the receiving means. At this time, the server uses an emotion engine to analyze emotions from the user's input content. The emotion engine can recognize the user's emotions (e.g., joy, sadness, anger) through natural language analysis or voice input analysis.

[0208] The server then uses a generator to generate appropriate code and settings based on the recognized emotion. For example, if the user is dissatisfied with the input, the server can adjust the notification method to provide more detailed information or additional feedback. The generator also automatically selects multiple public APIs and generates code and settings to effectively accomplish the task.

[0209] The generated code and settings are returned to the terminal through the return means. The terminal displays the code and settings received from the server so that the user can easily check them. The user can then run the generated code and settings in a local environment or a cloud environment.

[0210] For example, if a user inputs "Notify me of the weather forecast by email every day at 9 AM" and simultaneously recognizes that the user's emotion is excitement, the system can be implemented to notify the user not only of the weather forecast information for that specific snippet, but also of recommended tourist spots and outdoor events. This ensures that the results of the automated task match the user's emotion, providing a more satisfying experience.

[0211] Therefore, the system of the present invention allows users without advanced programming skills to easily accomplish complex automated tasks through natural language input and emotion recognition, which is expected to improve work efficiency and user experience.

[0212] The processing flow will be explained below.

[0213] Step 1:

[0214] The user uses the input method on the device to input the task they want to automate in natural language. For example, they might input "Notify me of the weather forecast by email every day at 9:00 AM."

[0215] Step 2:

[0216] The terminal transmits the input task content to the server using the transmission means. Specifically, the input text data in natural language is sent to the server in the form of an HTTP request.

[0217] Step 3:

[0218] The server receives the task content through the receiving means. The server-side API endpoint receives this data and prepares it for analysis.

[0219] Step 4:

[0220] The server passes the received task content to the emotion engine, which uses natural language analysis or voice input analysis to recognize the user's emotions. For example, it analyzes positive emotions (joy, excitement) and negative emotions (sadness, dissatisfaction).

[0221] Step 5:

[0222] The server uses a generation method to generate appropriate code and settings based on the recognized emotion. The generation method uses a natural language processing model to select APIs (e.g., weather forecast APIs and email sending APIs) while taking the user's emotion into consideration, and configures the optimal code.

[0223] Step 6:

[0224] The generated code and settings are returned to the terminal through the return means. The server transmits the generated code to the terminal in the form of an HTTP response.

[0225] Step 7:

[0226] The device receives the returned code and displays it for the user to review. The user can then review the generated code, deploy it in their environment, and run it.

[0227] Step 8:

[0228] The user runs the generated code, for example, by setting up a scheduled task to retrieve the weather forecast every day at 9:00 AM and send an email notification.

[0229] Example 2

[0230] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0231] Currently, automating tasks requires programming knowledge and skills, which is a significant hurdle for average users. Furthermore, performing tasks without considering the user's emotions can lead to unsatisfied users and result in poor results. Therefore, there is a need for a system that combines natural language task input with emotion recognition to improve user satisfaction and automate complex tasks.

[0232] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an input means for inputting a task that a user wants to automate in natural language, a transmission means for transmitting the content of the automation task acquired from the input means to the server, a reception means for receiving the content of the automation task transmitted by the transmission means, a sentiment analysis means for analyzing the user's sentiment based on the content of the automation task received by the reception means, a generation means for generating appropriate code and settings based on the sentiment recognized by the sentiment analysis means, and a return means for returning the code and settings generated by the generation means to the terminal. This enables a user without advanced programming skills to easily realize complex automation tasks through input in natural language and emotion recognition.

[0233] Understood. Below are definitions of key terms included in the rewritten claims.

[0234] A "user" is an individual or organization that utilizes the system to configure automated tasks.

[0235] A "task" is a series of operations or processes that a user wants to automate.

[0236] An "input means" is a device or software that includes an interface for a user to input a task in natural language.

[0237] The "transmission means" is a communication means for transmitting the task contents acquired from the input means to the server.

[0238] The "server" is a central processing unit that receives, analyzes, and processes task contents sent by users.

[0239] The "receiving means" is a means by which the server receives the task content transmitted by the transmitting means.

[0240] "Emotion analysis means" refers to software and algorithms that allow the server to analyze emotions from the user's input.

[0241] The "generating means" refers to a processing device and program for generating appropriate code and settings based on the emotion recognized by the emotion analyzing means.

[0242] The "returning means" is a communication means for returning the code and settings generated by the generating means from the server to the terminal.

[0243] "Terminal" means a device for a user to use input means and is used to display the code and settings returned from the server.

[0244] The system of the present invention allows a user to input an automation task using natural language, recognizes the user's emotions, and generates appropriate code and settings. This system is configured by combining an input unit, a transmission unit, a server, a reception unit, an emotion analysis unit, a generation unit, a return unit, and a terminal.

[0245] First, the user uses the input method on the device to input the task they want to automate in natural language. The input method can be a text field or a voice input interface. For example, they can input a specific task such as "Send me a weather forecast email every day at 9:00 AM."

[0246] The input task content is sent to the server by the sending means. A common communication protocol such as an HTTP request is used as the communication means. Next, the server receives the input task content through the receiving means. On the server side, the user's emotions are analyzed using emotion analysis means based on the received task content. The emotion analysis means used here may be a natural language analysis model (e.g., BERT or GPT-3).

[0247] The emotion analysis means recognizes emotions (e.g., joy, sadness, anger) from the user's input. The server then uses the generation means to automatically generate appropriate codes and settings based on the recognized emotions. For example, if the user feels "enjoyed" by the input, the generation means generates codes and settings to notify the user of recommended tourist spots in addition to the weather forecast.

[0248] The generating means automatically selects multiple published APIs and generates code for effectively performing the task. The generated code and settings are returned to the terminal through the returning means. The returning means may be an HTTP response or the like.

[0249] The device displays the code and configuration returned by the server to the user, who can review it and make any necessary corrections or approvals. The user can then run the completed code and configuration in a local or cloud environment. This allows users without advanced programming skills to easily accomplish complex automation tasks through natural language input and emotion recognition.

[0250] For example, if a user inputs "notify me of the weather forecast by email every day at 9:00 AM," and sentiment analysis identifies this as "excitement," the system will configure the system to notify users of information about recommended tourist spots and outdoor events in addition to the weather forecast. This allows users to achieve high satisfaction with the automated notifications.

[0251] An example of a prompt sentence might be:

[0252] "Every day at 9:00 AM, I'll email you the latest weather forecast and the best places to visit that day."

[0253] This allows users to intuitively and efficiently configure and execute automated tasks.

[0254] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0255] Step 1: User enters task

[0256] The user uses the input method on the device to input the task they want to automate in natural language. This interface includes a text field and voice input support. For example, the user might input "Receive a weather forecast email every day at 9:00 AM." The input is captured as text data.

[0257] Input: User's natural language task input (e.g., "Email me the weather forecast every day at 9 AM").

[0258] Output: Task content as text data.

[0259] Step 2: The device sends the task details to the server

[0260] The terminal transmits the acquired task content to the server via a transmission means. The transmission is performed using a communication protocol such as an HTTP request. Specifically, the terminal sends an HTTP "POST" request to the server, and includes the task content in the body of the request.

[0261] Input: Task content as text data.

[0262] Output: HTTP request to the server.

[0263] Step 3: The server receives the task

[0264] The server receives the task content sent through the receiving means. Specifically, the server analyzes the HTTP request and extracts the task content from the body. The received task content is stored as an internal data structure.

[0265] Input: HTTP request.

[0266] Output: Task details (text data) within the server.

[0267] Step 4: The server analyzes the emotions

[0268] The server uses a sentiment analysis method to analyze the sentiment of the user's input. A natural language analysis model such as BERT or GPT-3 can be used for this analysis. Specifically, the server passes the input text to a sentiment analysis API and obtains a sentiment label (e.g., "enjoyed").

[0269] Input: Task contents (text data) within the server.

[0270] Output: An emotion label (e.g., "enjoyment").

[0271] Step 5: The server generates the appropriate code and configuration

[0272] The server uses a generation method to generate appropriate code and settings based on the recognized emotion. Specifically, the server selects the appropriate API based on the emotion analysis results and automatically generates code to accomplish the task. For example, if the emotion is "excitement," it generates code that combines a weather forecast API and a tourist destination information API.

[0273] Input: emotion label, task content.

[0274] Output: Auto-generated code and configuration.

[0275] Step 6: The server returns the generated code

[0276] The generated code and settings are returned to the terminal through the return means. Specifically, the server sends the generated code to the terminal as an HTTP "POST" response.

[0277] Inputs: Auto-generated code and settings.

[0278] Output: HTTP response to the device.

[0279] Step 7: The terminal will display the code

[0280] The terminal displays the received code and settings to the user. Specifically, the terminal visually displays the received code on the interface so that the user can confirm the contents.

[0281] Input: The HTTP response from the server.

[0282] Output: The code and settings displayed on the interface.

[0283] Through the specific actions performed at each step, users can easily accomplish complex automation tasks through natural language input and emotion recognition, even without advanced programming skills.

[0284] (Application example 2)

[0285] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0286] Conventional automation systems could automatically process the content of tasks specified by the user, but they were unable to respond flexibly while taking into account the user's emotions and circumstances. This limited the improvement of user satisfaction and work efficiency. Furthermore, in production sites such as factories, there was a need for support to efficiently complete tasks while reducing worker stress and fatigue. A system that solves these issues and realizes task automation while taking into consideration the emotions of the user or worker is needed.

[0287] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means for a user to input a task they wish to automate in natural language, a generation means for generating appropriate code and settings based on the content of the automation task and the emotion analysis results acquired from the input means, and a return means for returning the code and settings generated by the generation means to the terminal. This enables task automation that takes the user's emotions into consideration, realizes more flexible and satisfying work support, and enables factory workers to efficiently complete tasks while reducing stress and fatigue.

[0288] An "input means" is a device or software that provides an interface for a user to input, in natural language, the task they wish to automate.

[0289] The "transmission means" is a device or software having a function for transmitting the content of the automation task acquired from the input means to the server via the network.

[0290] The "receiving means" is a device or software having a function for receiving the content of the automated task transmitted to the server by the transmitting means.

[0291] The "emotion analysis result" is emotional information analyzed from the user's input content, and is data generated by the emotion analysis engine.

[0292] The "generating means" is a device or software for generating appropriate code and settings based on the content of the automated task and the emotion analysis results received by the receiving means.

[0293] The "returning means" is a device or software having a function for returning the code and settings generated by the generating means to the terminal.

[0294] The term "user interface" refers to a screen or operating means that allows a user to input information in natural language on a terminal.

[0295] An "emotion analysis engine" is an algorithm and software that analyzes voice and text input from users and recognizes their emotions.

[0296] The embodiments of the present invention will be described in detail below.

[0297] First, a user uses a terminal to input the task to be automated in natural language. A user interface is provided on the terminal for the user to input instructions in natural language. This user interface is an interface that accepts natural language tasks through voice input or text input. For example, the user can input an instruction such as "move the next pallet to the conveyor belt."

[0298] Next, the input task content is transmitted from the terminal to the server, and this transmission means has a communication function for transmitting data to the server via a network.

[0299] The server analyzes the received task content and recognizes the user's emotions through a sentiment analysis engine. The sentiment analysis engine analyzes the text or voice data from the user's input to generate emotional data. For example, text analysis can determine whether the user is tired or happy.

[0300] The server then uses a generator to generate appropriate code and settings based on the received task content and sentiment analysis results. The generator automatically selects and combines a number of publicly available APIs to generate code to accomplish the required task.

[0301] The generated code and settings are returned from the server to the terminal. The return means has a communication function for transmitting the generated code and settings to the terminal via a network. The user can check the returned code and settings and make corrections or approvals if necessary.

[0302] For example, if a user inputs "move the next pallet to the conveyor belt" and the emotion analysis engine recognizes the user's emotion as "tired," the system can adjust to temporarily suspend the work, suggest a break, and notify the user when it's time to resume. In this way, it becomes possible to support efficient work while taking into consideration the worker's emotions.

[0303] Examples of prompts include:

[0304] "Move the next pallet to the conveyor belt."

[0305] "Text: 'Move the next pallet to the conveyor belt.'; Sentiment: 'Neutral'"

[0306] This system makes it possible to automate tasks while taking emotions into account, which is expected to improve user satisfaction and work efficiency.

[0307] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0308] Step 1:

[0309] The user uses the user interface on the device to input the task they want to automate in natural language. This input is captured by the input means and stored as digital data on the device. An example of this input might be, "Move the next pallet to the conveyor belt."

[0310] Step 2:

[0311] The input natural language task is sent to the server via the transmission means over the network. The transmission means packetizes the digital data stored on the terminal and converts it into a format that the server can receive. The output of this process is digital data containing the task content on the server side.

[0312] Step 3:

[0313] The server receives the transmitted task content using the receiving means. The received data includes the task content in natural language entered by the user. The server passes this data to the analysis engine and starts natural language analysis.

[0314] Step 4:

[0315] The server uses a natural language analysis engine to analyze the content of the input task, and also recognizes emotions through a sentiment analysis engine. The input is text data of the task content, and the sentiment analysis engine analyzes the text for positive, negative, and neutral aspects. For example, the input "Move the next pallet to the conveyor belt" is analyzed as "neutral."

[0316] Step 5:

[0317] The server generates appropriate code and settings based on the analyzed task content and emotion analysis results. The server uses a generation method to automatically select multiple APIs and dynamically generate the necessary code. For example, if the recognized emotion is "tired," it generates code that includes functions to pause and suggest breaks. The input of this step is the task content and emotion analysis results, and the output is the generated code and settings.

[0318] Step 6:

[0319] The generated code and settings are sent from the server to the terminal, again using the return means. The server packetizes the generated data over the network and converts it into a format acceptable to the terminal. The output of this process is the generated code and settings data that is transmitted to the user's terminal.

[0320] Step 7:

[0321] The device begins operation based on the received code and settings. The user can check the progress of the task and make corrections or approvals if necessary. For example, this can include time management during breaks and preparation for the next task. User feedback is input back into the system and can be reflected in subsequent task executions.

[0322] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0323] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0324] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0325] [Second embodiment]

[0326] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0327] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0328] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0329] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0330] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0331] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0332] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0333] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0334] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0335] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0336] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0337] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0338] The system of the present invention provides a means for generating and automating appropriate code based on a user's natural language input, making it easy to accomplish automation tasks.

[0339] The system provides an input method on the terminal for users to input the tasks they want to automate in natural language. Using this interface, users can input the details of the tasks they want to automate. For example, they can input a task such as "notify me of the weather forecast by email every day at 9:00 AM."

[0340] The input task content is transmitted to the server by the transmitting means, and the server has a receiving means for receiving the transmitted task content and analyzing the content.

[0341] Next, the server generates appropriate code and settings based on the received task content. The generation method uses a generation AI (e.g., a model using natural language processing). The server inputs the task content into the generation AI, selects the necessary APIs, and generates code that integrates them. For example, code is generated that retrieves data from a weather forecast API and uses an email sending API.

[0342] The generated code is returned to the terminal through the return means. The terminal displays the code received from the server or places it in an execution environment to automate the task. The user can check and implement the generated code.

[0343] As a concrete example, if a user inputs "Receive a weather forecast email every day at 9 AM," the system will operate as follows: First, the user's input is received and sent to the server. The server analyzes the natural language task content and generates code using appropriate APIs (for example, a weather forecast acquisition API and an email sending API). The generated code is then returned to the user's device. The user can then use the returned code to automate the desired task.

[0344] This system allows users to automate complex tasks through natural language input without requiring specialized programming knowledge, which is expected to improve work efficiency and save time.

[0345] The processing flow will be explained below.

[0346] Step 1:

[0347] The user uses the input method on the device to input the task they want to automate in natural language. For example, they might input "Notify me of the weather forecast by email every day at 9:00 AM."

[0348] Step 2:

[0349] The terminal transmits the input task content to the server using the transmission means. Specifically, the input text data in natural language is sent to the server in the form of an HTTP request.

[0350] Step 3:

[0351] The server receives the task content through the receiving means. The server-side API endpoint receives this data and prepares it for analysis.

[0352] Step 4:

[0353] The server passes data to a generation AI based on the received task content, and automatically generates appropriate code and settings. A natural language processing model is used as the generation method, and specific APIs (for example, a weather forecast API and an email sending API) are selected according to the user's request, and the code is constructed.

[0354] Step 5:

[0355] The code and settings generated by the generating means are returned to the terminal by the returning means. The server transmits the generated code to the terminal in the form of an HTTP response.

[0356] Step 6:

[0357] The device receives the returned code and displays it for the user to review. The user can then review the generated code, deploy it in their environment, and run it.

[0358] Step 7:

[0359] The user runs the generated code, for example, by setting up a scheduled task to retrieve the weather forecast every day at 9:00 AM and send an email notification.

[0360] Example 1

[0361] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0362] Conventional automation task configuration requires advanced programming knowledge, making it difficult for general users to use. Furthermore, when multiple APIs are used in combination, selecting and integrating them can be time-consuming, making the construction of automation tasks cumbersome. To solve these problems, there is a demand for a system that can easily configure automation tasks and automatically select and integrate multiple APIs.

[0363] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0364] In this invention, the server includes input means for a user to input a task they wish to automate in natural language, transmission means for transmitting the content of the automation task acquired from the input means to the server, reception means for receiving the content of the automation task transmitted by the transmission means, analysis means for analyzing the content of the automation task received by the reception means and selecting necessary APIs and their combinations, generation means for generating appropriate code and settings using a generative AI model based on the APIs and their combinations selected by the analysis means, return means for returning the code and settings generated by the generation means to the terminal, and execution means for providing an execution environment for executing the code and settings returned by the return means. This enables a user to set an automation task and automatically select and integrate multiple APIs through natural language input without requiring advanced programming knowledge.

[0365] An "input means" is a device or mechanism that allows a user to input the task they want to automate in natural language.

[0366] The "transmission means" is a device or mechanism for transmitting the content of the automation task acquired from the input means to the server.

[0367] The "receiving means" is a device or mechanism for receiving the content of the automation task transmitted by the transmitting means.

[0368] The "analysis means" is a device or mechanism for analyzing the content of the automation task received by the reception means and selecting the necessary APIs and their combinations.

[0369] The "generation means" refers to a device or mechanism for generating appropriate code and settings using a generative AI model based on the APIs and their combinations selected by the analysis means.

[0370] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to generate appropriate code for a given task.

[0371] The "returning means" is a device or mechanism for returning the code and settings generated by the generating means to the terminal.

[0372] The "execution means" is a device or mechanism that provides an environment for executing the code and settings returned by the return means.

[0373] "User interface" refers to the interface that a user operates directly on a terminal to input and send the details of an automated task.

[0374] "API" stands for Application Program Interface, a mechanism for sharing functions between multiple pieces of software.

[0375] The system of the present invention provides a means for a user to input a task to be automated in natural language, enabling the user to accomplish the task without specialized programming knowledge. Specifically, the process of generating appropriate code from the user's input and automating it is described in detail below.

[0376] Hardware and Software Configuration

[0377] The system includes the following major hardware and software components:

[0378] Device: A device used by a user, such as a computer, smartphone, or tablet.

[0379] Server: A remote computer system for data processing and code generation.

[0380] User Interface: The interface through which users input automation tasks. This can be a web application or a dedicated desktop application.

[0381] Generative AI model: An artificial intelligence model that uses natural language processing techniques to generate appropriate code from user input. For example, GPT-3 plays this role.

[0382] Data processing and calculation

[0383] 1. User task input:

[0384] Users input the task they want to automate in natural language into the user interface on their device, for example, "Receive a weather forecast email every day at 9 a.m."

[0385] 2. Submit your input:

[0386] The device sends the task details entered by the user to the server using an HTTP request, encrypted with SSL / TLS to ensure security.

[0387] 3. Task analysis on the server:

[0388] The server analyzes the received task content. A natural language processing (NLP) module is used for the analysis, and the necessary APIs and processing procedures are extracted from the text. For example, "spaCy" is used as an NLP module.

[0389] 4. Generate code:

[0390] The server analyzes the received task content, inputs prompts to the generative AI model, and generates code using appropriate APIs and their combinations. The generated code is set up to automate the user's task.

[0391] 5. Returning the code:

[0392] The generated code is returned from the server to the device using an HTTP response, with the response data sent in JSON format.

[0393] 6. Code Execution and Automation:

[0394] The terminal can display the returned code to the user or place it in an execution environment to automate tasks. After the user confirms it, the code will be automatically executed at the set time to automate the task.

[0395] Specific examples

[0396] For example, if a user enters "Receive a weather forecast email every day at 9:00 AM," the system will work as follows:

[0397] 1. The user enters the task details into the corresponding input form and clicks the submit button.

[0398] 2. The terminal sends this input to the server.

[0399] 3. The server analyzes the input and determines that a weather forecast API and an email sending API are required.

[0400] 4. Enter the following prompts into the generative AI model to generate the appropriate code:

[0401] Generate a code that will get the weather forecast every day at 9am and notify a specified email address. Integrate your preferred API.

[0402] 5. The generated code is returned from the server to the device.

[0403] 6. The device will either display the returned code or run it automatically at a set time, automating the task of emailing the weather forecast every day at 9:00 AM.

[0404] As described above, users can easily set up and execute automation tasks without advanced programming knowledge.

[0405] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0406] Step 1:

[0407] The user inputs the task they want to automate into the device's user interface in natural language. For example, the user might input "Notify me of the weather forecast by email every day at 9:00 AM." This input is passed to the device as input data.

[0408] Specific behavior:

[0409] The user opens a browser, accesses the specified URL, enters the task details in the displayed input form, and clicks the submit button. The input data is saved in the device's memory, and the device proceeds to the next step.

[0410] Step 2:

[0411] The device acquires the input task content and sends it to the server. The task content as input data is converted to JSON format and sent via an HTTP POST request. This transmission passes the task content to the server.

[0412] Specific behavior:

[0413] The terminal converts the input content into JSON format and sends it to "https: / / example.com / api / submitTask" using a POST request. At this time, SSL / TLS encryption is used.

[0414] Step 3:

[0415] The server receives the task content sent. It receives the task content as received data and prepares it for analysis.

[0416] Specific behavior:

[0417] The server receives the HTTP POST request and extracts the submitted JSON data for parsing, which is stored in memory and passed to the next parsing step.

[0418] Step 4:

[0419] The server analyzes the received task content. Using a natural language processing (NLP) module, it extracts the necessary APIs and their combinations from the task content. It analyzes the task content as input data and generates API selection data.

[0420] Specific behavior:

[0421] The server runs an NLP module (e.g., "spaCy") and extracts the "weather forecast API" and "email sending API" from the task content. The analysis results are stored in a database and passed to the next step.

[0422] Step 5:

[0423] The server inputs a prompt sentence into a generative AI model (e.g., GPT-3) based on the analysis results, and generates appropriate code. The API selection data is used as input data, and the generated code is obtained as output data.

[0424] Specific behavior:

[0425] The server inputs the following prompt to the generated AI model:

[0426] Generate a code that will get the weather forecast every day at 9am and notify a specified email address. Integrate your preferred API.

[0427] The code returned by the generative AI model is stored in memory and passed to the next step.

[0428] Step 6:

[0429] The server returns the generated code to the device. The generated output data is converted to JSON format and sent to the device as an HTTP response. This sends the code to the device.

[0430] Specific behavior:

[0431] The server converts the generated code into JSON format and sends it to the device as an HTTP response, using information stored in a directory or database to send it to the appropriate endpoint.

[0432] Step 7:

[0433] The device receives the returned code, displays it to the user, and, if necessary, places it in the execution environment to automate tasks. The code is then provided to the user and the execution environment as output data.

[0434] Specific behavior:

[0435] The terminal will display the returned code in an editor for the user to confirm, and after the user confirms, they can register the code as an execution script in a scheduler (for example, a cron job) and set it to run every day at 9:00 AM.

[0436] Through the above steps, users can set up and execute automation tasks through natural language input without any programming knowledge.

[0437] (Application example 1)

[0438] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0439] Operating and maintaining conventional factory robots requires advanced specialized knowledge and programming skills, and often takes a lot of time and effort. Automating tasks in particular requires specialized programming, which has been an obstacle to improving productivity and work efficiency.

[0440] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0441] In this invention, the server includes an input means for allowing a user to input a task they wish to automate in natural language, a transmission means for transmitting the content of the automation task acquired from the input means to the server, a receiving means for receiving the content of the automation task transmitted by the transmission means, a generation means for generating appropriate code and settings based on the content of the automation task received by the receiving means, a generative AI model used in the generation means and including a function for automatically selecting an API and generating code based on a prompt statement, and a return means for returning the code and settings generated by the generation means to the terminal. This makes it possible to easily generate code for automating the operation and maintenance of factory robots based on the task input by the user in natural language, thereby enabling efficient task automation without specialized knowledge.

[0442] A "user" is a person or business entity that utilizes the system to configure and operate automated tasks.

[0443] An "input means" is a device or software that provides an interface for a user to input an automated task in natural language.

[0444] The "transmission means" is a communication device or software for transmitting the content of the automation task acquired from the input means to the server.

[0445] The "receiving means" is a communication device or software for receiving the content of the automated task transmitted by the transmitting means at the server.

[0446] "Generating means" means a device or software that generates appropriate code and settings based on the content of the automation task received by the receiving means, including a generative AI model.

[0447] A "generative AI model" is an artificial intelligence model that uses natural language processing to analyze the user's task content and generate the necessary code and settings.

[0448] "API" stands for Application Programming Interface, an interface used for communication and data exchange between different software programs.

[0449] A "prompt sentence" is a sentence that represents the task content and is input to the generative AI model, and is used as a trigger for code generation.

[0450] The "returning means" is a communication device or software for returning the code and settings generated by the generating means to the user's terminal.

[0451] A "terminal" is an electronic device such as a computer or smartphone that is directly operated by a user.

[0452] This invention relates to a system for automating the operation and maintenance tasks of factory robots. The system allows a user to input a task in natural language, and generates appropriate code and settings based on the input, achieving automation.

[0453] The system uses the following hardware and software: The hardware uses terminals such as factory robot control panels and smartphones, while the software uses OpenAI's GPT-3 model, an internet connection for communication, and the Python programming language.

[0454] System flow

[0455] 1. User interface input

[0456] Users use the input interface on their devices to input tasks they want to automate in natural language, such as "Check the robot's status every day at 3:00 PM and notify me if there is an error."

[0457] 2. Sending task details

[0458] The terminal transmits the content of the automation task acquired from the input means to the server using the transmission means.

[0459] 3. Receiving and analyzing task content

[0460] The server receives the task content sent by the sending means using the receiving means and analyzes it. The receiving means has a function for interpreting data in JSON format or text format.

[0461] 4. Code Generation

[0462] Based on the received task content, a generative AI model (GPT-3) is used as a generator to generate appropriate code and settings. This generator includes the ability to automatically select an API and generate code using prompt statements.

[0463] 5. Returning the Code

[0464] The generated code and settings are returned to the terminal using the return means, and the user can check the generated code and execute it as appropriate to automate the task.

[0465] Specific examples

[0466] Specific task input example:

[0467] The user inputs "restart all systems of the robot every Monday, and then send the logs of each module." This information is sent to the server and analyzed. The server uses a generative AI model to generate the code necessary for the two tasks of "restarting the robot" and "sending the logs," and returns it.

[0468] Example prompt sentence:

[0469] Task: Restart all systems of the robot every Monday and then send the logs of each module.

[0470] code:

[0471] In this way, the operation and maintenance of factory robots can be efficiently automated based on tasks entered by the user in natural language, making it possible for even users without specialized programming knowledge to easily achieve advanced automation.

[0472] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0473] Step 1:

[0474] The user uses the input interface on the terminal to input the task to be automated in natural language.

[0475] Input: A natural language task description entered by the user (e.g., "Check the robot's status every day at 3 PM and notify me if there is an error").

[0476] Output: The task content is displayed in the input interface.

[0477] Specific actions: The user types or speaks the task details into the input screen of their smartphone or computer.

[0478] Step 2:

[0479] The terminal transmits the task content acquired using the input means to the server via the transmission means.

[0480] Input: Task content obtained from the input interface

[0481] Output: The data format in which the task content is sent to the server (e.g., JSON or text format)

[0482] Specific operation: When you click the "Send" button on the device, the task content is sent to the server via the Internet.

[0483] Step 3:

[0484] The server receives the task contents transmitted from the transmitting means using the receiving means.

[0485] Input: Data of the submitted task

[0486] Output: The received task data is saved on the server.

[0487] Specific operation: The server's receiving module records the task details in the database.

[0488] Step 4:

[0489] The server analyzes the received task content and generates the appropriate code and configuration using a generative AI model.

[0490] Input: Received task content data

[0491] Output: Generated code and configuration (e.g. Python scripts, API calls)

[0492] Specific operation: A generative AI model (such as GPT-3) analyzes the task content and generates appropriate APIs and code based on the prompt text.

[0493] Step 5:

[0494] The server returns the code and settings generated by the generating means to the terminal using the returning means.

[0495] Input: Generated code and settings

[0496] Output: Code and configuration data returned to the device

[0497] What it does: The server converts the code and settings into packets and sends them over the Internet to the device.

[0498] Step 6:

[0499] The device receives the code and settings returned from the server and executes it as an automated task.

[0500] Input: Code and settings returned from the server

[0501] Output: Executed automation task (e.g. robot status check, notification execution)

[0502] What happens: A script is launched to execute code on the device, performing robot operations and notification tasks.

[0503] Through these steps, the operation and maintenance of factory robots are automated based on tasks entered by the user in natural language, making it possible to automate tasks efficiently even without specialized knowledge.

[0504] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0505] The system of the present invention effectively realizes automated tasks by combining the user's input of automated tasks in natural language with an emotion engine that recognizes and analyzes the user's emotions.

[0506] This system provides an input method on the terminal for users to input the tasks they want to automate in natural language. Using this interface, users can input the tasks they want to automate. For example, they can input "Notify me of the weather forecast by email every day at 9:00 AM."

[0507] The input task content is transmitted to the server by the transmitting means. The server receives the transmitted task content through the receiving means. At this time, the server uses an emotion engine to analyze emotions from the user's input content. The emotion engine can recognize the user's emotions (e.g., joy, sadness, anger) through natural language analysis or voice input analysis.

[0508] The server then uses a generator to generate appropriate code and settings based on the recognized emotion. For example, if the user is dissatisfied with the input, the server can adjust the notification method to provide more detailed information or additional feedback. The generator also automatically selects multiple public APIs and generates code and settings to effectively accomplish the task.

[0509] The generated code and settings are returned to the terminal through the return means. The terminal displays the code and settings received from the server so that the user can easily check them. The user can then run the generated code and settings in a local environment or a cloud environment.

[0510] For example, if a user inputs "Notify me of the weather forecast by email every day at 9 AM" and simultaneously recognizes that the user's emotion is excitement, the system can be implemented to notify the user not only of the weather forecast information for that specific snippet, but also of recommended tourist spots and outdoor events. This ensures that the results of the automated task match the user's emotion, providing a more satisfying experience.

[0511] Therefore, the system of the present invention allows users without advanced programming skills to easily accomplish complex automated tasks through natural language input and emotion recognition, which is expected to improve work efficiency and user experience.

[0512] The processing flow will be explained below.

[0513] Step 1:

[0514] The user uses the input method on the device to input the task they want to automate in natural language. For example, they might input "Notify me of the weather forecast by email every day at 9:00 AM."

[0515] Step 2:

[0516] The terminal transmits the input task content to the server using the transmission means. Specifically, the input text data in natural language is sent to the server in the form of an HTTP request.

[0517] Step 3:

[0518] The server receives the task content through the receiving means. The server-side API endpoint receives this data and prepares it for analysis.

[0519] Step 4:

[0520] The server passes the received task content to the emotion engine, which uses natural language analysis or voice input analysis to recognize the user's emotions. For example, it analyzes positive emotions (joy, excitement) and negative emotions (sadness, dissatisfaction).

[0521] Step 5:

[0522] The server uses a generation method to generate appropriate code and settings based on the recognized emotion. The generation method uses a natural language processing model to select APIs (e.g., weather forecast APIs and email sending APIs) while taking the user's emotion into consideration, and configures the optimal code.

[0523] Step 6:

[0524] The generated code and settings are returned to the terminal through the return means. The server transmits the generated code to the terminal in the form of an HTTP response.

[0525] Step 7:

[0526] The device receives the returned code and displays it for the user to review. The user can then review the generated code, deploy it in their environment, and run it.

[0527] Step 8:

[0528] The user runs the generated code, for example, by setting up a scheduled task to retrieve the weather forecast every day at 9:00 AM and send an email notification.

[0529] Example 2

[0530] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0531] Currently, automating tasks requires programming knowledge and skills, which is a significant hurdle for average users. Furthermore, performing tasks without considering the user's emotions can lead to unsatisfied users and result in poor results. Therefore, there is a need for a system that combines natural language task input with emotion recognition to improve user satisfaction and automate complex tasks.

[0532] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an input means for inputting a task that a user wants to automate in natural language, a transmission means for transmitting the content of the automation task acquired from the input means to the server, a reception means for receiving the content of the automation task transmitted by the transmission means, a sentiment analysis means for analyzing the user's sentiment based on the content of the automation task received by the reception means, a generation means for generating appropriate code and settings based on the sentiment recognized by the sentiment analysis means, and a return means for returning the code and settings generated by the generation means to the terminal. This enables a user without advanced programming skills to easily realize complex automation tasks through input in natural language and emotion recognition.

[0533] Understood. Below are definitions of key terms included in the rewritten claims.

[0534] A "user" is an individual or organization that utilizes the system to configure automated tasks.

[0535] A "task" is a series of operations or processes that a user wants to automate.

[0536] An "input means" is a device or software that includes an interface for a user to input a task in natural language.

[0537] The "transmission means" is a communication means for transmitting the task contents acquired from the input means to the server.

[0538] The "server" is a central processing unit that receives, analyzes, and processes task contents sent by users.

[0539] The "receiving means" is a means by which the server receives the task content transmitted by the transmitting means.

[0540] "Emotion analysis means" refers to software and algorithms that allow the server to analyze emotions from the user's input.

[0541] The "generating means" refers to a processing device and program for generating appropriate code and settings based on the emotion recognized by the emotion analyzing means.

[0542] The "returning means" is a communication means for returning the code and settings generated by the generating means from the server to the terminal.

[0543] "Terminal" means a device for a user to use input means and is used to display the code and settings returned from the server.

[0544] The system of the present invention allows a user to input an automation task using natural language, recognizes the user's emotions, and generates appropriate code and settings. This system is configured by combining an input unit, a transmission unit, a server, a reception unit, an emotion analysis unit, a generation unit, a return unit, and a terminal.

[0545] First, the user uses the input method on the device to input the task they want to automate in natural language. The input method can be a text field or a voice input interface. For example, they can input a specific task such as "Send me a weather forecast email every day at 9:00 AM."

[0546] The input task content is sent to the server by the sending means. A common communication protocol such as an HTTP request is used as the communication means. Next, the server receives the input task content through the receiving means. On the server side, the user's emotions are analyzed using emotion analysis means based on the received task content. The emotion analysis means used here may be a natural language analysis model (e.g., BERT or GPT-3).

[0547] The emotion analysis means recognizes emotions (e.g., joy, sadness, anger) from the user's input. The server then uses the generation means to automatically generate appropriate codes and settings based on the recognized emotions. For example, if the user feels "enjoyed" by the input, the generation means generates codes and settings to notify the user of recommended tourist spots in addition to the weather forecast.

[0548] The generating means automatically selects multiple published APIs and generates code for effectively performing the task. The generated code and settings are returned to the terminal through the returning means. The returning means may be an HTTP response or the like.

[0549] The device displays the code and configuration returned by the server to the user, who can review it and make any necessary corrections or approvals. The user can then run the completed code and configuration in a local or cloud environment. This allows users without advanced programming skills to easily accomplish complex automation tasks through natural language input and emotion recognition.

[0550] For example, if a user inputs "notify me of the weather forecast by email every day at 9:00 AM," and sentiment analysis identifies this as "excitement," the system will configure the system to notify users of information about recommended tourist spots and outdoor events in addition to the weather forecast. This allows users to achieve high satisfaction with the automated notifications.

[0551] An example of a prompt sentence might be:

[0552] "Every day at 9:00 AM, I'll email you the latest weather forecast and the best places to visit that day."

[0553] This allows users to intuitively and efficiently configure and execute automated tasks.

[0554] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0555] Step 1: User enters task

[0556] The user uses the input method on the device to input the task they want to automate in natural language. This interface includes a text field and voice input support. For example, the user might input "Receive a weather forecast email every day at 9:00 AM." The input is captured as text data.

[0557] Input: User's natural language task input (e.g., "Email me the weather forecast every day at 9 AM").

[0558] Output: Task content as text data.

[0559] Step 2: The device sends the task details to the server

[0560] The terminal transmits the acquired task content to the server via a transmission means. The transmission is performed using a communication protocol such as an HTTP request. Specifically, the terminal sends an HTTP "POST" request to the server, and includes the task content in the body of the request.

[0561] Input: Task content as text data.

[0562] Output: HTTP request to the server.

[0563] Step 3: The server receives the task

[0564] The server receives the task content sent through the receiving means. Specifically, the server analyzes the HTTP request and extracts the task content from the body. The received task content is stored as an internal data structure.

[0565] Input: HTTP request.

[0566] Output: Task details (text data) within the server.

[0567] Step 4: The server analyzes the emotions

[0568] The server uses a sentiment analysis method to analyze the sentiment of the user's input. A natural language analysis model such as BERT or GPT-3 can be used for this analysis. Specifically, the server passes the input text to a sentiment analysis API and obtains a sentiment label (e.g., "enjoyed").

[0569] Input: Task contents (text data) within the server.

[0570] Output: An emotion label (e.g., "enjoyment").

[0571] Step 5: The server generates the appropriate code and configuration

[0572] The server uses a generation method to generate appropriate code and settings based on the recognized emotion. Specifically, the server selects the appropriate API based on the emotion analysis results and automatically generates code to accomplish the task. For example, if the emotion is "excitement," it generates code that combines a weather forecast API and a tourist destination information API.

[0573] Input: emotion label, task content.

[0574] Output: Auto-generated code and configuration.

[0575] Step 6: The server returns the generated code

[0576] The generated code and settings are returned to the terminal through the return means. Specifically, the server sends the generated code to the terminal as an HTTP "POST" response.

[0577] Inputs: Auto-generated code and settings.

[0578] Output: HTTP response to the device.

[0579] Step 7: The terminal will display the code

[0580] The terminal displays the received code and settings to the user. Specifically, the terminal visually displays the received code on the interface so that the user can confirm the contents.

[0581] Input: The HTTP response from the server.

[0582] Output: The code and settings displayed on the interface.

[0583] Through the specific actions performed at each step, users can easily accomplish complex automation tasks through natural language input and emotion recognition, even without advanced programming skills.

[0584] (Application example 2)

[0585] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0586] Conventional automation systems could automatically process the content of tasks specified by the user, but they were unable to respond flexibly while taking into account the user's emotions and circumstances. This limited the improvement of user satisfaction and work efficiency. Furthermore, in production sites such as factories, there was a need for support to efficiently complete tasks while reducing worker stress and fatigue. A system that solves these issues and realizes task automation while taking into consideration the emotions of the user or worker is needed.

[0587] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means for a user to input a task they wish to automate in natural language, a generation means for generating appropriate code and settings based on the content of the automation task and the emotion analysis results acquired from the input means, and a return means for returning the code and settings generated by the generation means to the terminal. This enables task automation that takes the user's emotions into consideration, realizes more flexible and satisfying work support, and enables factory workers to efficiently complete tasks while reducing stress and fatigue.

[0588] An "input means" is a device or software that provides an interface for a user to input, in natural language, the task they wish to automate.

[0589] The "transmission means" is a device or software having a function for transmitting the content of the automation task acquired from the input means to the server via the network.

[0590] The "receiving means" is a device or software having a function for receiving the content of the automated task transmitted to the server by the transmitting means.

[0591] The "emotion analysis result" is emotional information analyzed from the user's input content, and is data generated by the emotion analysis engine.

[0592] The "generating means" is a device or software for generating appropriate code and settings based on the content of the automated task and the emotion analysis results received by the receiving means.

[0593] The "returning means" is a device or software having a function for returning the code and settings generated by the generating means to the terminal.

[0594] The term "user interface" refers to a screen or operating means that allows a user to input information in natural language on a terminal.

[0595] An "emotion analysis engine" is an algorithm and software that analyzes voice and text input from users and recognizes their emotions.

[0596] The embodiments of the present invention will be described in detail below.

[0597] First, a user uses a terminal to input the task to be automated in natural language. A user interface is provided on the terminal for the user to input instructions in natural language. This user interface is an interface that accepts natural language tasks through voice input or text input. For example, the user can input an instruction such as "move the next pallet to the conveyor belt."

[0598] Next, the input task content is transmitted from the terminal to the server, and this transmission means has a communication function for transmitting data to the server via a network.

[0599] The server analyzes the received task content and recognizes the user's emotions through a sentiment analysis engine. The sentiment analysis engine analyzes the text or voice data from the user's input to generate emotional data. For example, text analysis can determine whether the user is tired or happy.

[0600] The server then uses a generator to generate appropriate code and settings based on the received task content and sentiment analysis results. The generator automatically selects and combines a number of publicly available APIs to generate code to accomplish the required task.

[0601] The generated code and settings are returned from the server to the terminal. The return means has a communication function for transmitting the generated code and settings to the terminal via a network. The user can check the returned code and settings and make corrections or approvals if necessary.

[0602] For example, if a user inputs "move the next pallet to the conveyor belt" and the emotion analysis engine recognizes the user's emotion as "tired," the system can adjust to temporarily suspend the work, suggest a break, and notify the user when it's time to resume. In this way, it becomes possible to support efficient work while taking into consideration the worker's emotions.

[0603] Examples of prompts include:

[0604] "Move the next pallet to the conveyor belt."

[0605] "Text: 'Move the next pallet to the conveyor belt.'; Sentiment: 'Neutral'"

[0606] This system makes it possible to automate tasks while taking emotions into account, which is expected to improve user satisfaction and work efficiency.

[0607] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0608] Step 1:

[0609] The user uses the user interface on the device to input the task they want to automate in natural language. This input is captured by the input means and stored as digital data on the device. An example of this input might be, "Move the next pallet to the conveyor belt."

[0610] Step 2:

[0611] The input natural language task is sent to the server via the transmission means over the network. The transmission means packetizes the digital data stored on the terminal and converts it into a format that the server can receive. The output of this process is digital data containing the task content on the server side.

[0612] Step 3:

[0613] The server receives the transmitted task content using the receiving means. The received data includes the task content in natural language entered by the user. The server passes this data to the analysis engine and starts natural language analysis.

[0614] Step 4:

[0615] The server uses a natural language analysis engine to analyze the content of the input task, and also recognizes emotions through a sentiment analysis engine. The input is text data of the task content, and the sentiment analysis engine analyzes the text for positive, negative, and neutral aspects. For example, the input "Move the next pallet to the conveyor belt" is analyzed as "neutral."

[0616] Step 5:

[0617] The server generates appropriate code and settings based on the analyzed task content and emotion analysis results. The server uses a generation method to automatically select multiple APIs and dynamically generate the necessary code. For example, if the recognized emotion is "tired," it generates code that includes functions to pause and suggest breaks. The input of this step is the task content and emotion analysis results, and the output is the generated code and settings.

[0618] Step 6:

[0619] The generated code and settings are sent from the server to the terminal, again using the return means. The server packetizes the generated data over the network and converts it into a format acceptable to the terminal. The output of this process is the generated code and settings data that is transmitted to the user's terminal.

[0620] Step 7:

[0621] The device begins operation based on the received code and settings. The user can check the progress of the task and make corrections or approvals if necessary. For example, this can include time management during breaks and preparation for the next task. User feedback is input back into the system and can be reflected in subsequent task executions.

[0622] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0623] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0624] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0625] [Third embodiment]

[0626] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0627] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0628] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0629] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0630] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0631] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0632] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0633] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0634] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0635] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0636] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0637] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0638] The system of the present invention provides a means for generating and automating appropriate code based on a user's natural language input, making it easy to accomplish automation tasks.

[0639] The system provides an input method on the terminal for users to input the tasks they want to automate in natural language. Using this interface, users can input the details of the tasks they want to automate. For example, they can input a task such as "notify me of the weather forecast by email every day at 9:00 AM."

[0640] The input task content is transmitted to the server by the transmitting means, and the server has a receiving means for receiving the transmitted task content and analyzing the content.

[0641] Next, the server generates appropriate code and settings based on the received task content. The generation method uses a generation AI (e.g., a model using natural language processing). The server inputs the task content into the generation AI, selects the necessary APIs, and generates code that integrates them. For example, code is generated that retrieves data from a weather forecast API and uses an email sending API.

[0642] The generated code is returned to the terminal through the return means. The terminal displays the code received from the server or places it in an execution environment to automate the task. The user can check and implement the generated code.

[0643] As a concrete example, if a user inputs "Receive a weather forecast email every day at 9 AM," the system will operate as follows: First, the user's input is received and sent to the server. The server analyzes the natural language task content and generates code using appropriate APIs (for example, a weather forecast acquisition API and an email sending API). The generated code is then returned to the user's device. The user can then use the returned code to automate the desired task.

[0644] This system allows users to automate complex tasks through natural language input without requiring specialized programming knowledge, which is expected to improve work efficiency and save time.

[0645] The processing flow will be explained below.

[0646] Step 1:

[0647] The user uses the input method on the device to input the task they want to automate in natural language. For example, they might input "Notify me of the weather forecast by email every day at 9:00 AM."

[0648] Step 2:

[0649] The terminal transmits the input task content to the server using the transmission means. Specifically, the input text data in natural language is sent to the server in the form of an HTTP request.

[0650] Step 3:

[0651] The server receives the task content through the receiving means. The server-side API endpoint receives this data and prepares it for analysis.

[0652] Step 4:

[0653] The server passes data to a generation AI based on the received task content, and automatically generates appropriate code and settings. A natural language processing model is used as the generation method, and specific APIs (for example, a weather forecast API and an email sending API) are selected according to the user's request, and the code is constructed.

[0654] Step 5:

[0655] The code and settings generated by the generating means are returned to the terminal by the returning means. The server transmits the generated code to the terminal in the form of an HTTP response.

[0656] Step 6:

[0657] The device receives the returned code and displays it for the user to review. The user can then review the generated code, deploy it in their environment, and run it.

[0658] Step 7:

[0659] The user runs the generated code, for example, by setting up a scheduled task to retrieve the weather forecast every day at 9:00 AM and send an email notification.

[0660] Example 1

[0661] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0662] Conventional automation task configuration requires advanced programming knowledge, making it difficult for general users to use. Furthermore, when multiple APIs are used in combination, selecting and integrating them can be time-consuming, making the construction of automation tasks cumbersome. To solve these problems, there is a demand for a system that can easily configure automation tasks and automatically select and integrate multiple APIs.

[0663] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0664] In this invention, the server includes input means for a user to input a task they wish to automate in natural language, transmission means for transmitting the content of the automation task acquired from the input means to the server, reception means for receiving the content of the automation task transmitted by the transmission means, analysis means for analyzing the content of the automation task received by the reception means and selecting necessary APIs and their combinations, generation means for generating appropriate code and settings using a generative AI model based on the APIs and their combinations selected by the analysis means, return means for returning the code and settings generated by the generation means to the terminal, and execution means for providing an execution environment for executing the code and settings returned by the return means. This enables a user to set an automation task and automatically select and integrate multiple APIs through natural language input without requiring advanced programming knowledge.

[0665] An "input means" is a device or mechanism that allows a user to input the task they want to automate in natural language.

[0666] The "transmission means" is a device or mechanism for transmitting the content of the automation task acquired from the input means to the server.

[0667] The "receiving means" is a device or mechanism for receiving the content of the automation task transmitted by the transmitting means.

[0668] The "analysis means" is a device or mechanism for analyzing the content of the automation task received by the reception means and selecting the necessary APIs and their combinations.

[0669] The "generation means" refers to a device or mechanism for generating appropriate code and settings using a generative AI model based on the APIs and their combinations selected by the analysis means.

[0670] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to generate appropriate code for a given task.

[0671] The "returning means" is a device or mechanism for returning the code and settings generated by the generating means to the terminal.

[0672] The "execution means" is a device or mechanism that provides an environment for executing the code and settings returned by the return means.

[0673] "User interface" refers to the interface that a user operates directly on a terminal to input and send the details of an automated task.

[0674] "API" stands for Application Program Interface, a mechanism for sharing functions between multiple pieces of software.

[0675] The system of the present invention provides a means for a user to input a task to be automated in natural language, enabling the user to accomplish the task without specialized programming knowledge. Specifically, the process of generating appropriate code from the user's input and automating it is described in detail below.

[0676] Hardware and Software Configuration

[0677] The system includes the following major hardware and software components:

[0678] Device: A device used by a user, such as a computer, smartphone, or tablet.

[0679] Server: A remote computer system for data processing and code generation.

[0680] User Interface: The interface through which users input automation tasks. This can be a web application or a dedicated desktop application.

[0681] Generative AI model: An artificial intelligence model that uses natural language processing techniques to generate appropriate code from user input. For example, GPT-3 plays this role.

[0682] Data processing and calculation

[0683] 1. User task input:

[0684] Users input the task they want to automate in natural language into the user interface on their device, for example, "Receive a weather forecast email every day at 9 a.m."

[0685] 2. Submit your input:

[0686] The device sends the task details entered by the user to the server using an HTTP request, encrypted with SSL / TLS to ensure security.

[0687] 3. Task analysis on the server:

[0688] The server analyzes the received task content. A natural language processing (NLP) module is used for the analysis, and the necessary APIs and processing procedures are extracted from the text. For example, "spaCy" is used as an NLP module.

[0689] 4. Generate code:

[0690] The server analyzes the received task content, inputs prompts to the generative AI model, and generates code using appropriate APIs and their combinations. The generated code is set up to automate the user's task.

[0691] 5. Returning the code:

[0692] The generated code is returned from the server to the device using an HTTP response, with the response data sent in JSON format.

[0693] 6. Code Execution and Automation:

[0694] The terminal can display the returned code to the user or place it in an execution environment to automate tasks. After the user confirms it, the code will be automatically executed at the set time to automate the task.

[0695] Specific examples

[0696] For example, if a user enters "Receive a weather forecast email every day at 9:00 AM," the system will work as follows:

[0697] 1. The user enters the task details into the corresponding input form and clicks the submit button.

[0698] 2. The terminal sends this input to the server.

[0699] 3. The server analyzes the input and determines that a weather forecast API and an email sending API are required.

[0700] 4. Enter the following prompts into the generative AI model to generate the appropriate code:

[0701] Generate a code that will get the weather forecast every day at 9am and notify a specified email address. Integrate your preferred API.

[0702] 5. The generated code is returned from the server to the device.

[0703] 6. The device will either display the returned code or run it automatically at a set time, automating the task of emailing the weather forecast every day at 9:00 AM.

[0704] As described above, users can easily set up and execute automation tasks without advanced programming knowledge.

[0705] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0706] Step 1:

[0707] The user inputs the task they want to automate into the device's user interface in natural language. For example, the user might input "Notify me of the weather forecast by email every day at 9:00 AM." This input is passed to the device as input data.

[0708] Specific behavior:

[0709] The user opens a browser, accesses the specified URL, enters the task details in the displayed input form, and clicks the submit button. The input data is saved in the device's memory, and the device proceeds to the next step.

[0710] Step 2:

[0711] The device acquires the input task content and sends it to the server. The task content as input data is converted to JSON format and sent via an HTTP POST request. This transmission passes the task content to the server.

[0712] Specific behavior:

[0713] The terminal converts the input content into JSON format and sends it to "https: / / example.com / api / submitTask" using a POST request. At this time, SSL / TLS encryption is used.

[0714] Step 3:

[0715] The server receives the task content sent. It receives the task content as received data and prepares it for analysis.

[0716] Specific behavior:

[0717] The server receives the HTTP POST request and extracts the submitted JSON data for parsing, which is stored in memory and passed to the next parsing step.

[0718] Step 4:

[0719] The server analyzes the received task content. Using a natural language processing (NLP) module, it extracts the necessary APIs and their combinations from the task content. It analyzes the task content as input data and generates API selection data.

[0720] Specific behavior:

[0721] The server runs an NLP module (e.g., "spaCy") and extracts the "weather forecast API" and "email sending API" from the task content. The analysis results are stored in a database and passed to the next step.

[0722] Step 5:

[0723] The server inputs a prompt sentence into a generative AI model (e.g., GPT-3) based on the analysis results, and generates appropriate code. The API selection data is used as input data, and the generated code is obtained as output data.

[0724] Specific behavior:

[0725] The server inputs the following prompt to the generated AI model:

[0726] Generate a code that will get the weather forecast every day at 9am and notify a specified email address. Integrate your preferred API.

[0727] The code returned by the generative AI model is stored in memory and passed to the next step.

[0728] Step 6:

[0729] The server returns the generated code to the device. The generated output data is converted to JSON format and sent to the device as an HTTP response. This sends the code to the device.

[0730] Specific behavior:

[0731] The server converts the generated code into JSON format and sends it to the device as an HTTP response, using information stored in a directory or database to send it to the appropriate endpoint.

[0732] Step 7:

[0733] The device receives the returned code, displays it to the user, and, if necessary, places it in the execution environment to automate tasks. The code is then provided to the user and the execution environment as output data.

[0734] Specific behavior:

[0735] The terminal will display the returned code in an editor for the user to confirm, and after the user confirms, they can register the code as an execution script in a scheduler (for example, a cron job) and set it to run every day at 9:00 AM.

[0736] Through the above steps, users can set up and execute automation tasks through natural language input without any programming knowledge.

[0737] (Application example 1)

[0738] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0739] Operating and maintaining conventional factory robots requires advanced specialized knowledge and programming skills, and often takes a lot of time and effort. Automating tasks in particular requires specialized programming, which has been an obstacle to improving productivity and work efficiency.

[0740] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0741] In this invention, the server includes an input means for allowing a user to input a task they wish to automate in natural language, a transmission means for transmitting the content of the automation task acquired from the input means to the server, a receiving means for receiving the content of the automation task transmitted by the transmission means, a generation means for generating appropriate code and settings based on the content of the automation task received by the receiving means, a generative AI model used in the generation means and including a function for automatically selecting an API and generating code based on a prompt statement, and a return means for returning the code and settings generated by the generation means to the terminal. This makes it possible to easily generate code for automating the operation and maintenance of factory robots based on the task input by the user in natural language, thereby enabling efficient task automation without specialized knowledge.

[0742] A "user" is a person or business entity that utilizes the system to configure and operate automated tasks.

[0743] An "input means" is a device or software that provides an interface for a user to input an automated task in natural language.

[0744] The "transmission means" is a communication device or software for transmitting the content of the automation task acquired from the input means to the server.

[0745] The "receiving means" is a communication device or software for receiving the content of the automated task transmitted by the transmitting means at the server.

[0746] "Generating means" means a device or software that generates appropriate code and settings based on the content of the automation task received by the receiving means, including a generative AI model.

[0747] A "generative AI model" is an artificial intelligence model that uses natural language processing to analyze the user's task content and generate the necessary code and settings.

[0748] "API" stands for Application Programming Interface, an interface used for communication and data exchange between different software programs.

[0749] A "prompt sentence" is a sentence that represents the task content and is input to the generative AI model, and is used as a trigger for code generation.

[0750] The "returning means" is a communication device or software for returning the code and settings generated by the generating means to the user's terminal.

[0751] A "terminal" is an electronic device such as a computer or smartphone that is directly operated by a user.

[0752] This invention relates to a system for automating the operation and maintenance tasks of factory robots. The system allows a user to input a task in natural language, and generates appropriate code and settings based on the input, achieving automation.

[0753] The system uses the following hardware and software: The hardware uses terminals such as factory robot control panels and smartphones, while the software uses OpenAI's GPT-3 model, an internet connection for communication, and the Python programming language.

[0754] System flow

[0755] 1. User interface input

[0756] Users use the input interface on their devices to input tasks they want to automate in natural language, such as "Check the robot's status every day at 3:00 PM and notify me if there is an error."

[0757] 2. Sending task details

[0758] The terminal transmits the content of the automation task acquired from the input means to the server using the transmission means.

[0759] 3. Receiving and analyzing task content

[0760] The server receives the task content sent by the sending means using the receiving means and analyzes it. The receiving means has a function for interpreting data in JSON format or text format.

[0761] 4. Code Generation

[0762] Based on the received task content, a generative AI model (GPT-3) is used as a generator to generate appropriate code and settings. This generator includes the ability to automatically select an API and generate code using prompt statements.

[0763] 5. Returning the Code

[0764] The generated code and settings are returned to the terminal using the return means, and the user can check the generated code and execute it as appropriate to automate the task.

[0765] Specific examples

[0766] Specific task input example:

[0767] The user inputs "restart all systems of the robot every Monday, and then send the logs of each module." This information is sent to the server and analyzed. The server uses a generative AI model to generate the code necessary for the two tasks of "restarting the robot" and "sending the logs," and returns it.

[0768] Example prompt sentence:

[0769] Task: Restart all systems of the robot every Monday and then send the logs of each module.

[0770] code:

[0771] In this way, the operation and maintenance of factory robots can be efficiently automated based on tasks entered by the user in natural language, making it possible for even users without specialized programming knowledge to easily achieve advanced automation.

[0772] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0773] Step 1:

[0774] The user uses the input interface on the terminal to input the task to be automated in natural language.

[0775] Input: A natural language task description entered by the user (e.g., "Check the robot's status every day at 3 PM and notify me if there is an error").

[0776] Output: The task content is displayed in the input interface.

[0777] Specific actions: The user types or speaks the task details into the input screen of their smartphone or computer.

[0778] Step 2:

[0779] The terminal transmits the task content acquired using the input means to the server via the transmission means.

[0780] Input: Task content obtained from the input interface

[0781] Output: The data format in which the task content is sent to the server (e.g., JSON or text format)

[0782] Specific operation: When you click the "Send" button on the device, the task content is sent to the server via the Internet.

[0783] Step 3:

[0784] The server receives the task contents transmitted from the transmitting means using the receiving means.

[0785] Input: Data of the submitted task

[0786] Output: The received task data is saved on the server.

[0787] Specific operation: The server's receiving module records the task details in the database.

[0788] Step 4:

[0789] The server analyzes the received task content and generates the appropriate code and configuration using a generative AI model.

[0790] Input: Received task content data

[0791] Output: Generated code and configuration (e.g. Python scripts, API calls)

[0792] Specific operation: A generative AI model (such as GPT-3) analyzes the task content and generates appropriate APIs and code based on the prompt text.

[0793] Step 5:

[0794] The server returns the code and settings generated by the generating means to the terminal using the returning means.

[0795] Input: Generated code and settings

[0796] Output: Code and configuration data returned to the device

[0797] What it does: The server converts the code and settings into packets and sends them over the Internet to the device.

[0798] Step 6:

[0799] The device receives the code and settings returned from the server and executes it as an automated task.

[0800] Input: Code and settings returned from the server

[0801] Output: Executed automation task (e.g. robot status check, notification execution)

[0802] What happens: A script is launched to execute code on the device, performing robot operations and notification tasks.

[0803] Through these steps, the operation and maintenance of factory robots are automated based on tasks entered by the user in natural language, making it possible to automate tasks efficiently even without specialized knowledge.

[0804] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0805] The system of the present invention effectively realizes automated tasks by combining the user's input of automated tasks in natural language with an emotion engine that recognizes and analyzes the user's emotions.

[0806] This system provides an input method on the terminal for users to input the tasks they want to automate in natural language. Using this interface, users can input the tasks they want to automate. For example, they can input "Notify me of the weather forecast by email every day at 9:00 AM."

[0807] The input task content is transmitted to the server by the transmitting means. The server receives the transmitted task content through the receiving means. At this time, the server uses an emotion engine to analyze emotions from the user's input content. The emotion engine can recognize the user's emotions (e.g., joy, sadness, anger) through natural language analysis or voice input analysis.

[0808] The server then uses a generator to generate appropriate code and settings based on the recognized emotion. For example, if the user is dissatisfied with the input, the server can adjust the notification method to provide more detailed information or additional feedback. The generator also automatically selects multiple public APIs and generates code and settings to effectively accomplish the task.

[0809] The generated code and settings are returned to the terminal through the return means. The terminal displays the code and settings received from the server so that the user can easily check them. The user can then run the generated code and settings in a local environment or a cloud environment.

[0810] For example, if a user inputs "Notify me of the weather forecast by email every day at 9 AM" and simultaneously recognizes that the user's emotion is excitement, the system can be implemented to notify the user not only of the weather forecast information for that specific snippet, but also of recommended tourist spots and outdoor events. This ensures that the results of the automated task match the user's emotion, providing a more satisfying experience.

[0811] Therefore, the system of the present invention allows users without advanced programming skills to easily accomplish complex automated tasks through natural language input and emotion recognition, which is expected to improve work efficiency and user experience.

[0812] The processing flow will be explained below.

[0813] Step 1:

[0814] The user uses the input method on the device to input the task they want to automate in natural language. For example, they might input "Notify me of the weather forecast by email every day at 9:00 AM."

[0815] Step 2:

[0816] The terminal transmits the input task content to the server using the transmission means. Specifically, the input text data in natural language is sent to the server in the form of an HTTP request.

[0817] Step 3:

[0818] The server receives the task content through the receiving means. The server-side API endpoint receives this data and prepares it for analysis.

[0819] Step 4:

[0820] The server passes the received task content to the emotion engine, which uses natural language analysis or voice input analysis to recognize the user's emotions. For example, it analyzes positive emotions (joy, excitement) and negative emotions (sadness, dissatisfaction).

[0821] Step 5:

[0822] The server uses a generation method to generate appropriate code and settings based on the recognized emotion. The generation method uses a natural language processing model to select APIs (e.g., weather forecast APIs and email sending APIs) while taking the user's emotion into consideration, and configures the optimal code.

[0823] Step 6:

[0824] The generated code and settings are returned to the terminal through the return means. The server transmits the generated code to the terminal in the form of an HTTP response.

[0825] Step 7:

[0826] The device receives the returned code and displays it for the user to review. The user can then review the generated code, deploy it in their environment, and run it.

[0827] Step 8:

[0828] The user runs the generated code, for example, by setting up a scheduled task to retrieve the weather forecast every day at 9:00 AM and send an email notification.

[0829] Example 2

[0830] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0831] Currently, automating tasks requires programming knowledge and skills, which is a significant hurdle for average users. Furthermore, performing tasks without considering the user's emotions can lead to unsatisfied users and result in poor results. Therefore, there is a need for a system that combines natural language task input with emotion recognition to improve user satisfaction and automate complex tasks.

[0832] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an input means for inputting a task that a user wants to automate in natural language, a transmission means for transmitting the content of the automation task acquired from the input means to the server, a reception means for receiving the content of the automation task transmitted by the transmission means, a sentiment analysis means for analyzing the user's sentiment based on the content of the automation task received by the reception means, a generation means for generating appropriate code and settings based on the sentiment recognized by the sentiment analysis means, and a return means for returning the code and settings generated by the generation means to the terminal. This enables a user without advanced programming skills to easily realize complex automation tasks through input in natural language and emotion recognition.

[0833] Understood. Below are definitions of key terms included in the rewritten claims.

[0834] A "user" is an individual or organization that utilizes the system to configure automated tasks.

[0835] A "task" is a series of operations or processes that a user wants to automate.

[0836] An "input means" is a device or software that includes an interface for a user to input a task in natural language.

[0837] The "transmission means" is a communication means for transmitting the task contents acquired from the input means to the server.

[0838] The "server" is a central processing unit that receives, analyzes, and processes task contents sent by users.

[0839] The "receiving means" is a means by which the server receives the task content transmitted by the transmitting means.

[0840] "Emotion analysis means" refers to software and algorithms that allow the server to analyze emotions from the user's input.

[0841] The "generating means" refers to a processing device and program for generating appropriate code and settings based on the emotion recognized by the emotion analyzing means.

[0842] The "returning means" is a communication means for returning the code and settings generated by the generating means from the server to the terminal.

[0843] "Terminal" means a device for a user to use input means and is used to display the code and settings returned from the server.

[0844] The system of the present invention allows a user to input an automation task using natural language, recognizes the user's emotions, and generates appropriate code and settings. This system is configured by combining an input unit, a transmission unit, a server, a reception unit, an emotion analysis unit, a generation unit, a return unit, and a terminal.

[0845] First, the user uses the input method on the device to input the task they want to automate in natural language. The input method can be a text field or a voice input interface. For example, they can input a specific task such as "Send me a weather forecast email every day at 9:00 AM."

[0846] The input task content is sent to the server by the sending means. A common communication protocol such as an HTTP request is used as the communication means. Next, the server receives the input task content through the receiving means. On the server side, the user's emotions are analyzed using emotion analysis means based on the received task content. The emotion analysis means used here may be a natural language analysis model (e.g., BERT or GPT-3).

[0847] The emotion analysis means recognizes emotions (e.g., joy, sadness, anger) from the user's input. The server then uses the generation means to automatically generate appropriate codes and settings based on the recognized emotions. For example, if the user feels "enjoyed" by the input, the generation means generates codes and settings to notify the user of recommended tourist spots in addition to the weather forecast.

[0848] The generating means automatically selects multiple published APIs and generates code for effectively performing the task. The generated code and settings are returned to the terminal through the returning means. The returning means may be an HTTP response or the like.

[0849] The device displays the code and configuration returned by the server to the user, who can review it and make any necessary corrections or approvals. The user can then run the completed code and configuration in a local or cloud environment. This allows users without advanced programming skills to easily accomplish complex automation tasks through natural language input and emotion recognition.

[0850] For example, if a user inputs "notify me of the weather forecast by email every day at 9:00 AM," and sentiment analysis identifies this as "excitement," the system will configure the system to notify users of information about recommended tourist spots and outdoor events in addition to the weather forecast. This allows users to achieve high satisfaction with the automated notifications.

[0851] An example of a prompt sentence might be:

[0852] "Every day at 9:00 AM, I'll email you the latest weather forecast and the best places to visit that day."

[0853] This allows users to intuitively and efficiently configure and execute automated tasks.

[0854] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0855] Step 1: User enters task

[0856] The user uses the input method on the device to input the task they want to automate in natural language. This interface includes a text field and voice input support. For example, the user might input "Receive a weather forecast email every day at 9:00 AM." The input is captured as text data.

[0857] Input: User's natural language task input (e.g., "Email me the weather forecast every day at 9 AM").

[0858] Output: Task content as text data.

[0859] Step 2: The device sends the task details to the server

[0860] The terminal transmits the acquired task content to the server via a transmission means. The transmission is performed using a communication protocol such as an HTTP request. Specifically, the terminal sends an HTTP "POST" request to the server, and includes the task content in the body of the request.

[0861] Input: Task content as text data.

[0862] Output: HTTP request to the server.

[0863] Step 3: The server receives the task

[0864] The server receives the task content sent through the receiving means. Specifically, the server analyzes the HTTP request and extracts the task content from the body. The received task content is stored as an internal data structure.

[0865] Input: HTTP request.

[0866] Output: Task details (text data) within the server.

[0867] Step 4: The server analyzes the emotions

[0868] The server uses a sentiment analysis method to analyze the sentiment of the user's input. A natural language analysis model such as BERT or GPT-3 can be used for this analysis. Specifically, the server passes the input text to a sentiment analysis API and obtains a sentiment label (e.g., "enjoyed").

[0869] Input: Task contents (text data) within the server.

[0870] Output: An emotion label (e.g., "enjoyment").

[0871] Step 5: The server generates the appropriate code and configuration

[0872] The server uses a generation method to generate appropriate code and settings based on the recognized emotion. Specifically, the server selects the appropriate API based on the emotion analysis results and automatically generates code to accomplish the task. For example, if the emotion is "excitement," it generates code that combines a weather forecast API and a tourist destination information API.

[0873] Input: emotion label, task content.

[0874] Output: Auto-generated code and configuration.

[0875] Step 6: The server returns the generated code

[0876] The generated code and settings are returned to the terminal through the return means. Specifically, the server sends the generated code to the terminal as an HTTP "POST" response.

[0877] Inputs: Auto-generated code and settings.

[0878] Output: HTTP response to the device.

[0879] Step 7: The terminal will display the code

[0880] The terminal displays the received code and settings to the user. Specifically, the terminal visually displays the received code on the interface so that the user can confirm the contents.

[0881] Input: The HTTP response from the server.

[0882] Output: The code and settings displayed on the interface.

[0883] Through the specific actions performed at each step, users can easily accomplish complex automation tasks through natural language input and emotion recognition, even without advanced programming skills.

[0884] (Application example 2)

[0885] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0886] Conventional automation systems could automatically process the content of tasks specified by the user, but they were unable to respond flexibly while taking into account the user's emotions and circumstances. This limited the improvement of user satisfaction and work efficiency. Furthermore, in production sites such as factories, there was a need for support to efficiently complete tasks while reducing worker stress and fatigue. A system that solves these issues and realizes task automation while taking into consideration the emotions of the user or worker is needed.

[0887] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means for a user to input a task they wish to automate in natural language, a generation means for generating appropriate code and settings based on the content of the automation task and the emotion analysis results acquired from the input means, and a return means for returning the code and settings generated by the generation means to the terminal. This enables task automation that takes the user's emotions into consideration, realizes more flexible and satisfying work support, and enables factory workers to efficiently complete tasks while reducing stress and fatigue.

[0888] An "input means" is a device or software that provides an interface for a user to input, in natural language, the task they wish to automate.

[0889] The "transmission means" is a device or software having a function for transmitting the content of the automation task acquired from the input means to the server via the network.

[0890] The "receiving means" is a device or software having a function for receiving the content of the automated task transmitted to the server by the transmitting means.

[0891] The "emotion analysis result" is emotional information analyzed from the user's input content, and is data generated by the emotion analysis engine.

[0892] The "generating means" is a device or software for generating appropriate code and settings based on the content of the automated task and the emotion analysis results received by the receiving means.

[0893] The "returning means" is a device or software having a function for returning the code and settings generated by the generating means to the terminal.

[0894] The term "user interface" refers to a screen or operating means that allows a user to input information in natural language on a terminal.

[0895] An "emotion analysis engine" is an algorithm and software that analyzes voice and text input from users and recognizes their emotions.

[0896] The embodiments of the present invention will be described in detail below.

[0897] First, a user uses a terminal to input the task to be automated in natural language. A user interface is provided on the terminal for the user to input instructions in natural language. This user interface is an interface that accepts natural language tasks through voice input or text input. For example, the user can input an instruction such as "move the next pallet to the conveyor belt."

[0898] Next, the input task content is transmitted from the terminal to the server, and this transmission means has a communication function for transmitting data to the server via a network.

[0899] The server analyzes the received task content and recognizes the user's emotions through a sentiment analysis engine. The sentiment analysis engine analyzes the text or voice data from the user's input to generate emotional data. For example, text analysis can determine whether the user is tired or happy.

[0900] The server then uses a generator to generate appropriate code and settings based on the received task content and sentiment analysis results. The generator automatically selects and combines a number of publicly available APIs to generate code to accomplish the required task.

[0901] The generated code and settings are returned from the server to the terminal. The return means has a communication function for transmitting the generated code and settings to the terminal via a network. The user can check the returned code and settings and make corrections or approvals if necessary.

[0902] For example, if a user inputs "move the next pallet to the conveyor belt" and the emotion analysis engine recognizes the user's emotion as "tired," the system can adjust to temporarily suspend the work, suggest a break, and notify the user when it's time to resume. In this way, it becomes possible to support efficient work while taking into consideration the worker's emotions.

[0903] Examples of prompts include:

[0904] "Move the next pallet to the conveyor belt."

[0905] "Text: 'Move the next pallet to the conveyor belt.'; Sentiment: 'Neutral'"

[0906] This system makes it possible to automate tasks while taking emotions into account, which is expected to improve user satisfaction and work efficiency.

[0907] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0908] Step 1:

[0909] The user uses the user interface on the device to input the task they want to automate in natural language. This input is captured by the input means and stored as digital data on the device. An example of this input might be, "Move the next pallet to the conveyor belt."

[0910] Step 2:

[0911] The input natural language task is sent to the server via the transmission means over the network. The transmission means packetizes the digital data stored on the terminal and converts it into a format that the server can receive. The output of this process is digital data containing the task content on the server side.

[0912] Step 3:

[0913] The server receives the transmitted task content using the receiving means. The received data includes the task content in natural language entered by the user. The server passes this data to the analysis engine and starts natural language analysis.

[0914] Step 4:

[0915] The server uses a natural language analysis engine to analyze the content of the input task, and also recognizes emotions through a sentiment analysis engine. The input is text data of the task content, and the sentiment analysis engine analyzes the text for positive, negative, and neutral aspects. For example, the input "Move the next pallet to the conveyor belt" is analyzed as "neutral."

[0916] Step 5:

[0917] The server generates appropriate code and settings based on the analyzed task content and emotion analysis results. The server uses a generation method to automatically select multiple APIs and dynamically generate the necessary code. For example, if the recognized emotion is "tired," it generates code that includes functions to pause and suggest breaks. The input of this step is the task content and emotion analysis results, and the output is the generated code and settings.

[0918] Step 6:

[0919] The generated code and settings are sent from the server to the terminal, again using the return means. The server packetizes the generated data over the network and converts it into a format acceptable to the terminal. The output of this process is the generated code and settings data that is transmitted to the user's terminal.

[0920] Step 7:

[0921] The device begins operation based on the received code and settings. The user can check the progress of the task and make corrections or approvals if necessary. For example, this can include time management during breaks and preparation for the next task. User feedback is input back into the system and can be reflected in subsequent task executions.

[0922] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0923] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0924] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0925] [Fourth embodiment]

[0926] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0927] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0928] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0929] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0930] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0931] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0932] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0933] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0934] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0935] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0936] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0937] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0938] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0939] The system of the present invention provides a means for generating and automating appropriate code based on a user's natural language input, making it easy to accomplish automation tasks.

[0940] The system provides an input method on the terminal for users to input the tasks they want to automate in natural language. Using this interface, users can input the details of the tasks they want to automate. For example, they can input a task such as "notify me of the weather forecast by email every day at 9:00 AM."

[0941] The input task content is transmitted to the server by the transmitting means, and the server has a receiving means for receiving the transmitted task content and analyzing the content.

[0942] Next, the server generates appropriate code and settings based on the received task content. The generation method uses a generation AI (e.g., a model using natural language processing). The server inputs the task content into the generation AI, selects the necessary APIs, and generates code that integrates them. For example, code is generated that retrieves data from a weather forecast API and uses an email sending API.

[0943] The generated code is returned to the terminal through the return means. The terminal displays the code received from the server or places it in an execution environment to automate the task. The user can check and implement the generated code.

[0944] As a concrete example, if a user inputs "Receive a weather forecast email every day at 9 AM," the system will operate as follows: First, the user's input is received and sent to the server. The server analyzes the natural language task content and generates code using appropriate APIs (for example, a weather forecast acquisition API and an email sending API). The generated code is then returned to the user's device. The user can then use the returned code to automate the desired task.

[0945] This system allows users to automate complex tasks through natural language input without requiring specialized programming knowledge, which is expected to improve work efficiency and save time.

[0946] The processing flow will be explained below.

[0947] Step 1:

[0948] The user uses the input method on the device to input the task they want to automate in natural language. For example, they might input "Notify me of the weather forecast by email every day at 9:00 AM."

[0949] Step 2:

[0950] The terminal transmits the input task content to the server using the transmission means. Specifically, the input text data in natural language is sent to the server in the form of an HTTP request.

[0951] Step 3:

[0952] The server receives the task content through the receiving means. The server-side API endpoint receives this data and prepares it for analysis.

[0953] Step 4:

[0954] The server passes data to a generation AI based on the received task content, and automatically generates appropriate code and settings. A natural language processing model is used as the generation method, and specific APIs (for example, a weather forecast API and an email sending API) are selected according to the user's request, and the code is constructed.

[0955] Step 5:

[0956] The code and settings generated by the generating means are returned to the terminal by the returning means. The server transmits the generated code to the terminal in the form of an HTTP response.

[0957] Step 6:

[0958] The device receives the returned code and displays it for the user to review. The user can then review the generated code, deploy it in their environment, and run it.

[0959] Step 7:

[0960] The user runs the generated code, for example, by setting up a scheduled task to retrieve the weather forecast every day at 9:00 AM and send an email notification.

[0961] Example 1

[0962] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0963] Conventional automation task configuration requires advanced programming knowledge, making it difficult for general users to use. Furthermore, when multiple APIs are used in combination, selecting and integrating them can be time-consuming, making the construction of automation tasks cumbersome. To solve these problems, there is a demand for a system that can easily configure automation tasks and automatically select and integrate multiple APIs.

[0964] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0965] In this invention, the server includes input means for a user to input a task they wish to automate in natural language, transmission means for transmitting the content of the automation task acquired from the input means to the server, reception means for receiving the content of the automation task transmitted by the transmission means, analysis means for analyzing the content of the automation task received by the reception means and selecting necessary APIs and their combinations, generation means for generating appropriate code and settings using a generative AI model based on the APIs and their combinations selected by the analysis means, return means for returning the code and settings generated by the generation means to the terminal, and execution means for providing an execution environment for executing the code and settings returned by the return means. This enables a user to set an automation task and automatically select and integrate multiple APIs through natural language input without requiring advanced programming knowledge.

[0966] An "input means" is a device or mechanism that allows a user to input the task they want to automate in natural language.

[0967] The "transmission means" is a device or mechanism for transmitting the content of the automation task acquired from the input means to the server.

[0968] The "receiving means" is a device or mechanism for receiving the content of the automation task transmitted by the transmitting means.

[0969] The "analysis means" is a device or mechanism for analyzing the content of the automation task received by the reception means and selecting the necessary APIs and their combinations.

[0970] The "generation means" refers to a device or mechanism for generating appropriate code and settings using a generative AI model based on the APIs and their combinations selected by the analysis means.

[0971] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to generate appropriate code for a given task.

[0972] The "returning means" is a device or mechanism for returning the code and settings generated by the generating means to the terminal.

[0973] The "execution means" is a device or mechanism that provides an environment for executing the code and settings returned by the return means.

[0974] "User interface" refers to the interface that a user operates directly on a terminal to input and send the details of an automated task.

[0975] "API" stands for Application Program Interface, a mechanism for sharing functions between multiple pieces of software.

[0976] The system of the present invention provides a means for a user to input a task to be automated in natural language, enabling the user to accomplish the task without specialized programming knowledge. Specifically, the process of generating appropriate code from the user's input and automating it is described in detail below.

[0977] Hardware and Software Configuration

[0978] The system includes the following major hardware and software components:

[0979] Device: A device used by a user, such as a computer, smartphone, or tablet.

[0980] Server: A remote computer system for data processing and code generation.

[0981] User Interface: The interface through which users input automation tasks. This can be a web application or a dedicated desktop application.

[0982] Generative AI model: An artificial intelligence model that uses natural language processing techniques to generate appropriate code from user input. For example, GPT-3 plays this role.

[0983] Data processing and calculation

[0984] 1. User task input:

[0985] Users input the task they want to automate in natural language into the user interface on their device, for example, "Receive a weather forecast email every day at 9 a.m."

[0986] 2. Submit your input:

[0987] The device sends the task details entered by the user to the server using an HTTP request, encrypted with SSL / TLS to ensure security.

[0988] 3. Task analysis on the server:

[0989] The server analyzes the received task content. A natural language processing (NLP) module is used for the analysis, and the necessary APIs and processing procedures are extracted from the text. For example, "spaCy" is used as an NLP module.

[0990] 4. Generate code:

[0991] The server analyzes the received task content, inputs prompts to the generative AI model, and generates code using appropriate APIs and their combinations. The generated code is set up to automate the user's task.

[0992] 5. Returning the code:

[0993] The generated code is returned from the server to the device using an HTTP response, with the response data sent in JSON format.

[0994] 6. Code Execution and Automation:

[0995] The terminal can display the returned code to the user or place it in an execution environment to automate tasks. After the user confirms it, the code will be automatically executed at the set time to automate the task.

[0996] Specific examples

[0997] For example, if a user enters "Receive a weather forecast email every day at 9:00 AM," the system will work as follows:

[0998] 1. The user enters the task details into the corresponding input form and clicks the submit button.

[0999] 2. The terminal sends this input to the server.

[1000] 3. The server analyzes the input and determines that a weather forecast API and an email sending API are required.

[1001] 4. Enter the following prompts into the generative AI model to generate the appropriate code:

[1002] Generate a code that will get the weather forecast every day at 9am and notify a specified email address. Integrate your preferred API.

[1003] 5. The generated code is returned from the server to the device.

[1004] 6. The device will either display the returned code or run it automatically at a set time, automating the task of emailing the weather forecast every day at 9:00 AM.

[1005] As described above, users can easily set up and execute automation tasks without advanced programming knowledge.

[1006] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1007] Step 1:

[1008] The user inputs the task they want to automate into the device's user interface in natural language. For example, the user might input "Notify me of the weather forecast by email every day at 9:00 AM." This input is passed to the device as input data.

[1009] Specific behavior:

[1010] The user opens a browser, accesses the specified URL, enters the task details in the displayed input form, and clicks the submit button. The input data is saved in the device's memory, and the device proceeds to the next step.

[1011] Step 2:

[1012] The device acquires the input task content and sends it to the server. The task content as input data is converted to JSON format and sent via an HTTP POST request. This transmission passes the task content to the server.

[1013] Specific behavior:

[1014] The terminal converts the input content into JSON format and sends it to "https: / / example.com / api / submitTask" using a POST request. At this time, SSL / TLS encryption is used.

[1015] Step 3:

[1016] The server receives the task content sent. It receives the task content as received data and prepares it for analysis.

[1017] Specific behavior:

[1018] The server receives the HTTP POST request and extracts the submitted JSON data for parsing, which is stored in memory and passed to the next parsing step.

[1019] Step 4:

[1020] The server analyzes the received task content. Using a natural language processing (NLP) module, it extracts the necessary APIs and their combinations from the task content. It analyzes the task content as input data and generates API selection data.

[1021] Specific behavior:

[1022] The server runs an NLP module (e.g., "spaCy") and extracts the "weather forecast API" and "email sending API" from the task content. The analysis results are stored in a database and passed to the next step.

[1023] Step 5:

[1024] The server inputs a prompt sentence into a generative AI model (e.g., GPT-3) based on the analysis results, and generates appropriate code. The API selection data is used as input data, and the generated code is obtained as output data.

[1025] Specific behavior:

[1026] The server inputs the following prompt to the generated AI model:

[1027] Generate a code that will get the weather forecast every day at 9am and notify a specified email address. Integrate your preferred API.

[1028] The code returned by the generative AI model is stored in memory and passed to the next step.

[1029] Step 6:

[1030] The server returns the generated code to the device. The generated output data is converted to JSON format and sent to the device as an HTTP response. This sends the code to the device.

[1031] Specific behavior:

[1032] The server converts the generated code into JSON format and sends it to the device as an HTTP response, using information stored in a directory or database to send it to the appropriate endpoint.

[1033] Step 7:

[1034] The device receives the returned code, displays it to the user, and, if necessary, places it in the execution environment to automate tasks. The code is then provided to the user and the execution environment as output data.

[1035] Specific behavior:

[1036] The terminal will display the returned code in an editor for the user to confirm, and after the user confirms, they can register the code as an execution script in a scheduler (for example, a cron job) and set it to run every day at 9:00 AM.

[1037] Through the above steps, users can set up and execute automation tasks through natural language input without any programming knowledge.

[1038] (Application example 1)

[1039] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1040] Operating and maintaining conventional factory robots requires advanced specialized knowledge and programming skills, and often takes a lot of time and effort. Automating tasks in particular requires specialized programming, which has been an obstacle to improving productivity and work efficiency.

[1041] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1042] In this invention, the server includes an input means for allowing a user to input a task they wish to automate in natural language, a transmission means for transmitting the content of the automation task acquired from the input means to the server, a receiving means for receiving the content of the automation task transmitted by the transmission means, a generation means for generating appropriate code and settings based on the content of the automation task received by the receiving means, a generative AI model used in the generation means and including a function for automatically selecting an API and generating code based on a prompt statement, and a return means for returning the code and settings generated by the generation means to the terminal. This makes it possible to easily generate code for automating the operation and maintenance of factory robots based on the task input by the user in natural language, thereby enabling efficient task automation without specialized knowledge.

[1043] A "user" is a person or business entity that utilizes the system to configure and operate automated tasks.

[1044] An "input means" is a device or software that provides an interface for a user to input an automated task in natural language.

[1045] The "transmission means" is a communication device or software for transmitting the content of the automation task acquired from the input means to the server.

[1046] The "receiving means" is a communication device or software for receiving the content of the automated task transmitted by the transmitting means at the server.

[1047] "Generating means" means a device or software that generates appropriate code and settings based on the content of the automation task received by the receiving means, including a generative AI model.

[1048] A "generative AI model" is an artificial intelligence model that uses natural language processing to analyze the user's task content and generate the necessary code and settings.

[1049] "API" stands for Application Programming Interface, an interface used for communication and data exchange between different software programs.

[1050] A "prompt sentence" is a sentence that represents the task content and is input to the generative AI model, and is used as a trigger for code generation.

[1051] The "returning means" is a communication device or software for returning the code and settings generated by the generating means to the user's terminal.

[1052] A "terminal" is an electronic device such as a computer or smartphone that is directly operated by a user.

[1053] This invention relates to a system for automating the operation and maintenance tasks of factory robots. The system allows a user to input a task in natural language, and generates appropriate code and settings based on the input, achieving automation.

[1054] The system uses the following hardware and software: The hardware uses terminals such as factory robot control panels and smartphones, while the software uses OpenAI's GPT-3 model, an internet connection for communication, and the Python programming language.

[1055] System flow

[1056] 1. User interface input

[1057] Users use the input interface on their devices to input tasks they want to automate in natural language, such as "Check the robot's status every day at 3:00 PM and notify me if there is an error."

[1058] 2. Sending task details

[1059] The terminal transmits the content of the automation task acquired from the input means to the server using the transmission means.

[1060] 3. Receiving and analyzing task content

[1061] The server receives the task content sent by the sending means using the receiving means and analyzes it. The receiving means has a function for interpreting data in JSON format or text format.

[1062] 4. Code Generation

[1063] Based on the received task content, a generative AI model (GPT-3) is used as a generator to generate appropriate code and settings. This generator includes the ability to automatically select an API and generate code using prompt statements.

[1064] 5. Returning the Code

[1065] The generated code and settings are returned to the terminal using the return means, and the user can check the generated code and execute it as appropriate to automate the task.

[1066] Specific examples

[1067] Specific task input example:

[1068] The user inputs "restart all systems of the robot every Monday, and then send the logs of each module." This information is sent to the server and analyzed. The server uses a generative AI model to generate the code necessary for the two tasks of "restarting the robot" and "sending the logs," and returns it.

[1069] Example prompt sentence:

[1070] Task: Restart all systems of the robot every Monday and then send the logs of each module.

[1071] code:

[1072] In this way, the operation and maintenance of factory robots can be efficiently automated based on tasks entered by the user in natural language, making it possible for even users without specialized programming knowledge to easily achieve advanced automation.

[1073] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1074] Step 1:

[1075] The user uses the input interface on the terminal to input the task to be automated in natural language.

[1076] Input: A natural language task description entered by the user (e.g., "Check the robot's status every day at 3 PM and notify me if there is an error").

[1077] Output: The task content is displayed in the input interface.

[1078] Specific actions: The user types or speaks the task details into the input screen of their smartphone or computer.

[1079] Step 2:

[1080] The terminal transmits the task content acquired using the input means to the server via the transmission means.

[1081] Input: Task content obtained from the input interface

[1082] Output: The data format in which the task content is sent to the server (e.g., JSON or text format)

[1083] Specific operation: When you click the "Send" button on the device, the task content is sent to the server via the Internet.

[1084] Step 3:

[1085] The server receives the task contents transmitted from the transmitting means using the receiving means.

[1086] Input: Data of the submitted task

[1087] Output: The received task data is saved on the server.

[1088] Specific operation: The server's receiving module records the task details in the database.

[1089] Step 4:

[1090] The server analyzes the received task content and generates the appropriate code and configuration using a generative AI model.

[1091] Input: Received task content data

[1092] Output: Generated code and configuration (e.g. Python scripts, API calls)

[1093] Specific operation: A generative AI model (such as GPT-3) analyzes the task content and generates appropriate APIs and code based on the prompt text.

[1094] Step 5:

[1095] The server returns the code and settings generated by the generating means to the terminal using the returning means.

[1096] Input: Generated code and settings

[1097] Output: Code and configuration data returned to the device

[1098] What it does: The server converts the code and settings into packets and sends them over the Internet to the device.

[1099] Step 6:

[1100] The device receives the code and settings returned from the server and executes it as an automated task.

[1101] Input: Code and settings returned from the server

[1102] Output: Executed automation task (e.g. robot status check, notification execution)

[1103] What happens: A script is launched to execute code on the device, performing robot operations and notification tasks.

[1104] Through these steps, the operation and maintenance of factory robots are automated based on tasks entered by the user in natural language, making it possible to automate tasks efficiently even without specialized knowledge.

[1105] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1106] The system of the present invention effectively realizes automated tasks by combining the user's input of automated tasks in natural language with an emotion engine that recognizes and analyzes the user's emotions.

[1107] This system provides an input method on the terminal for users to input the tasks they want to automate in natural language. Using this interface, users can input the tasks they want to automate. For example, they can input "Notify me of the weather forecast by email every day at 9:00 AM."

[1108] The input task content is transmitted to the server by the transmitting means. The server receives the transmitted task content through the receiving means. At this time, the server uses an emotion engine to analyze emotions from the user's input content. The emotion engine can recognize the user's emotions (e.g., joy, sadness, anger) through natural language analysis or voice input analysis.

[1109] The server then uses a generator to generate appropriate code and settings based on the recognized emotion. For example, if the user is dissatisfied with the input, the server can adjust the notification method to provide more detailed information or additional feedback. The generator also automatically selects multiple public APIs and generates code and settings to effectively accomplish the task.

[1110] The generated code and settings are returned to the terminal through the return means. The terminal displays the code and settings received from the server so that the user can easily check them. The user can then run the generated code and settings in a local environment or a cloud environment.

[1111] For example, if a user inputs "Notify me of the weather forecast by email every day at 9 AM" and simultaneously recognizes that the user's emotion is excitement, the system can be implemented to notify the user not only of the weather forecast information for that specific snippet, but also of recommended tourist spots and outdoor events. This ensures that the results of the automated task match the user's emotion, providing a more satisfying experience.

[1112] Therefore, the system of the present invention allows users without advanced programming skills to easily accomplish complex automated tasks through natural language input and emotion recognition, which is expected to improve work efficiency and user experience.

[1113] The processing flow will be explained below.

[1114] Step 1:

[1115] The user uses the input method on the device to input the task they want to automate in natural language. For example, they might input "Notify me of the weather forecast by email every day at 9:00 AM."

[1116] Step 2:

[1117] The terminal transmits the input task content to the server using the transmission means. Specifically, the input text data in natural language is sent to the server in the form of an HTTP request.

[1118] Step 3:

[1119] The server receives the task content through the receiving means. The server-side API endpoint receives this data and prepares it for analysis.

[1120] Step 4:

[1121] The server passes the received task content to the emotion engine, which uses natural language analysis or voice input analysis to recognize the user's emotions. For example, it analyzes positive emotions (joy, excitement) and negative emotions (sadness, dissatisfaction).

[1122] Step 5:

[1123] The server uses a generation method to generate appropriate code and settings based on the recognized emotion. The generation method uses a natural language processing model to select APIs (e.g., weather forecast APIs and email sending APIs) while taking the user's emotion into consideration, and configures the optimal code.

[1124] Step 6:

[1125] The generated code and settings are returned to the terminal through the return means. The server transmits the generated code to the terminal in the form of an HTTP response.

[1126] Step 7:

[1127] The device receives the returned code and displays it for the user to review. The user can then review the generated code, deploy it in their environment, and run it.

[1128] Step 8:

[1129] The user runs the generated code, for example, by setting up a scheduled task to retrieve the weather forecast every day at 9:00 AM and send an email notification.

[1130] Example 2

[1131] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1132] Currently, automating tasks requires programming knowledge and skills, which is a significant hurdle for average users. Furthermore, performing tasks without considering the user's emotions can lead to unsatisfied users and result in poor results. Therefore, there is a need for a system that combines natural language task input with emotion recognition to improve user satisfaction and automate complex tasks.

[1133] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an input means for inputting a task that a user wants to automate in natural language, a transmission means for transmitting the content of the automation task acquired from the input means to the server, a reception means for receiving the content of the automation task transmitted by the transmission means, a sentiment analysis means for analyzing the user's sentiment based on the content of the automation task received by the reception means, a generation means for generating appropriate code and settings based on the sentiment recognized by the sentiment analysis means, and a return means for returning the code and settings generated by the generation means to the terminal. This enables a user without advanced programming skills to easily realize complex automation tasks through input in natural language and emotion recognition.

[1134] Understood. Below are definitions of key terms included in the rewritten claims.

[1135] A "user" is an individual or organization that utilizes the system to configure automated tasks.

[1136] A "task" is a series of operations or processes that a user wants to automate.

[1137] An "input means" is a device or software that includes an interface for a user to input a task in natural language.

[1138] The "transmission means" is a communication means for transmitting the task contents acquired from the input means to the server.

[1139] The "server" is a central processing unit that receives, analyzes, and processes task contents sent by users.

[1140] The "receiving means" is a means by which the server receives the task content transmitted by the transmitting means.

[1141] "Emotion analysis means" refers to software and algorithms that allow the server to analyze emotions from the user's input.

[1142] The "generating means" refers to a processing device and program for generating appropriate code and settings based on the emotion recognized by the emotion analyzing means.

[1143] The "returning means" is a communication means for returning the code and settings generated by the generating means from the server to the terminal.

[1144] "Terminal" means a device for a user to use input means and is used to display the code and settings returned from the server.

[1145] The system of the present invention allows a user to input an automation task using natural language, recognizes the user's emotions, and generates appropriate code and settings. This system is configured by combining an input unit, a transmission unit, a server, a reception unit, an emotion analysis unit, a generation unit, a return unit, and a terminal.

[1146] First, the user uses the input method on the device to input the task they want to automate in natural language. The input method can be a text field or a voice input interface. For example, they can input a specific task such as "Send me a weather forecast email every day at 9:00 AM."

[1147] The input task content is sent to the server by the sending means. A common communication protocol such as an HTTP request is used as the communication means. Next, the server receives the input task content through the receiving means. On the server side, the user's emotions are analyzed using emotion analysis means based on the received task content. The emotion analysis means used here may be a natural language analysis model (e.g., BERT or GPT-3).

[1148] The emotion analysis means recognizes emotions (e.g., joy, sadness, anger) from the user's input. The server then uses the generation means to automatically generate appropriate codes and settings based on the recognized emotions. For example, if the user feels "enjoyed" by the input, the generation means generates codes and settings to notify the user of recommended tourist spots in addition to the weather forecast.

[1149] The generating means automatically selects multiple published APIs and generates code for effectively performing the task. The generated code and settings are returned to the terminal through the returning means. The returning means may be an HTTP response or the like.

[1150] The device displays the code and configuration returned by the server to the user, who can review it and make any necessary corrections or approvals. The user can then run the completed code and configuration in a local or cloud environment. This allows users without advanced programming skills to easily accomplish complex automation tasks through natural language input and emotion recognition.

[1151] For example, if a user inputs "notify me of the weather forecast by email every day at 9:00 AM," and sentiment analysis identifies this as "excitement," the system will configure the system to notify users of information about recommended tourist spots and outdoor events in addition to the weather forecast. This allows users to achieve high satisfaction with the automated notifications.

[1152] An example of a prompt sentence might be:

[1153] "Every day at 9:00 AM, I'll email you the latest weather forecast and the best places to visit that day."

[1154] This allows users to intuitively and efficiently configure and execute automated tasks.

[1155] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1156] Step 1: User enters task

[1157] The user uses the input method on the device to input the task they want to automate in natural language. This interface includes a text field and voice input support. For example, the user might input "Receive a weather forecast email every day at 9:00 AM." The input is captured as text data.

[1158] Input: User's natural language task input (e.g., "Email me the weather forecast every day at 9 AM").

[1159] Output: Task content as text data.

[1160] Step 2: The device sends the task details to the server

[1161] The terminal transmits the acquired task content to the server via a transmission means. The transmission is performed using a communication protocol such as an HTTP request. Specifically, the terminal sends an HTTP "POST" request to the server, and includes the task content in the body of the request.

[1162] Input: Task content as text data.

[1163] Output: HTTP request to the server.

[1164] Step 3: The server receives the task

[1165] The server receives the task content sent through the receiving means. Specifically, the server analyzes the HTTP request and extracts the task content from the body. The received task content is stored as an internal data structure.

[1166] Input: HTTP request.

[1167] Output: Task details (text data) within the server.

[1168] Step 4: The server analyzes the emotions

[1169] The server uses a sentiment analysis method to analyze the sentiment of the user's input. A natural language analysis model such as BERT or GPT-3 can be used for this analysis. Specifically, the server passes the input text to a sentiment analysis API and obtains a sentiment label (e.g., "enjoyed").

[1170] Input: Task contents (text data) within the server.

[1171] Output: An emotion label (e.g., "enjoyment").

[1172] Step 5: The server generates the appropriate code and configuration

[1173] The server uses a generation method to generate appropriate code and settings based on the recognized emotion. Specifically, the server selects the appropriate API based on the emotion analysis results and automatically generates code to accomplish the task. For example, if the emotion is "excitement," it generates code that combines a weather forecast API and a tourist destination information API.

[1174] Input: emotion label, task content.

[1175] Output: Auto-generated code and configuration.

[1176] Step 6: The server returns the generated code

[1177] The generated code and settings are returned to the terminal through the return means. Specifically, the server sends the generated code to the terminal as an HTTP "POST" response.

[1178] Inputs: Auto-generated code and settings.

[1179] Output: HTTP response to the device.

[1180] Step 7: The terminal will display the code

[1181] The terminal displays the received code and settings to the user. Specifically, the terminal visually displays the received code on the interface so that the user can confirm the contents.

[1182] Input: The HTTP response from the server.

[1183] Output: The code and settings displayed on the interface.

[1184] Through the specific actions performed at each step, users can easily accomplish complex automation tasks through natural language input and emotion recognition, even without advanced programming skills.

[1185] (Application example 2)

[1186] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1187] Conventional automation systems could automatically process the content of tasks specified by the user, but they were unable to respond flexibly while taking into account the user's emotions and circumstances. This limited the improvement of user satisfaction and work efficiency. Furthermore, in production sites such as factories, there was a need for support to efficiently complete tasks while reducing worker stress and fatigue. A system that solves these issues and realizes task automation while taking into consideration the emotions of the user or worker is needed.

[1188] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes an input means for a user to input a task they wish to automate in natural language, a generation means for generating appropriate code and settings based on the content of the automation task and the emotion analysis results acquired from the input means, and a return means for returning the code and settings generated by the generation means to the terminal. This enables task automation that takes the user's emotions into consideration, realizes more flexible and satisfying work support, and enables factory workers to efficiently complete tasks while reducing stress and fatigue.

[1189] An "input means" is a device or software that provides an interface for a user to input, in natural language, the task they wish to automate.

[1190] The "transmission means" is a device or software having a function for transmitting the content of the automation task acquired from the input means to the server via the network.

[1191] The "receiving means" is a device or software having a function for receiving the content of the automated task transmitted to the server by the transmitting means.

[1192] The "emotion analysis result" is emotional information analyzed from the user's input content, and is data generated by the emotion analysis engine.

[1193] The "generating means" is a device or software for generating appropriate code and settings based on the content of the automated task and the emotion analysis results received by the receiving means.

[1194] The "returning means" is a device or software having a function for returning the code and settings generated by the generating means to the terminal.

[1195] The term "user interface" refers to a screen or operating means that allows a user to input information in natural language on a terminal.

[1196] An "emotion analysis engine" is an algorithm and software that analyzes voice and text input from users and recognizes their emotions.

[1197] The embodiments of the present invention will be described in detail below.

[1198] First, a user uses a terminal to input the task to be automated in natural language. A user interface is provided on the terminal for the user to input instructions in natural language. This user interface is an interface that accepts natural language tasks through voice input or text input. For example, the user can input an instruction such as "move the next pallet to the conveyor belt."

[1199] Next, the input task content is transmitted from the terminal to the server, and this transmission means has a communication function for transmitting data to the server via a network.

[1200] The server analyzes the received task content and recognizes the user's emotions through a sentiment analysis engine. The sentiment analysis engine analyzes the text or voice data from the user's input to generate emotional data. For example, text analysis can determine whether the user is tired or happy.

[1201] The server then uses a generator to generate appropriate code and settings based on the received task content and sentiment analysis results. The generator automatically selects and combines a number of publicly available APIs to generate code to accomplish the required task.

[1202] The generated code and settings are returned from the server to the terminal. The return means has a communication function for transmitting the generated code and settings to the terminal via a network. The user can check the returned code and settings and make corrections or approvals if necessary.

[1203] For example, if a user inputs "move the next pallet to the conveyor belt" and the emotion analysis engine recognizes the user's emotion as "tired," the system can adjust to temporarily suspend the work, suggest a break, and notify the user when it's time to resume. In this way, it becomes possible to support efficient work while taking into consideration the worker's emotions.

[1204] Examples of prompts include:

[1205] "Move the next pallet to the conveyor belt."

[1206] "Text: 'Move the next pallet to the conveyor belt.'; Sentiment: 'Neutral'"

[1207] This system makes it possible to automate tasks while taking emotions into account, which is expected to improve user satisfaction and work efficiency.

[1208] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1209] Step 1:

[1210] The user uses the user interface on the device to input the task they want to automate in natural language. This input is captured by the input means and stored as digital data on the device. An example of this input might be, "Move the next pallet to the conveyor belt."

[1211] Step 2:

[1212] The input natural language task is sent to the server via the transmission means over the network. The transmission means packetizes the digital data stored on the terminal and converts it into a format that the server can receive. The output of this process is digital data containing the task content on the server side.

[1213] Step 3:

[1214] The server receives the transmitted task content using the receiving means. The received data includes the task content in natural language entered by the user. The server passes this data to the analysis engine and starts natural language analysis.

[1215] Step 4:

[1216] The server uses a natural language analysis engine to analyze the content of the input task, and also recognizes emotions through a sentiment analysis engine. The input is text data of the task content, and the sentiment analysis engine analyzes the text for positive, negative, and neutral aspects. For example, the input "Move the next pallet to the conveyor belt" is analyzed as "neutral."

[1217] Step 5:

[1218] The server generates appropriate code and settings based on the analyzed task content and emotion analysis results. The server uses a generation method to automatically select multiple APIs and dynamically generate the necessary code. For example, if the recognized emotion is "tired," it generates code that includes functions to pause and suggest breaks. The input of this step is the task content and emotion analysis results, and the output is the generated code and settings.

[1219] Step 6:

[1220] The generated code and settings are sent from the server to the terminal, again using the return means. The server packetizes the generated data over the network and converts it into a format acceptable to the terminal. The output of this process is the generated code and settings data that is transmitted to the user's terminal.

[1221] Step 7:

[1222] The device begins operation based on the received code and settings. The user can check the progress of the task and make corrections or approvals if necessary. For example, this can include time management during breaks and preparation for the next task. User feedback is input back into the system and can be reflected in subsequent task executions.

[1223] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1224] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1225] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1226] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1227] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1228] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1229] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1230] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1231] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1232] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1233] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1234] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1235] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1236] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1237] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1238] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1239] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1240] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1241] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1242] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1243] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1244] The following is further disclosed regarding the above embodiment.

[1245] (Claim 1)

[1246] an input means for a user to input a task that the user wants to automate in natural language;

[1247] a transmission means for transmitting the content of the automation task acquired from the input means to a server;

[1248] a receiving means for receiving the content of the automation task transmitted by the transmitting means;

[1249] a generating means for generating appropriate code and settings based on the content of the automation task received by the receiving means;

[1250] a return means for returning the code and settings generated by the generation means to the terminal;

[1251] A system including:

[1252] (Claim 2)

[1253] 10. The system of claim 1, wherein the generating means automatically selects and uses a plurality of exposed APIs in generating the generated code and configuration.

[1254] (Claim 3)

[1255] 2. The system of claim 1, wherein the input means includes a user interface provided on a terminal.

[1256] "Example 1"

[1257] (Claim 1)

[1258] an input means for a user to input a task that the user wants to automate in natural language;

[1259] a transmission means for transmitting the content of the automation task acquired from the input means to a server;

[1260] a receiving means for receiving the content of the automation task transmitted by the transmitting means;

[1261] an analysis means for analyzing the content of the automation task received by the receiving means and selecting a required API and a combination thereof;

[1262] a generating means for generating appropriate code and settings using a generating AI model based on the APIs and combinations thereof selected by the analyzing means;

[1263] a return means for returning the code and settings generated by the generation means to the terminal;

[1264] execution means for providing an execution environment for executing the code and settings returned by the return means;

[1265] A system including:

[1266] (Claim 2)

[1267] 10. The system of claim 1, wherein the generating means automatically selects and uses a plurality of exposed APIs in generating the generated code and configuration.

[1268] (Claim 3)

[1269] 2. The system of claim 1, wherein the input means includes a user interface provided on a terminal.

[1270] "Application Example 1"

[1271] (Claim 1)

[1272] an input means for a user to input a task that the user wants to automate in natural language;

[1273] a transmission means for transmitting the content of the automation task acquired from the input means to a server;

[1274] a receiving means for receiving the content of the automation task transmitted by the transmitting means;

[1275] a generating means for generating appropriate code and settings based on the content of the automation task received by the receiving means;

[1276] The generating means uses a generating AI model and includes a function of automatically selecting an API and generating code according to a prompt sentence;

[1277] a return means for returning the code and settings generated by the generation means to the terminal;

[1278] A system including:

[1279] (Claim 2)

[1280] 10. The system of claim 1, wherein the generating means automatically selects and uses a plurality of exposed APIs in generating the generated code and configuration.

[1281] (Claim 3)

[1282] 2. The system of claim 1, wherein the input means includes a user interface provided on a terminal.

[1283] "Example 2: Combining Emotion Engines"

[1284] Understood. Here are the rewritten claims:

[1285] (Claim 1)

[1286] an input means for a user to input a task that the user wants to automate in natural language;

[1287] a transmission means for transmitting the content of the automation task acquired from the input means to a server;

[1288] a receiving means for receiving the content of the automation task transmitted by the transmitting means;

[1289] emotion analysis means for analyzing the emotion of a user based on the content of the automated task received by the receiving means;

[1290] a generating means for generating appropriate code and settings based on the emotion recognized by the emotion analyzing means;

[1291] a return means for returning the code and settings generated by the generation means to the terminal;

[1292] A system including:

[1293] (Claim 2)

[1294] 10. The system of claim 1, wherein the generating means automatically selects and uses a plurality of exposed APIs in generating the generated code and configuration.

[1295] (Claim 3)

[1296] 2. The system of claim 1, wherein the input means includes a user interface provided on a terminal.

[1297] "Application example 2 when combining emotion engines"

[1298] (Claim 1)

[1299] an input means for a user to input a task that the user wants to automate in natural language;

[1300] a transmission means for transmitting the content of the automation task acquired from the input means to a server;

[1301] a receiving means for receiving the content of the automation task transmitted by the transmitting means;

[1302] a generating means for generating appropriate code and settings based on the content of the automation task and the emotion analysis result received by the receiving means;

[1303] a return means for returning the code and settings generated by the generation means to the terminal;

[1304] A system including:

[1305] (Claim 2)

[1306] 10. The system of claim 1, wherein the generating means automatically selects and uses a plurality of exposed APIs in generating the generated code and configuration.

[1307] (Claim 3)

[1308] 2. The system of claim 1, wherein the input means includes a user interface and an emotion analysis engine provided on a terminal. [Explanation of symbols]

[1309] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. an input means for a user to input a task that the user wants to automate in natural language; a transmission means for transmitting the content of the automation task acquired from the input means to a server; a receiving means for receiving the content of the automation task transmitted by the transmitting means; a generating means for generating appropriate code and settings based on the content of the automation task received by the receiving means; a return means for returning the code and settings generated by the generation means to the terminal; A system including:

2. 2. The system of claim 1, wherein the generating means automatically selects and uses a plurality of exposed APIs in generating the generated code and configuration.

3. 2. The system of claim 1, wherein the input means includes a user interface provided on a terminal.

Images