system

Abstract

Provide a system. A means for inputting instructions in natural language; means for analyzing input instructions and extracting components; means for generating robotic process automation code based on the extracted components; a means for displaying the generated code to the user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Robotic Process Automation (RPA) technology is widely used to improve the efficiency of business processes, but its creation requires advanced skills and specialized knowledge. This makes it difficult for general users to adopt RPA, and the increased implementation costs and development time are barriers. Furthermore, existing tools are difficult to operate intuitively, requiring a high learning curve for non-experts. The purpose of this invention is to solve these issues and make RPA technology easily accessible to more people. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for inputting instructions in natural language, a means for analyzing the input instructions and extracting components, a means for generating robot process automation code based on the extracted components, and a means for displaying the generated code to a user. The input instructions are analyzed by a generative AI model, and appropriate RPA code is automatically generated. This system allows users to intuitively create RPA robots without specialized knowledge and supports user operation. Furthermore, the generated code is user-editable, providing high flexibility.

[0006] A "means for inputting instructions in natural language" is a technical means that provides an interface that allows users to input instructions using everyday words and sentences.

[0007] "Means for analyzing input instructions and extracting components" refers to technical means for interpreting natural language instructions obtained from a user and identifying and extracting the important components that make up the instructions (e.g., actions, conditions, targets, etc.).

[0008] A "generative AI model" is an artificial intelligence model that uses natural language processing and machine learning to understand user input and provide appropriate information and processing methods.

[0009] The "means for generating robotic process automation code" refers to a technical means for automatically creating RPA (robotic process automation) program code based on analyzed components.

[0010] "Means for displaying generated code to the user" refers to technical means for visually presenting the generated RPA code to the user and providing an interface for the user to check and edit the content as necessary.

[0011] "User-editable means" refers to technical means that allow users to manually modify or change the generated RPA code. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] In order to implement the present invention, the server, the terminal, and the user each play a specific role. Specific embodiments for implementing the present invention are described below.

[0034] overview

[0035] This system automatically generates code for RPA (Robotic Process Automation) robots by inputting instructions in natural language from the user. This system is realized by linking a server and terminals.

[0036] Program Operation

[0037] 1. User input

[0038] The user inputs instructions in natural language through the input interface of the terminal.

[0039] For example: "Back up all files in a specific folder to another folder every Monday at 9 AM."

[0040] 2. Sending instructions

[0041] The terminal transmits the input instructions to the server.

[0042] This allows the user's requests to be communicated throughout the system.

[0043] 3. Parsing the instructions

[0044] The server analyzes the received instruction in an instruction analysis module.

[0045] Specifically, it uses a generative AI model to break down input natural language instructions into their component parts.

[0046] In this example, the elements "every Monday at 9 AM," "specific folder," "all files," and "backup to another folder" are extracted.

[0047] 4. Generate RPA code

[0048] Based on the analysis results, the server generates appropriate code using the RPA robot generation module.

[0049] Based on the extracted elements, an RPA script is automatically generated in a programming language such as Python.

[0050] As an example result, code is generated to back up all files in a specified folder at a specified time.

[0051] 5. Sending the product

[0052] The generated code is sent from the server to the device.

[0053] This is so that the user can verify this code.

[0054] 6. Viewing and editing the generated output

[0055] The terminal displays the received code to the user.

[0056] The user can review the displayed code and edit it if necessary.

[0057] For example, you can manually change the path of the backup folder, change the backup time, etc.

[0058] Specific examples

[0059] Example 1: Backing up files periodically

[0060] User prompts:

[0061] The user enters "Copy files in the working folder to the backup folder every day at 5 p.m." into the terminal's input interface.

[0062] Parsing instructions:

[0063] The server parses the instructions and extracts the components "every day at 5pm", "working folder", and "copy to backup folder".

[0064] RPA code generation:

[0065] The server automatically generates a Python script based on the extracted information, which backs up files in a specified folder every day at 5 PM.

[0066] View and edit the artifact:

[0067] The terminal will display the generated script to the user, who can edit the folder path and time, and give a final confirmation.

[0068] Example 2: Frequent email checks

[0069] User prompts:

[0070] The user types into the terminal, "Check email every hour and notify me if there is new email."

[0071] Parsing instructions:

[0072] The server analyzes the components "every hour," "check email," and "notify if new email arrives."

[0073] RPA code generation:

[0074] The server generates Python code that checks for emails every specified time interval and notifies you if new emails arrive.

[0075] View and edit the artifact:

[0076] The terminal displays the generated code for the user to review and make any necessary corrections.

[0077] This allows users to efficiently create and operate RPA robots without advanced programming knowledge. By using generative AI models, it is possible to perform advanced analysis of natural language and provide accurate automation code.

[0078] The processing flow will be explained below.

[0079] Step 1:

[0080] The user uses the input interface of the device to input instructions in natural language, for example, "Copy files in a specific folder to a backup folder every day at 3 p.m."

[0081] Step 2:

[0082] The device sends the input natural language instructions to the server, usually using an HTTP request.

[0083] Step 3:

[0084] To parse the received instructions, the server invokes an instruction parsing module, which uses a generative AI model to break the instructions down into their component parts.

[0085] Step 4:

[0086] The server's instruction analysis module breaks down the natural language instruction into its components, in this case extracting elements such as "every day at 3 p.m.", "specific folder," and "copy to backup folder."

[0087] Step 5:

[0088] The RPA robot generation module on the server automatically generates RPA scripts, such as Python code, based on the extracted elements. The generated code executes the specified actions at the specified time.

[0089] Step 6:

[0090] The server sends the generated RPA code to the terminal, allowing the user to check the generated code.

[0091] Step 7:

[0092] The terminal displays the received RPA code to the user, who can visually check the code and edit it if necessary.

[0093] Step 8:

[0094] Users can edit the generated code as needed, for example, to change the folder path or backup time.

[0095] Step 9:

[0096] The user performs a final check and deploys the modified code to a runtime environment. Testing may also be performed to ensure the generated code works correctly.

[0097] This series of steps allows users to easily create and operate RPA robots based on instructions entered in natural language.

[0098] Example 1

[0099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] Conventional robotic process automation systems require advanced programming knowledge and are difficult for average users to operate. In addition, it is difficult to input instructions in natural language, which makes it difficult to efficiently analyze input content and generate appropriate automation code.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0102] In this invention, the server includes a means for a user to input instructions in natural language, a means for analyzing the input instructions using a generative AI model and extracting components, and a means for generating robotic process automation code based on the extracted components, thereby enabling users without advanced programming knowledge to efficiently execute automation tasks through instructions in natural language, improving the operability and convenience of the entire system.

[0103] "User" refers to an entity that uses the system to input instructions in natural language.

[0104] A "terminal" refers to a device through which a user inputs instructions and transmits the instructions to a server.

[0105] "Server" refers to a central control device that analyzes received instructions, generates robotic process automation code, and sends it to the terminal.

[0106] A "generative AI model" refers to an artificial intelligence algorithm that analyzes input natural language instructions and extracts components.

[0107] "Instructions in natural language" refers to instructions input by the user using a language that humans use on a daily basis (e.g., Japanese or English).

[0108] "Parsing instructions" refers to the process of understanding received natural language instructions and breaking them down into actionable components.

[0109] "Component extraction" refers to the process of extracting the elements needed for an automation task from the parsed natural language instructions.

[0110] "Robotic process automation code" refers to program code for performing automated tasks based on extracted components.

[0111] "Generating code" refers to the process of writing out the extracted components as program code.

[0112] "Sending the code" refers to the process of sending the generated program code from the server to the terminal.

[0113] "Displaying the code" refers to the process of presenting the generated program code to the user in a visible form on the terminal screen.

[0114] "Editing the code" refers to the process by which a user adjusts, adds to, or modifies the content of the generated program code.

[0115] The present invention provides a system for automatically generating robotic process automation (RPA) code based on instructions input by a user in natural language. Specific embodiments for carrying out the invention are described below.

[0116] overview

[0117] The user inputs instructions in natural language through the terminal, and the input instructions are analyzed by the server, which generates the automation code. This system is realized mainly through the cooperation between the terminal and the server.

[0118] Hardware and software used

[0119] Devices: PCs, smartphones, tablets, etc.

[0120] Server: Cloud server, on-premise server

[0121] Generative AI models: AI models used for natural language analysis (e.g., GPT-4 (registered trademark))

[0122] Programming language: Python

[0123] Data processing and calculation

[0124] 1. Input interface:

[0125] The user inputs instructions in natural language through the device's input interface. Example: "Copy the files in the working folder to the backup folder every day at 5 p.m."

[0126] The instructions are converted into formatted data (e.g., JSON) and sent to the server.

[0127] 2. Parsing the instructions:

[0128] The server analyzes the received instructions using a generative AI model, and in an example of extracting components, the elements extracted are "every day at 5 p.m.", "working folder," and "copy to backup folder."

[0129] 3. Generate RPA code:

[0130] The server generates a Python script based on the extracted components. Based on the extracted elements, code is generated to back up files in a specific folder at a specified time.

[0131] 4. Send and display the code:

[0132] The generated code is sent from the server to the terminal, which displays the code to the user, who can then view and edit the displayed code.

[0133] Specific examples

[0134] Example 1: Backing up files periodically

[0135] Example of user input:

[0136] The user enters "Copy files in the working folder to the backup folder every day at 5 p.m." into the terminal's input interface.

[0137] Parsing instructions:

[0138] The server extracts the components "every day at 5pm", "working folder", and "copy to backup folder".

[0139] RPA code generation:

[0140] The server automatically generates a Python script based on the extracted information, which backs up files in a specified folder every day at 5 PM.

[0141] Send and display the artifact:

[0142] The terminal will display the generated script to the user, who can edit the folder path and time, and give a final confirmation.

[0143] Example 2: Frequent email checks

[0144] Example of user input:

[0145] The user types into the terminal, "Check email every hour and notify me if there is new email."

[0146] Parsing instructions:

[0147] The server analyzes the components "every hour," "check email," and "notify if new email arrives."

[0148] RPA code generation:

[0149] The server generates Python code that checks for emails every specified time interval and notifies you if new emails arrive.

[0150] Send and display the artifact:

[0151] The terminal displays the generated code for the user to review and make any necessary corrections.

[0152] The system allows users to efficiently execute automation tasks by inputting instructions in natural language without advanced programming knowledge, and uses a generative AI model to intelligently analyze natural language and generate accurate and efficient automation code.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Step 1:

[0155] User instruction input

[0156] Input: The user types instructions into the terminal in natural language. Example: "Copy the files in my working folder to the backup folder every day at 5pm."

[0157] Data processing and data calculation: The device converts input instructions into formatted data (e.g., JSON).

[0158] Output: Formatted data is produced.

[0159] How it works: The user opens a dedicated app on a device such as a PC or smartphone and enters instructions. The device converts the input into JSON format data.

[0160] Step 2:

[0161] Sending instructions

[0162] Input: Formatted data (e.g. JSON)

[0163] Data processing and data calculation: The device sends formatted data to the server using an HTTPS request.

[0164] Output: Data is sent to the server.

[0165] Specific behavior: The device sends the input natural language instruction to the server via an HTTPS request. The data is sent as a POST request to the endpoint (e.g., https: / / api.example.com / parse_instruction).

[0166] Step 3:

[0167] Parsing instructions

[0168] Input: Formatted data received by the server (e.g. JSON)

[0169] Data processing and data calculation: The server invokes the generative AI model, parses the natural language instructions, and extracts the components.

[0170] Output: Extracted components (e.g. "Every day at 5 PM", "Working folder", "Copy to backup folder")

[0171] What happens: The server uses a generative AI model (e.g., the GPT-4 model) to analyze the received data. As a result of the analysis, instructions are broken down into their component parts.

[0172] Step 4:

[0173] RPA code generation

[0174] Input: Extracted components

[0175] Data processing and data calculation: The server generates a Python script using the RPA robot generation module.

[0176] Output: Generated Python script

[0177] Specific operation: Based on the parsed components, the server automatically generates a Python script that executes the specified action, such as backing up files in a specific folder at a specified time.

[0178] Step 5:

[0179] Sending artifacts

[0180] Input: Generated Python script

[0181] Data processing and data calculation: The server sends the generated script to the terminal as JSON format data.

[0182] Output: Submitted output (Python script)

[0183] Specific operation: The server sends the generated script to the terminal. The generated script is returned as part of JSON in response to the endpoint (e.g. https: / / api.example.com / get_generated_code).

[0184] Step 6:

[0185] Viewing and editing the generated output

[0186] Input: Submitted product (Python script)

[0187] Data processing and data calculation: The terminal displays the received script to the user and provides an interface for the user to edit it.

[0188] Output: Edited Python script (if needed)

[0189] Specific operation: The terminal displays the received script to the user. The user can check the displayed script and make any necessary modifications. For example, the user can check the contents of the script in the GUI and edit the backup folder path or execution time. When the user saves the edited content, the new instructions are resent to the server, and a new script reflecting the necessary modifications is generated.

[0190] (Application example 1)

[0191] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0192] In recent years, there has been a growing need for inventory management and work efficiency in brick-and-mortar stores. However, traditional inventory management methods require a lot of manual work, which takes time and effort, making efficient business operations difficult. In particular, periodic tasks such as replenishing stock and managing unsold items place a heavy burden on store staff. To solve this problem, a system is needed that can automate a series of inventory management tasks simply by inputting instructions in natural language.

[0193] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0194] In this invention, the server includes a means for inputting instructions in natural language, a means for analyzing the input instructions and extracting components, a means for generating robotic process automation code based on the extracted components, a means for executing the process at a specific time to automate inventory management tasks in a physical store, and a means for displaying the generated code to a user. This allows store clerks to automate inventory management tasks simply by inputting instructions in natural language, thereby significantly reducing time and labor. Furthermore, the input instructions can be accurately analyzed using a generative AI model, enabling an efficient automation process.

[0195] "Natural language" refers to the language that humans use on a daily basis, and is used to communicate intentions through dialogue, etc.

[0196] An "instruction" is a command or order given to perform a particular action or process.

[0197] "Analysis" is the process of breaking down complex data or information to reveal individual elements and their relationships.

[0198] A "component" is an individual part or element that makes up a whole.

[0199] "Robotic process automation code" refers to a program written to automatically execute a specific business process.

[0200] A "brick and mortar store" is a retail establishment that has a physical location and offers goods and services.

[0201] "Inventory management work" refers to the work of managing the receipt, dispatch, storage, and movement of goods and materials.

[0202] The term "specific time" refers to a specific designated time or date.

[0203] A "generative AI model" is an algorithm that uses artificial intelligence to analyze natural language and generate code or data tailored to a specific purpose.

[0204] "User" means any individual or entity that uses the System or Services.

[0205] A "server" is a computer system that processes data and provides services over a network.

[0206] The present invention provides a system for automating inventory management tasks in a physical store. Specific embodiments of the invention will be described below.

[0207] First, a user uses a smartphone application to input instructions in natural language, such as "Replenish the shelves in the cosmetics section every day at 3 p.m." This instruction is then sent from the smartphone to the server.

[0208] The server uses a generative AI model to analyze the received natural language instructions and break them down into their component parts. The software used is a natural language processing tool powered by a generative AI model. By analyzing the instructions, elements such as the time, target section, and task to be performed are extracted.

[0209] The server then generates robotic process automation code based on the extracted components to automate inventory management tasks in physical stores. This code is written in a programming language such as Python and performs tasks such as replenishing specific stock at a specified time. The code generation uses the Python libraries "schedule" and "shutill."

[0210] The generated code is sent from the server to the smartphone and displayed to the user. The user can check the generated code and edit it if necessary. This allows the user to change, for example, the path of the backup folder or the processing time.

[0211] In the present invention, the following specific examples are given.

[0212] For example, if a store associate types in "Restock the shelves in the cosmetics section every day at 3 PM," this instruction is parsed and Python code is generated that will automatically replenish the shelves from a specific folder every day at 3 PM.

[0213] An example of a prompt sentence to input to the generative AI model is as follows:

[0214] "Please parse the instructions entered by the user appropriately and generate the following RPA robot code:

[0215] Instructions: Restock the shelves in the cosmetics section every day at 3pm.

[0216] Required tasks:

[0217] Copy the files in the specified folder to the backup folder.

[0218] In this way, users can implement efficient automated processes tailored to their own business operations without needing advanced programming knowledge, which will significantly reduce the workload of inventory management in physical stores.

[0219] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0220] Step 1:

[0221] A user launches a smartphone application and inputs instructions in natural language, such as "Replenish the shelves in the cosmetics section every day at 3:00 PM." This instruction is entered into the smartphone and sent to the server.

[0222] Input: User's natural language instructions

[0223] Output: Send instructions to the server

[0224] Step 2:

[0225] The server retrieves the received instructions and parses them using a generative AI model. The instruction parsing module analyzes the natural language instructions and breaks them down into their component parts: "every day at 3 p.m.", "cosmetics section," and "replenish."

[0226] Input: User's natural language instructions

[0227] Data processing: natural language analysis and component extraction

[0228] Output: Decomposed components

[0229] Step 3:

[0230] The server generates robotic process automation code based on the analyzed components. Specifically, the generative AI model generates Python code that restocks the shelves in the cosmetics section every day at 3:00 PM.

[0231] Input: Parsed components

[0232] Data transformation: component-based code generation

[0233] Output: Generated Python code

[0234] Step 4:

[0235] The generated Python code is sent from the server to the smartphone, and is displayed on the application screen so that the user can easily check the generated code.

[0236] Input: Generated Python code

[0237] Output: Sends and displays code on smartphone

[0238] Step 5:

[0239] Users can check the generated code on their smartphone and edit it if necessary, for example, to change the backup folder path or processing time.

[0240] Input: Generated Python code

[0241] Output: User can review and edit the code

[0242] Step 6:

[0243] Once the user has finished editing the Python code, it is sent back to the server, which receives it and stores it for use the next time the task is run.

[0244] Input: User-edited Python code

[0245] Output: Send the code to the server and save it

[0246] Step 7:

[0247] The server executes the Python code at the specified time according to the schedule, which then executes the process of replenishing inventory from the specified folder, automating the process.

[0248] Input: Edited Python code

[0249] Output: Automatic execution of inventory replenishment work

[0250] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0251] A specific embodiment of the present invention is a system in which a server, a terminal, a user, and an emotion engine work in cooperation with each other, as will be described in detail below.

[0252] overview

[0253] This system automatically generates code for RPA (Robotic Process Automation) robots based on instructions entered by the user in natural language, and further adjusts the generation process and display content based on the user's emotional state. The system is composed of a server, terminals, users, and an emotion engine.

[0254] Program Operation

[0255] 1. User input

[0256] The user uses the input interface of the device to input instructions in natural language, for example, "Copy files in a specific folder to a backup folder every day at 3 p.m."

[0257] 2. Sending instructions

[0258] The device sends the input natural language instructions to the server, usually using an HTTP request.

[0259] 3. Parsing the instructions

[0260] The server parses the received instructions in an instruction parsing module, which uses a generative AI model to break the instructions down into their component parts.

[0261] 4. Emotion recognition

[0262] In parallel, the server activates an emotion engine to recognize the user's emotional state, which identifies emotions from the user's input, facial expressions, and tone of voice.

[0263] 5. Extracting Components

[0264] The server's instruction analysis module breaks down the natural language instruction into its components, in this case extracting elements such as "every day at 3 p.m.", "specific folder," and "copy to backup folder."

[0265] 6. RPA Code Generation

[0266] The RPA robot generation module on the server automatically generates RPA scripts, such as Python code, based on the extracted elements. The generated code executes the specified actions at the specified time.

[0267] 7. Emotion-Based Adjustment

[0268] The emotion engine can adjust the generation process and display content based on the user's emotions. For example, if the user is feeling stressed, the engine can provide a detailed explanation of the generated code or a simplified interface.

[0269] 8. Sending the Product

[0270] The server sends the generated RPA code to the terminal, allowing the user to check the generated code.

[0271] 9. Viewing and editing the generated output

[0272] The terminal displays the received RPA code to the user, who can visually check the code and edit it if necessary.

[0273] 10. User Verification and Execution

[0274] The user edits the generated code as needed, performs a final check, and deploys the modified code to the execution environment to verify that it works correctly.

[0275] Specific examples

[0276] Example 1: Backing up files periodically

[0277] User prompts:

[0278] The user types into the terminal's input interface, "Copy the files in the working folder to the backup folder every day at 5 p.m."

[0279] Instruction analysis and emotion recognition:

[0280] The server analyzes the instruction and extracts the components "every day at 5 PM," "work folder," and "copy to backup folder." At the same time, the emotion engine analyzes the user's emotions and recognizes that they are feeling stressed.

[0281] RPA code generation and emotion-based adjustment:

[0282] The server generates a backup script based on the extracted information. Because the user is stressed, the emotion engine provides an interface with detailed explanations.

[0283] View and edit the artifact:

[0284] The terminal will display the generated script to the user, and the user can edit the folder path and time, with easy editing options to reduce frustration.

[0285] Example 2: Frequent email checks

[0286] User prompts:

[0287] The user types into the terminal, "Check email every hour and notify me if there is new email."

[0288] Instruction analysis and emotion recognition:

[0289] The server analyzes the components "every hour," "check email," and "notify if new email arrives," and the emotion engine recognizes that the user is excited.

[0290] RPA code generation and emotion-based adjustment:

[0291] The server generates the email checking script and adjusts the emotion engine to provide a simple interface.

[0292] View and edit the artifact:

[0293] The terminal displays the generated code for the user to review and make any necessary corrections.

[0294] In this way, by combining the user's natural language input with emotion recognition functionality, it is possible to create an RPA robot that is easier to use and meets individual needs. By linking the generative AI model with the emotion engine, it is possible to provide appropriate support according to the user's condition.

[0295] The processing flow will be explained below.

[0296] Step 1:

[0297] The user uses the input interface of the terminal to input instructions in natural language, for example, "Copy files in a specific folder to a backup folder every day at 3 p.m."

[0298] Step 2:

[0299] The device sends the input natural language instructions to the server, typically via an HTTP request.

[0300] Step 3:

[0301] The server invokes an instruction parsing module to parse the received instruction, using a generative AI model to break the instruction down into its component parts.

[0302] Step 4:

[0303] The server's instruction analysis module breaks down the natural language instruction into its component parts, in this case extracting elements such as "every day at 3pm," "specific folder," and "copy to backup folder."

[0304] Step 5:

[0305] In parallel, the server activates an emotion engine to recognize the user's emotional state, which identifies emotions from the user's input, facial expressions, and tone of voice.

[0306] Step 6:

[0307] The emotion engine recognizes the user's emotions and feeds that information back to the instruction analysis module. For example, if it determines that the user is feeling stressed, that information is incorporated into the analysis process.

[0308] Step 7:

[0309] The server's RPA robot generation module automatically generates RPA scripts such as Python code based on the extracted elements and emotion information. If the user is feeling stressed, the module adjusts the script to generate detailed comments and easy-to-understand code.

[0310] Step 8:

[0311] The server sends the generated RPA code to the terminal, allowing the user to check the generated code.

[0312] Step 9:

[0313] The terminal displays the received RPA code to the user, who can visually check the code and edit it if necessary.

[0314] Step 10:

[0315] Users can edit the generated code as needed, for example, to change the folder path or backup time. The emotion engine continuously monitors the user's emotions and adjusts the interface as needed.

[0316] Step 11:

[0317] The user performs a final check and deploys the modified code to the execution environment. The server also provides a function to support testing to ensure that the generated code works correctly.

[0318] This specific processing flow allows users to input instructions in natural language and easily create, edit, and execute the RPA robot code generated based on those instructions. Furthermore, the introduction of an emotion engine makes it possible to provide flexible support according to the user's emotional state.

[0319] Example 2

[0320] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0321] Systems that generate robotic process automation (RPA) code are required to allow users to easily input instructions using natural language, and to adapt the generated code and interface to suit the user's emotional state. Conventional systems have struggled to accurately analyze user input and generate appropriate RPA code, often resulting in frustration for users. Another problem is that the generated code is not intuitive and difficult to edit.

[0322] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0323] In this invention, the server includes means for analyzing instructions and extracting components, means for recognizing the user's emotional state, and means for generating robotic process automation code based on the extracted components and the user's emotional state. This allows the user to simply input instructions in natural language, and automatically generates appropriate RPA code that takes the user's emotional state into consideration, providing an intuitive and easy-to-edit interface.

[0324] "Means for inputting instructions in natural language" refers to technology that provides an interface for users to input instructions and commands to a system using natural language, which is a human language.

[0325] The "means for transmitting input instructions to a server" refers to a technology that provides a communication protocol or mechanism for a terminal to send instructions input by a user to a server.

[0326] "Means for analyzing instructions received by the server and extracting components" refers to a technology that analyzes natural language instructions received by the server using an artificial intelligence model or natural language processing technology, and extracts the elements that make up the meaning of the instructions.

[0327] "Means for the server to recognize the user's emotional state" refers to a technology in which the server analyzes the user's input and other indicators (facial expressions, tone of voice, etc.) to determine the user's emotions and mood.

[0328] The "means for generating robotic process automation code based on extracted components and emotional state" is a technology that automatically generates appropriate robotic process automation code based on the components of instructions extracted through analysis and the user's current emotional state.

[0329] "Means for adjusting the generation process and display content based on the user's emotional state" refers to technology that adjusts the code generation process and the screen and interface that the user sees to optimal forms depending on the user's emotional state.

[0330] "Means for transmitting the generated code to the user's terminal" refers to a communication technology for transmitting the robotic process automation code generated by the server back to the user's device.

[0331] "Means for displaying the generated code to the user and making it editable" refers to technology that visually displays the generated code to the user on the terminal and provides an interface that allows the user to edit and modify the code.

[0332] A "generative AI model" is an artificial intelligence model used to analyze natural language instructions entered by a user and extract components, including, for example, machine learning and deep learning algorithms.

[0333] A "prompt" is an input text given to a generative AI model, containing instructions and information that enable the model to understand and parse natural language.

[0334] As a specific embodiment for carrying out the present invention, a system is used in which a server, a terminal, a user, and an emotion engine work in cooperation with each other, as will be described in detail below.

[0335] System configuration

[0336] This system automatically generates robotic process automation (RPA) code based on instructions entered by the user in natural language, and further adjusts the generation process and display content based on the user's emotional state. This system mainly consists of the following elements:

[0337] 1. User input

[0338] The user inputs instructions in natural language using the input interface of the device. The device can be a general computer, smartphone, tablet, or other device. For example, the user might input "Copy files in a specific folder to a backup folder every day at 3:00 PM."

[0339] 2. Sending instructions

[0340] The terminal sends the input instructions in natural language to the server, which receives the instructions using a specified communication protocol (e.g., HTTP).

[0341] 3. Parsing the instructions

[0342] The server analyzes the received instructions and extracts the components through an instruction analysis module. A generative AI model (e.g., OpenAI's GPT-3) is used for this analysis. An example of a prompt is, "I will understand your instructions and break them down into specific steps. The information required for these steps is listed below."

[0343] 4. Emotion recognition

[0344] The server simultaneously launches an emotion engine to recognize the user's emotional state. The emotion engine utilizes services such as Microsoft's Emotion API to determine the user's emotions based on their input, facial expressions, and tone of voice.

[0345] 5. Generate RPA code

[0346] The RPA robot generation module on the server generates RPA code based on the extracted components and the user's emotional state. This RPA code is a script written in a programming language such as Python to perform a specified action at a given time.

[0347] 6. Emotion-Based Regulation

[0348] The emotion engine can adjust the generation process and display content based on the user's emotions. For example, if the user is feeling stressed, the engine can provide detailed explanations of the generated code or a simplified interface.

[0349] 7. Sending the Product

[0350] The server sends the generated RPA code to the terminal, which then displays the received code to the user in a visually easy-to-verify format.

[0351] 8. Viewing and editing the generated output

[0352] The terminal displays the received RPA code to the user, who can then review the code and edit it if necessary. The editing interface is intuitive and designed to allow users to easily make corrections.

[0353] 9. User confirmation and execution

[0354] The user performs a final check of the generated code, makes any necessary corrections, and then deploys it to the execution environment to check whether it works correctly.

[0355] Specific examples

[0356] Example 1: Backing up files periodically

[0357] User input: The user inputs "Copy files in the working folder to the backup folder every day at 5 PM" into the terminal's input interface.

[0358] Instruction analysis and emotion recognition: The server analyzes the instruction and extracts the components "every day at 5 PM," "work folder," and "copy to backup folder." At the same time, the emotion engine analyzes the user's emotions and recognizes that the user is feeling stressed.

[0359] RPA code generation and emotion-based adjustment: The server generates a backup script based on the extracted information. Since the user is stressed, the emotion engine provides an interface with detailed explanations.

[0360] View and edit the generated script: A terminal will display the generated script to the user, allowing the user to edit the folder path and time, and convenient editing options are provided to reduce frustration.

[0361] Example 2: Frequent email checks

[0362] User input: The user types into the device, "Check email every hour and notify me if new email arrives."

[0363] Instruction analysis and emotion recognition: The server analyzes the components "every hour," "check email," and "notify when new email arrives," and the emotion engine recognizes that the user is excited.

[0364] RPA code generation and emotion-based adjustment: The server generates the email checking script and the emotion engine adjusts it to provide a simple interface.

[0365] View and edit the generated code: The terminal displays the generated code, allowing you to review and make any necessary modifications.

[0366] In this way, by combining the user's natural language instructions with emotion recognition functionality, it is possible to generate RPA robots that are easier to use and meet individual needs. By linking the generative AI model with the emotion engine, it is possible to provide appropriate support according to the user's condition.

[0367] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0368] Step 1:

[0369] The user uses the input interface of the terminal to input instructions in natural language.

[0370] Specific operation: For example, the user enters "Copy files in a specific folder to a backup folder every day at 3 PM."

[0371] Input: Natural language instructions.

[0372] Output: Input data displayed on the terminal.

[0373] Step 2:

[0374] The terminal transmits the input natural language instruction to the server.

[0375] Specific operation: The device sends the user's input data to the server using an HTTP request.

[0376] Input: Data entered into a terminal as natural language instructions.

[0377] Output: An HTTP request containing instructions is sent to the server.

[0378] Step 3:

[0379] The server analyzes the received instruction in an instruction analysis module.

[0380] Specific operation: The server provides the received data to a generative AI model (e.g., GPT-3) as a prompt sentence, which breaks down the instructions into their component parts.

[0381] Input: Natural language instructions sent to the server.

[0382] Output: Parsed component data.

[0383] Step 4:

[0384] The server concurrently launches an emotion engine to recognize the user's emotional state.

[0385] Specific operation: The server calls an emotion engine (e.g., Microsoft Emotion API) and analyzes the user's input, facial expressions, tone of voice, etc.

[0386] Input: Emotional data such as what the user types, facial expressions, and tone of voice.

[0387] Output: User's emotional state (e.g., stressed, excited, relaxed).

[0388] Step 5:

[0389] An instruction parsing module in the server breaks down the natural language instruction into its constituent components.

[0390] Specific operation: The server extracts important components (e.g., "every day at 3 p.m.", "specific folder," "copy to backup folder," etc.) from the output of the generated AI model.

[0391] Input: The data parsed by the generative AI model.

[0392] Output: The extracted component data.

[0393] Step 6:

[0394] The server's RPA robot generation module automatically generates an RPA script based on the extracted elements.

[0395] What it does: The server uses a programming language such as Python code to generate an RPA script based on the extracted components.

[0396] Input: Extracted component data.

[0397] Output: The generated RPA code.

[0398] Step 7:

[0399] The generation process and display content are adjusted based on the user's emotions recognized by the emotion engine.

[0400] Specific behavior: The server adjusts the generated code and interface depending on the user's emotional state (e.g., if the user is feeling stressed, it provides an interface with detailed explanations).

[0401] Input: The user's emotional state.

[0402] Output: The coordinated production process or display content.

[0403] Step 8:

[0404] The server sends the generated RPA code to the terminal.

[0405] Specific operation: The server sends the generated code to the device as an HTTP response.

[0406] Input: The generated RPA code.

[0407] Output: The RPA code sent to the terminal.

[0408] Step 9:

[0409] The terminal displays the received RPA code to the user and provides an interface that allows editing.

[0410] What it does: The device displays the generated code to the user and provides an interface for editing the code.

[0411] Input: The RPA code sent by the server.

[0412] Output: The RPA code that is visible to the user and can be edited.

[0413] Step 10:

[0414] The user checks the generated code, edits it if necessary, and finally deploys it to the execution environment.

[0415] Specific behavior: After the user checks the code and makes any necessary corrections, they press the run button to deploy the script to the execution environment and check its behavior.

[0416] Input: RPA code reviewed and edited by the user.

[0417] Output: RPA code that runs in the execution environment.

[0418] (Application example 2)

[0419] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0420] Existing self-driving vehicles do not adjust navigation or the in-car environment to take the user's emotional state into consideration, resulting in a lack of quality in the riding experience. Furthermore, there is no mechanism for optimizing the vehicle's driving experience based on the user's emotional state, so safety and comfort are not sufficiently ensured. Therefore, there is a need to provide a system that adjusts navigation and the in-car environment according to the user's emotional state.

[0421] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting instructions in natural language, means for analyzing the input instructions and extracting components, means for generating robot process automation code based on the extracted components, means for displaying the generated code to the user, emotion recognition means for recognizing the user's emotional state, means for adjusting the generation process and display content based on the emotional state, and means for adjusting the vehicle navigation and in-vehicle environment based on the user's emotional state. This makes it possible to provide optimal navigation and a comfortable in-vehicle environment according to the user's emotional state.

[0422] A "means for inputting instructions in natural language" is an interface that allows a user to convey commands or wishes to a system using words and sentences that they use on a daily basis.

[0423] "Means for analyzing input instructions and extracting components" refers to technology that understands natural language commands received from a user and breaks them down into specific actions and elements.

[0424] The "means for generating robotic process automation code based on the extracted components" is a function for creating RPA code for automatically executing tasks from the analyzed components.

[0425] "Means for displaying the generated code to the user" refers to a method for visually providing the automatically generated RPA code so that the user can check it.

[0426] "Emotion recognition means for recognizing the user's emotional state" refers to technology that grasps emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0427] "Means for adjusting the generation process and display content based on emotional state" refers to a method for changing and optimizing the information provided by the system and the content of the interface according to the recognized emotion.

[0428] "Means for adjusting vehicle navigation and in-vehicle environment based on the user's emotional state" refers to technology for optimizing the route of an autonomous vehicle and environmental settings such as lighting, music, and air conditioning in the vehicle, taking into account the user's emotional state.

[0429] A specific embodiment of the present invention is a system in which a server, a terminal, a user, and an emotion engine work in cooperation with each other, as will be described in detail below.

[0430] System Configuration

[0431] The system operates inside self-driving vehicles and automatically adjusts navigation and the interior environment based on instructions entered by the user in natural language.

[0432] 1. Server

[0433] The server has the following functions:

[0434] Natural Language Input Analysis: Analyzes natural language instructions sent from the device, extracts their components, and uses generative AI models to translate the instructions into specific actions.

[0435] Emotion Recognition: The car is equipped with an emotion engine that recognizes the user's emotional state. It analyzes the user's facial expressions and tone of voice based on data sent from the in-car camera and microphone to identify emotions.

[0436] RPA Code Generation: Generates robotic process automation (RPA) code based on the extracted components.

[0437] Navigation and environmental adjustment: Generate instructions to adjust the navigation route or the in-car environment (e.g., music, lighting, and air conditioning settings) based on the recognized emotional state.

[0438] 2. Terminal

[0439] The device has the following features:

[0440] User interface: Provides an interface for users to enter instructions in natural language.

[0441] Send instructions: Sends input instructions in natural language to a server, usually using an HTTP request.

[0442] Display of results: Displays the RPA code and adjustment instructions sent from the server to the user. The user can check and edit this information.

[0443] 3. Users

[0444] The user uses the terminal to input instructions in natural language, such as "send a message every day at 3pm" or "play relaxing music."

[0445] Hardware and software used

[0446] Hardware: autonomous vehicle control units, in-car cameras, voice recognition microphones, entertainment systems

[0447] Software: Emotion recognition engines (e.g., OpenVINO, Google® Cloud AI), autonomous driving systems (e.g., Waymo, Tesla AutoPilot), maps and navigation software (e.g., Google Maps API)

[0448] Specific examples

[0449] Specific examples are shown below.

[0450] Example prompt sentence:

[0451] "If the user is feeling stressed, it will suggest a relaxing route and play classical music. It will move at a gentler pace, providing a calmer driving experience."

[0452] The user inputs a natural language command into the in-car device, such as "Play relaxing music every day at 3 p.m." The device sends this command to the server, which not only analyzes the command but also recognizes the user's emotional state using the in-car camera and microphone. If the server detects that the user is feeling stressed, it calculates a relaxing route and instructs the entertainment system to play classical music, allowing the user to enjoy a gentler, more relaxed driving experience.

[0453] The above is a specific embodiment for carrying out the present invention.

[0454] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0455] Step 1:

[0456] The user inputs a command in natural language. The user uses the input interface of the device to input a command such as "Play relaxing music every day at 3 pm."

[0457] Input: Natural language instructions ("Play relaxing music every day at 3pm")

[0458] Output: Natural language instruction data entered into the terminal

[0459] Step 2:

[0460] The device sends input natural language instructions to the server, usually using an HTTP request.

[0461] Input: Natural language instruction data entered into the terminal

[0462] Output: HTTP request sent to the server

[0463] Step 3:

[0464] The server analyzes the received instructions and extracts their components, using a generative AI model to break down the instructions into their components.

[0465] Input: Natural language instructions sent to the server

[0466] Data processing: Generative AI models analyze instructions and extract components

[0467] Output: Component data such as "everyday at 3pm" and "relaxing music"

[0468] Step 4:

[0469] The server activates the emotion engine to recognize the user's emotional state, analyzing the user's facial expressions and tone of voice using data from the in-car camera and microphone.

[0470] Input: In-car camera and microphone data (video and audio)

[0471] Data Computation: Emotion engine analyzes video and audio data to identify emotional states

[0472] Output: User's emotional state data (e.g., stress)

[0473] Step 5:

[0474] The server generates robotic process automation (RPA) code based on the extracted components, which performs a specified action (e.g., playing relaxing music).

[0475] Input: Component data ("every day at 3 pm", "relaxing music")

[0476] Data processing: Automatically generate code using the RPA code generation module

[0477] Output: RPA code (e.g. music playback script)

[0478] Step 6:

[0479] The server adjusts the generation process and display content based on the user's emotional state as recognized by the emotion engine. For example, if the user is feeling stressed, the server provides an interface with detailed explanations.

[0480] Input: Emotional state data (e.g., stress)

[0481] Data processing: Modify and optimize the generation process and display content based on emotional state

[0482] Output: Coordinated production process and display content

[0483] Step 7:

[0484] The server sends the generated RPA code and adjustment details to the terminal, allowing the user to check the generated code and the adjusted display details.

[0485] Input: RPA code, tailored generated processes and display content

[0486] Output: RPA code sent to the terminal and its display content

[0487] Step 8:

[0488] The terminal displays the received RPA code and its contents to the user, who can visually check the code and edit it if necessary.

[0489] Input: RPA code sent from the server and display content

[0490] Output: The RPA code and generated content displayed to the user

[0491] Step 9:

[0492] The user checks the generated code and edits it as necessary. After making a final check, the modified code is deployed to the execution environment and verified to work correctly.

[0493] Input: The displayed RPA code and the generated content

[0494] Output: Reviewed and edited RPA code, and post-deployment verification

[0495] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0496] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0497] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0498] [Second embodiment]

[0499] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0500] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0501] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0502] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0503] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0504] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0505] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0506] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0507] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0508] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0509] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0510] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0511] In order to implement the present invention, the server, the terminal, and the user each play a specific role. Specific embodiments for implementing the present invention are described below.

[0512] overview

[0513] This system automatically generates code for RPA (Robotic Process Automation) robots by inputting instructions in natural language from the user. This system is realized by linking a server and terminals.

[0514] Program Operation

[0515] 1. User input

[0516] The user inputs instructions in natural language through the input interface of the terminal.

[0517] For example: "Back up all files in a specific folder to another folder every Monday at 9 AM."

[0518] 2. Sending instructions

[0519] The terminal transmits the input instructions to the server.

[0520] This allows the user's requests to be communicated throughout the system.

[0521] 3. Parsing the instructions

[0522] The server analyzes the received instruction in an instruction analysis module.

[0523] Specifically, it uses a generative AI model to break down input natural language instructions into their component parts.

[0524] In this example, the elements "every Monday at 9 AM," "specific folder," "all files," and "backup to another folder" are extracted.

[0525] 4. Generate RPA code

[0526] Based on the analysis results, the server generates appropriate code using the RPA robot generation module.

[0527] Based on the extracted elements, an RPA script is automatically generated in a programming language such as Python.

[0528] As an example result, code is generated to back up all files in a specified folder at a specified time.

[0529] 5. Sending the product

[0530] The generated code is sent from the server to the device.

[0531] This is so that the user can verify this code.

[0532] 6. Viewing and editing the generated output

[0533] The terminal displays the received code to the user.

[0534] The user can review the displayed code and edit it if necessary.

[0535] For example, you can manually change the path of the backup folder, change the backup time, etc.

[0536] Specific examples

[0537] Example 1: Backing up files periodically

[0538] User prompts:

[0539] The user enters "Copy files in the working folder to the backup folder every day at 5 p.m." into the terminal's input interface.

[0540] Parsing instructions:

[0541] The server parses the instructions and extracts the components "every day at 5pm", "working folder", and "copy to backup folder".

[0542] RPA code generation:

[0543] The server automatically generates a Python script based on the extracted information, which backs up files in a specified folder every day at 5 PM.

[0544] View and edit the artifact:

[0545] The terminal will display the generated script to the user, who can edit the folder path and time, and give a final confirmation.

[0546] Example 2: Frequent email checks

[0547] User prompts:

[0548] The user types into the terminal, "Check email every hour and notify me if there is new email."

[0549] Parsing instructions:

[0550] The server analyzes the components "every hour," "check email," and "notify if new email arrives."

[0551] RPA code generation:

[0552] The server generates Python code that checks for emails every specified time interval and notifies you if new emails arrive.

[0553] View and edit the artifact:

[0554] The terminal displays the generated code for the user to review and make any necessary corrections.

[0555] This allows users to efficiently create and operate RPA robots without advanced programming knowledge. By using generative AI models, it is possible to perform advanced analysis of natural language and provide accurate automation code.

[0556] The processing flow will be explained below.

[0557] Step 1:

[0558] The user uses the input interface of the device to input instructions in natural language, for example, "Copy files in a specific folder to a backup folder every day at 3 p.m."

[0559] Step 2:

[0560] The device sends the input natural language instructions to the server, usually using an HTTP request.

[0561] Step 3:

[0562] To parse the received instructions, the server invokes an instruction parsing module, which uses a generative AI model to break the instructions down into their component parts.

[0563] Step 4:

[0564] The server's instruction analysis module breaks down the natural language instruction into its components, in this case extracting elements such as "every day at 3 p.m.", "specific folder," and "copy to backup folder."

[0565] Step 5:

[0566] The RPA robot generation module on the server automatically generates RPA scripts, such as Python code, based on the extracted elements. The generated code executes the specified actions at the specified time.

[0567] Step 6:

[0568] The server sends the generated RPA code to the terminal, allowing the user to check the generated code.

[0569] Step 7:

[0570] The terminal displays the received RPA code to the user, who can visually check the code and edit it if necessary.

[0571] Step 8:

[0572] Users can edit the generated code as needed, for example, to change the folder path or backup time.

[0573] Step 9:

[0574] The user performs a final check and deploys the modified code to a runtime environment. Testing may also be performed to ensure the generated code works correctly.

[0575] This series of steps allows users to easily create and operate RPA robots based on instructions entered in natural language.

[0576] Example 1

[0577] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0578] Conventional robotic process automation systems require advanced programming knowledge and are difficult for average users to operate. In addition, it is difficult to input instructions in natural language, which makes it difficult to efficiently analyze input content and generate appropriate automation code.

[0579] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0580] In this invention, the server includes a means for a user to input instructions in natural language, a means for analyzing the input instructions using a generative AI model and extracting components, and a means for generating robotic process automation code based on the extracted components, thereby enabling users without advanced programming knowledge to efficiently execute automation tasks through instructions in natural language, improving the operability and convenience of the entire system.

[0581] "User" refers to an entity that uses the system to input instructions in natural language.

[0582] A "terminal" refers to a device through which a user inputs instructions and transmits the instructions to a server.

[0583] "Server" refers to a central control device that analyzes received instructions, generates robotic process automation code, and sends it to the terminal.

[0584] A "generative AI model" refers to an artificial intelligence algorithm that analyzes input natural language instructions and extracts components.

[0585] "Instructions in natural language" refers to instructions input by the user using a language that humans use on a daily basis (e.g., Japanese or English).

[0586] "Parsing instructions" refers to the process of understanding received natural language instructions and breaking them down into actionable components.

[0587] "Component extraction" refers to the process of extracting the elements needed for an automation task from the parsed natural language instructions.

[0588] "Robotic process automation code" refers to program code for performing automated tasks based on extracted components.

[0589] "Generating code" refers to the process of writing out the extracted components as program code.

[0590] "Sending the code" refers to the process of sending the generated program code from the server to the terminal.

[0591] "Displaying the code" refers to the process of presenting the generated program code to the user in a visible form on the terminal screen.

[0592] "Editing the code" refers to the process by which a user adjusts, adds to, or modifies the content of the generated program code.

[0593] The present invention provides a system for automatically generating robotic process automation (RPA) code based on instructions input by a user in natural language. Specific embodiments for carrying out the invention are described below.

[0594] overview

[0595] The user inputs instructions in natural language through the terminal, and the input instructions are analyzed by the server, which generates the automation code. This system is realized mainly through the cooperation between the terminal and the server.

[0596] Hardware and software used

[0597] Devices: PCs, smartphones, tablets, etc.

[0598] Server: Cloud server, on-premise server

[0599] Generative AI models: AI models used for natural language analysis (e.g., GPT-4)

[0600] Programming language: Python

[0601] Data processing and calculation

[0602] 1. Input interface:

[0603] The user inputs instructions in natural language through the device's input interface. Example: "Copy the files in the working folder to the backup folder every day at 5 p.m."

[0604] The instructions are converted into formatted data (e.g., JSON) and sent to the server.

[0605] 2. Parsing the instructions:

[0606] The server analyzes the received instructions using a generative AI model, and in an example of extracting components, the elements extracted are "every day at 5 p.m.", "working folder," and "copy to backup folder."

[0607] 3. Generate RPA code:

[0608] The server generates a Python script based on the extracted components. Based on the extracted elements, code is generated to back up files in a specific folder at a specified time.

[0609] 4. Send and display the code:

[0610] The generated code is sent from the server to the terminal, which displays the code to the user, who can then view and edit the displayed code.

[0611] Specific examples

[0612] Example 1: Backing up files periodically

[0613] Example of user input:

[0614] The user enters "Copy files in the working folder to the backup folder every day at 5 p.m." into the terminal's input interface.

[0615] Parsing instructions:

[0616] The server extracts the components "every day at 5pm", "working folder", and "copy to backup folder".

[0617] RPA code generation:

[0618] The server automatically generates a Python script based on the extracted information, which backs up files in a specified folder every day at 5 PM.

[0619] Send and display the artifact:

[0620] The terminal will display the generated script to the user, who can edit the folder path and time, and give a final confirmation.

[0621] Example 2: Frequent email checks

[0622] Example of user input:

[0623] The user types into the terminal, "Check email every hour and notify me if there is new email."

[0624] Parsing instructions:

[0625] The server analyzes the components "every hour," "check email," and "notify if new email arrives."

[0626] RPA code generation:

[0627] The server generates Python code that checks for emails every specified time interval and notifies you if new emails arrive.

[0628] Send and display the artifact:

[0629] The terminal displays the generated code for the user to review and make any necessary corrections.

[0630] The system allows users to efficiently execute automation tasks by inputting instructions in natural language without advanced programming knowledge, and uses a generative AI model to intelligently analyze natural language and generate accurate and efficient automation code.

[0631] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0632] Step 1:

[0633] User instruction input

[0634] Input: The user types instructions into the terminal in natural language. Example: "Copy the files in my working folder to the backup folder every day at 5pm."

[0635] Data processing and data calculation: The device converts input instructions into formatted data (e.g., JSON).

[0636] Output: Formatted data is produced.

[0637] How it works: The user opens a dedicated app on a device such as a PC or smartphone and enters instructions. The device converts the input into JSON format data.

[0638] Step 2:

[0639] Sending instructions

[0640] Input: Formatted data (e.g. JSON)

[0641] Data processing and data calculation: The device sends formatted data to the server using an HTTPS request.

[0642] Output: Data is sent to the server.

[0643] Specific behavior: The device sends the input natural language instruction to the server via an HTTPS request. The data is sent as a POST request to the endpoint (e.g., https: / / api.example.com / parse_instruction).

[0644] Step 3:

[0645] Parsing instructions

[0646] Input: Formatted data received by the server (e.g. JSON)

[0647] Data processing and data calculation: The server invokes the generative AI model, parses the natural language instructions, and extracts the components.

[0648] Output: Extracted components (e.g. "Every day at 5 PM", "Working folder", "Copy to backup folder")

[0649] What happens: The server uses a generative AI model (e.g., the GPT-4 model) to analyze the received data. As a result of the analysis, instructions are broken down into their component parts.

[0650] Step 4:

[0651] RPA code generation

[0652] Input: Extracted components

[0653] Data processing and data calculation: The server generates a Python script using the RPA robot generation module.

[0654] Output: Generated Python script

[0655] Specific operation: Based on the parsed components, the server automatically generates a Python script that executes the specified action, such as backing up files in a specific folder at a specified time.

[0656] Step 5:

[0657] Sending artifacts

[0658] Input: Generated Python script

[0659] Data processing and data calculation: The server sends the generated script to the terminal as JSON format data.

[0660] Output: Submitted output (Python script)

[0661] Specific operation: The server sends the generated script to the terminal. The generated script is returned as part of JSON in response to the endpoint (e.g. https: / / api.example.com / get_generated_code).

[0662] Step 6:

[0663] Viewing and editing the generated output

[0664] Input: Submitted product (Python script)

[0665] Data processing and data calculation: The terminal displays the received script to the user and provides an interface for the user to edit it.

[0666] Output: Edited Python script (if needed)

[0667] Specific operation: The terminal displays the received script to the user. The user can check the displayed script and make any necessary modifications. For example, the user can check the contents of the script in the GUI and edit the backup folder path or execution time. When the user saves the edited content, the new instructions are resent to the server, and a new script reflecting the necessary modifications is generated.

[0668] (Application example 1)

[0669] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0670] In recent years, there has been a growing need for inventory management and work efficiency in brick-and-mortar stores. However, traditional inventory management methods require a lot of manual work, which takes time and effort, making efficient business operations difficult. In particular, periodic tasks such as replenishing stock and managing unsold items place a heavy burden on store staff. To solve this problem, a system is needed that can automate a series of inventory management tasks simply by inputting instructions in natural language.

[0671] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0672] In this invention, the server includes a means for inputting instructions in natural language, a means for analyzing the input instructions and extracting components, a means for generating robotic process automation code based on the extracted components, a means for executing the process at a specific time to automate inventory management tasks in a physical store, and a means for displaying the generated code to a user. This allows store clerks to automate inventory management tasks simply by inputting instructions in natural language, thereby significantly reducing time and labor. Furthermore, the input instructions can be accurately analyzed using a generative AI model, enabling an efficient automation process.

[0673] "Natural language" refers to the language that humans use on a daily basis, and is used to communicate intentions through dialogue, etc.

[0674] An "instruction" is a command or order given to perform a particular action or process.

[0675] "Analysis" is the process of breaking down complex data or information to reveal individual elements and their relationships.

[0676] A "component" is an individual part or element that makes up a whole.

[0677] "Robotic process automation code" refers to a program written to automatically execute a specific business process.

[0678] A "brick and mortar store" is a retail establishment that has a physical location and offers goods and services.

[0679] "Inventory management work" refers to the work of managing the receipt, dispatch, storage, and movement of goods and materials.

[0680] The term "specific time" refers to a specific designated time or date.

[0681] A "generative AI model" is an algorithm that uses artificial intelligence to analyze natural language and generate code or data tailored to a specific purpose.

[0682] "User" means any individual or entity that uses the System or Services.

[0683] A "server" is a computer system that processes data and provides services over a network.

[0684] The present invention provides a system for automating inventory management tasks in a physical store. Specific embodiments of the invention will be described below.

[0685] First, a user uses a smartphone application to input instructions in natural language, such as "Replenish the shelves in the cosmetics section every day at 3 p.m." This instruction is then sent from the smartphone to the server.

[0686] The server uses a generative AI model to analyze the received natural language instructions and break them down into their component parts. The software used is a natural language processing tool powered by a generative AI model. By analyzing the instructions, elements such as the time, target section, and task to be performed are extracted.

[0687] The server then generates robotic process automation code based on the extracted components to automate inventory management tasks in physical stores. This code is written in a programming language such as Python and performs tasks such as replenishing specific stock at a specified time. The code generation uses the Python libraries "schedule" and "shutill."

[0688] The generated code is sent from the server to the smartphone and displayed to the user. The user can check the generated code and edit it if necessary. This allows the user to change, for example, the path of the backup folder or the processing time.

[0689] In the present invention, the following specific examples are given.

[0690] For example, if a store associate types in "Restock the shelves in the cosmetics section every day at 3 PM," this instruction is parsed and Python code is generated that will automatically replenish the shelves from a specific folder every day at 3 PM.

[0691] An example of a prompt sentence to input to the generative AI model is as follows:

[0692] "Please parse the instructions entered by the user appropriately and generate the following RPA robot code:

[0693] Instructions: Restock the shelves in the cosmetics section every day at 3pm.

[0694] Required tasks:

[0695] Copy the files in the specified folder to the backup folder.

[0696] In this way, users can implement efficient automated processes tailored to their own business operations without needing advanced programming knowledge, which will significantly reduce the workload of inventory management in physical stores.

[0697] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0698] Step 1:

[0699] A user launches a smartphone application and inputs instructions in natural language, such as "Replenish the shelves in the cosmetics section every day at 3:00 PM." This instruction is entered into the smartphone and sent to the server.

[0700] Input: User's natural language instructions

[0701] Output: Send instructions to the server

[0702] Step 2:

[0703] The server retrieves the received instructions and parses them using a generative AI model. The instruction parsing module analyzes the natural language instructions and breaks them down into their component parts: "every day at 3 p.m.", "cosmetics section," and "replenish."

[0704] Input: User's natural language instructions

[0705] Data processing: natural language analysis and component extraction

[0706] Output: Decomposed components

[0707] Step 3:

[0708] The server generates robotic process automation code based on the analyzed components. Specifically, the generative AI model generates Python code that restocks the shelves in the cosmetics section every day at 3:00 PM.

[0709] Input: Parsed components

[0710] Data transformation: component-based code generation

[0711] Output: Generated Python code

[0712] Step 4:

[0713] The generated Python code is sent from the server to the smartphone, and is displayed on the application screen so that the user can easily check the generated code.

[0714] Input: Generated Python code

[0715] Output: Sends and displays code on smartphone

[0716] Step 5:

[0717] Users can check the generated code on their smartphone and edit it if necessary, for example, to change the backup folder path or processing time.

[0718] Input: Generated Python code

[0719] Output: User can review and edit the code

[0720] Step 6:

[0721] Once the user has finished editing the Python code, it is sent back to the server, which receives it and stores it for use the next time the task is run.

[0722] Input: User-edited Python code

[0723] Output: Send the code to the server and save it

[0724] Step 7:

[0725] The server executes the Python code at the specified time according to the schedule, which then executes the process of replenishing inventory from the specified folder, automating the process.

[0726] Input: Edited Python code

[0727] Output: Automatic execution of inventory replenishment work

[0728] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0729] A specific embodiment of the present invention is a system in which a server, a terminal, a user, and an emotion engine work in cooperation with each other, as will be described in detail below.

[0730] overview

[0731] This system automatically generates code for RPA (Robotic Process Automation) robots based on instructions entered by the user in natural language, and further adjusts the generation process and display content based on the user's emotional state. The system is composed of a server, terminals, users, and an emotion engine.

[0732] Program Operation

[0733] 1. User input

[0734] The user uses the input interface of the device to input instructions in natural language, for example, "Copy files in a specific folder to a backup folder every day at 3 p.m."

[0735] 2. Sending instructions

[0736] The device sends the input natural language instructions to the server, usually using an HTTP request.

[0737] 3. Parsing the instructions

[0738] The server parses the received instructions in an instruction parsing module, which uses a generative AI model to break the instructions down into their component parts.

[0739] 4. Emotion recognition

[0740] In parallel, the server activates an emotion engine to recognize the user's emotional state, which identifies emotions from the user's input, facial expressions, and tone of voice.

[0741] 5. Extracting Components

[0742] The server's instruction analysis module breaks down the natural language instruction into its components, in this case extracting elements such as "every day at 3 p.m.", "specific folder," and "copy to backup folder."

[0743] 6. RPA Code Generation

[0744] The RPA robot generation module on the server automatically generates RPA scripts, such as Python code, based on the extracted elements. The generated code executes the specified actions at the specified time.

[0745] 7. Emotion-Based Adjustment

[0746] The emotion engine can adjust the generation process and display content based on the user's emotions. For example, if the user is feeling stressed, the engine can provide a detailed explanation of the generated code or a simplified interface.

[0747] 8. Sending the Product

[0748] The server sends the generated RPA code to the terminal, allowing the user to check the generated code.

[0749] 9. Viewing and editing the generated output

[0750] The terminal displays the received RPA code to the user, who can visually check the code and edit it if necessary.

[0751] 10. User Verification and Execution

[0752] The user edits the generated code as needed, performs a final check, and deploys the modified code to the execution environment to verify that it works correctly.

[0753] Specific examples

[0754] Example 1: Backing up files periodically

[0755] User prompts:

[0756] The user types into the terminal's input interface, "Copy the files in the working folder to the backup folder every day at 5 p.m."

[0757] Instruction analysis and emotion recognition:

[0758] The server analyzes the instruction and extracts the components "every day at 5 PM," "work folder," and "copy to backup folder." At the same time, the emotion engine analyzes the user's emotions and recognizes that they are feeling stressed.

[0759] RPA code generation and emotion-based adjustment:

[0760] The server generates a backup script based on the extracted information. Because the user is stressed, the emotion engine provides an interface with detailed explanations.

[0761] View and edit the artifact:

[0762] The terminal will display the generated script to the user, and the user can edit the folder path and time, with easy editing options to reduce frustration.

[0763] Example 2: Frequent email checks

[0764] User prompts:

[0765] The user types into the terminal, "Check email every hour and notify me if there is new email."

[0766] Instruction analysis and emotion recognition:

[0767] The server analyzes the components "every hour," "check email," and "notify if new email arrives," and the emotion engine recognizes that the user is excited.

[0768] RPA code generation and emotion-based adjustment:

[0769] The server generates the email checking script and adjusts the emotion engine to provide a simple interface.

[0770] View and edit the artifact:

[0771] The terminal displays the generated code for the user to review and make any necessary corrections.

[0772] In this way, by combining the user's natural language input with emotion recognition functionality, it is possible to create an RPA robot that is easier to use and meets individual needs. By linking the generative AI model with the emotion engine, it is possible to provide appropriate support according to the user's condition.

[0773] The processing flow will be explained below.

[0774] Step 1:

[0775] The user uses the input interface of the terminal to input instructions in natural language, for example, "Copy files in a specific folder to a backup folder every day at 3 p.m."

[0776] Step 2:

[0777] The device sends the input natural language instructions to the server, typically via an HTTP request.

[0778] Step 3:

[0779] The server invokes an instruction parsing module to parse the received instruction, using a generative AI model to break the instruction down into its component parts.

[0780] Step 4:

[0781] The server's instruction analysis module breaks down the natural language instruction into its component parts, in this case extracting elements such as "every day at 3pm," "specific folder," and "copy to backup folder."

[0782] Step 5:

[0783] In parallel, the server activates an emotion engine to recognize the user's emotional state, which identifies emotions from the user's input, facial expressions, and tone of voice.

[0784] Step 6:

[0785] The emotion engine recognizes the user's emotions and feeds that information back to the instruction analysis module. For example, if it determines that the user is feeling stressed, that information is incorporated into the analysis process.

[0786] Step 7:

[0787] The server's RPA robot generation module automatically generates RPA scripts such as Python code based on the extracted elements and emotion information. If the user is feeling stressed, the module adjusts the script to generate detailed comments and easy-to-understand code.

[0788] Step 8:

[0789] The server sends the generated RPA code to the terminal, allowing the user to check the generated code.

[0790] Step 9:

[0791] The terminal displays the received RPA code to the user, who can visually check the code and edit it if necessary.

[0792] Step 10:

[0793] Users can edit the generated code as needed, for example, to change the folder path or backup time. The emotion engine continuously monitors the user's emotions and adjusts the interface as needed.

[0794] Step 11:

[0795] The user performs a final check and deploys the modified code to the execution environment. The server also provides a function to support testing to ensure that the generated code works correctly.

[0796] This specific processing flow allows users to input instructions in natural language and easily create, edit, and execute the RPA robot code generated based on those instructions. Furthermore, the introduction of an emotion engine makes it possible to provide flexible support according to the user's emotional state.

[0797] Example 2

[0798] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0799] Systems that generate robotic process automation (RPA) code are required to allow users to easily input instructions using natural language, and to adapt the generated code and interface to suit the user's emotional state. Conventional systems have struggled to accurately analyze user input and generate appropriate RPA code, often resulting in frustration for users. Another problem is that the generated code is not intuitive and difficult to edit.

[0800] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0801] In this invention, the server includes means for analyzing instructions and extracting components, means for recognizing the user's emotional state, and means for generating robotic process automation code based on the extracted components and the user's emotional state. This allows the user to simply input instructions in natural language, and automatically generates appropriate RPA code that takes the user's emotional state into consideration, providing an intuitive and easy-to-edit interface.

[0802] "Means for inputting instructions in natural language" refers to technology that provides an interface for users to input instructions and commands to a system using natural language, which is a human language.

[0803] The "means for transmitting input instructions to a server" refers to a technology that provides a communication protocol or mechanism for a terminal to send instructions input by a user to a server.

[0804] "Means for analyzing instructions received by the server and extracting components" refers to a technology that analyzes natural language instructions received by the server using an artificial intelligence model or natural language processing technology, and extracts the elements that make up the meaning of the instructions.

[0805] "Means for the server to recognize the user's emotional state" refers to a technology in which the server analyzes the user's input and other indicators (facial expressions, tone of voice, etc.) to determine the user's emotions and mood.

[0806] The "means for generating robotic process automation code based on extracted components and emotional state" is a technology that automatically generates appropriate robotic process automation code based on the components of instructions extracted through analysis and the user's current emotional state.

[0807] "Means for adjusting the generation process and display content based on the user's emotional state" refers to technology that adjusts the code generation process and the screen and interface that the user sees to optimal forms depending on the user's emotional state.

[0808] "Means for transmitting the generated code to the user's terminal" refers to a communication technology for transmitting the robotic process automation code generated by the server back to the user's device.

[0809] "Means for displaying the generated code to the user and making it editable" refers to technology that visually displays the generated code to the user on the terminal and provides an interface that allows the user to edit and modify the code.

[0810] A "generative AI model" is an artificial intelligence model used to analyze natural language instructions entered by a user and extract components, including, for example, machine learning and deep learning algorithms.

[0811] A "prompt" is an input text given to a generative AI model, containing instructions and information that enable the model to understand and parse natural language.

[0812] As a specific embodiment for carrying out the present invention, a system is used in which a server, a terminal, a user, and an emotion engine work in cooperation with each other, as will be described in detail below.

[0813] System configuration

[0814] This system automatically generates robotic process automation (RPA) code based on instructions entered by the user in natural language, and further adjusts the generation process and display content based on the user's emotional state. This system mainly consists of the following elements:

[0815] 1. User input

[0816] The user inputs instructions in natural language using the input interface of the device. The device can be a general computer, smartphone, tablet, or other device. For example, the user might input "Copy files in a specific folder to a backup folder every day at 3:00 PM."

[0817] 2. Sending instructions

[0818] The terminal sends the input instructions in natural language to the server, which receives the instructions using a specified communication protocol (e.g., HTTP).

[0819] 3. Parsing the instructions

[0820] The server analyzes the received instructions and extracts the components through an instruction analysis module. This analysis uses a generative AI model (e.g., OpenAI's GPT-3). An example of a prompt is, "I will understand your instructions and break them down into specific steps. The information required for these steps is listed below."

[0821] 4. Emotion recognition

[0822] The server simultaneously launches an emotion engine to recognize the user's emotional state. The emotion engine utilizes services such as Microsoft's Emotion API to determine emotions from the user's input, facial expressions, and tone of voice.

[0823] 5. Generate RPA code

[0824] The RPA robot generation module on the server generates RPA code based on the extracted components and the user's emotional state. This RPA code is a script written in a programming language such as Python to perform a specified action at a predetermined time.

[0825] 6. Emotion-Based Regulation

[0826] The emotion engine can adjust the generation process and display content based on the user's emotions. For example, if the user is feeling stressed, the engine can provide detailed explanations of the generated code or a simplified interface.

[0827] 7. Sending the Product

[0828] The server sends the generated RPA code to the terminal, which then displays the received code to the user in a visually easy-to-verify format.

[0829] 8. Viewing and editing the generated output

[0830] The terminal displays the received RPA code to the user, who can then review the code and edit it if necessary. The editing interface is intuitive and designed to allow users to easily make corrections.

[0831] 9. User confirmation and execution

[0832] The user performs a final check of the generated code, makes any necessary corrections, and then deploys it to the execution environment to check whether it works correctly.

[0833] Specific examples

[0834] Example 1: Backing up files periodically

[0835] User input: The user inputs "Copy files in the working folder to the backup folder every day at 5 PM" into the terminal's input interface.

[0836] Instruction analysis and emotion recognition: The server analyzes the instruction and extracts the components "every day at 5 PM," "work folder," and "copy to backup folder." At the same time, the emotion engine analyzes the user's emotions and recognizes that the user is feeling stressed.

[0837] RPA code generation and emotion-based adjustment: The server generates a backup script based on the extracted information. Since the user is stressed, the emotion engine provides an interface with detailed explanations.

[0838] View and edit the generated script: A terminal will display the generated script to the user, allowing the user to edit the folder path and time, and convenient editing options are provided to reduce frustration.

[0839] Example 2: Frequent email checks

[0840] User input: The user types into the device, "Check email every hour and notify me if new email arrives."

[0841] Instruction analysis and emotion recognition: The server analyzes the components "every hour," "check email," and "notify when new email arrives," and the emotion engine recognizes that the user is excited.

[0842] RPA code generation and emotion-based adjustment: The server generates the email checking script and the emotion engine adjusts it to provide a simple interface.

[0843] View and edit the generated code: The terminal displays the generated code, allowing you to review and make any necessary modifications.

[0844] In this way, by combining the user's natural language instructions with emotion recognition functionality, it is possible to generate RPA robots that are easier to use and meet individual needs. By linking the generative AI model with the emotion engine, it is possible to provide appropriate support according to the user's condition.

[0845] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0846] Step 1:

[0847] The user uses the input interface of the terminal to input instructions in natural language.

[0848] Specific operation: For example, the user enters "Copy files in a specific folder to a backup folder every day at 3 PM."

[0849] Input: Natural language instructions.

[0850] Output: Input data displayed on the terminal.

[0851] Step 2:

[0852] The terminal transmits the input natural language instruction to the server.

[0853] Specific operation: The device sends the user's input data to the server using an HTTP request.

[0854] Input: Data entered into a terminal as natural language instructions.

[0855] Output: An HTTP request containing instructions is sent to the server.

[0856] Step 3:

[0857] The server analyzes the received instruction in an instruction analysis module.

[0858] Specific operation: The server provides the received data to a generative AI model (e.g., GPT-3) as a prompt sentence, which breaks down the instructions into their component parts.

[0859] Input: Natural language instructions sent to the server.

[0860] Output: Parsed component data.

[0861] Step 4:

[0862] The server concurrently launches an emotion engine to recognize the user's emotional state.

[0863] Specific operation: The server calls an emotion engine (e.g., Microsoft Emotion API) and analyzes the user's input, facial expressions, tone of voice, etc.

[0864] Input: Emotional data such as what the user types, facial expressions, and tone of voice.

[0865] Output: User's emotional state (e.g., stressed, excited, relaxed).

[0866] Step 5:

[0867] An instruction parsing module in the server breaks down the natural language instruction into its constituent components.

[0868] Specific operation: The server extracts important components (e.g., "every day at 3 p.m.", "specific folder," "copy to backup folder," etc.) from the output of the generated AI model.

[0869] Input: The data parsed by the generative AI model.

[0870] Output: The extracted component data.

[0871] Step 6:

[0872] The server's RPA robot generation module automatically generates an RPA script based on the extracted elements.

[0873] What it does: The server uses a programming language such as Python code to generate an RPA script based on the extracted components.

[0874] Input: Extracted component data.

[0875] Output: The generated RPA code.

[0876] Step 7:

[0877] The generation process and display content are adjusted based on the user's emotions recognized by the emotion engine.

[0878] Specific behavior: The server adjusts the generated code and interface depending on the user's emotional state (e.g., if the user is feeling stressed, it provides an interface with detailed explanations).

[0879] Input: The user's emotional state.

[0880] Output: The coordinated production process or display content.

[0881] Step 8:

[0882] The server sends the generated RPA code to the terminal.

[0883] Specific operation: The server sends the generated code to the device as an HTTP response.

[0884] Input: The generated RPA code.

[0885] Output: The RPA code sent to the terminal.

[0886] Step 9:

[0887] The terminal displays the received RPA code to the user and provides an interface that allows editing.

[0888] What it does: The device displays the generated code to the user and provides an interface for editing the code.

[0889] Input: The RPA code sent by the server.

[0890] Output: The RPA code that is visible to the user and can be edited.

[0891] Step 10:

[0892] The user checks the generated code, edits it if necessary, and finally deploys it to the execution environment.

[0893] Specific behavior: After the user checks the code and makes any necessary corrections, they press the run button to deploy the script to the execution environment and check its behavior.

[0894] Input: RPA code reviewed and edited by the user.

[0895] Output: RPA code that runs in the execution environment.

[0896] (Application example 2)

[0897] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0898] Existing self-driving vehicles do not adjust navigation or the in-car environment to take the user's emotional state into consideration, resulting in a lack of quality in the riding experience. Furthermore, there is no mechanism for optimizing the vehicle's driving experience based on the user's emotional state, so safety and comfort are not sufficiently ensured. Therefore, there is a need to provide a system that adjusts navigation and the in-car environment according to the user's emotional state.

[0899] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting instructions in natural language, means for analyzing the input instructions and extracting components, means for generating robot process automation code based on the extracted components, means for displaying the generated code to the user, emotion recognition means for recognizing the user's emotional state, means for adjusting the generation process and display content based on the emotional state, and means for adjusting the vehicle navigation and in-vehicle environment based on the user's emotional state. This makes it possible to provide optimal navigation and a comfortable in-vehicle environment according to the user's emotional state.

[0900] A "means for inputting instructions in natural language" is an interface that allows a user to convey commands or wishes to a system using words and sentences that they use on a daily basis.

[0901] "Means for analyzing input instructions and extracting components" refers to technology that understands natural language commands received from a user and breaks them down into specific actions and elements.

[0902] The "means for generating robotic process automation code based on the extracted components" is a function for creating RPA code for automatically executing tasks from the analyzed components.

[0903] "Means for displaying the generated code to the user" refers to a method for visually providing the automatically generated RPA code so that the user can check it.

[0904] "Emotion recognition means for recognizing the user's emotional state" refers to technology that grasps emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0905] "Means for adjusting the generation process and display content based on emotional state" refers to a method for changing and optimizing the information and interface content provided by the system according to the recognized emotion.

[0906] "Means for adjusting vehicle navigation and in-vehicle environment based on the user's emotional state" refers to technology for optimizing the route traveled by an autonomous vehicle and environmental settings such as lighting, music, and air conditioning in the vehicle, taking into account the user's emotional state.

[0907] A specific embodiment of the present invention is a system in which a server, a terminal, a user, and an emotion engine work in cooperation with each other, as will be described in detail below.

[0908] System Configuration

[0909] The system operates inside self-driving vehicles and automatically adjusts navigation and the interior environment based on instructions entered by the user in natural language.

[0910] 1. Server

[0911] The server has the following functions:

[0912] Natural Language Input Analysis: Analyzes natural language instructions sent from the device, extracts their components, and uses generative AI models to translate the instructions into specific actions.

[0913] Emotion Recognition: The car is equipped with an emotion engine that recognizes the user's emotional state. It analyzes the user's facial expressions and tone of voice based on data sent from the in-car camera and microphone to identify emotions.

[0914] RPA Code Generation: Generates robotic process automation (RPA) code based on the extracted components.

[0915] Navigation and environmental adjustment: Generate instructions to adjust the navigation route or the in-car environment (e.g., music, lighting, and air conditioning settings) based on the perceived emotional state.

[0916] 2. Terminal

[0917] The device has the following features:

[0918] User interface: Provides an interface for users to enter instructions in natural language.

[0919] Send instructions: Sends input instructions in natural language to a server, usually using an HTTP request.

[0920] Display of results: Displays the RPA code and adjustment instructions sent from the server to the user. The user can check and edit this information.

[0921] 3. Users

[0922] The user uses the terminal to input instructions in natural language, such as "send a message every day at 3pm" or "play relaxing music."

[0923] Hardware and software used

[0924] Hardware: autonomous vehicle control units, in-car cameras, voice recognition microphones, entertainment systems

[0925] Software: Emotion recognition engines (e.g., OpenVINO, Google Cloud AI), autonomous driving systems (e.g., Waymo, Tesla AutoPilot), maps and navigation software (e.g., Google Maps API)

[0926] Specific examples

[0927] Specific examples are shown below.

[0928] Example prompt sentence:

[0929] "If the user is feeling stressed, it will suggest a relaxing route and play classical music. It will move at a gentler pace, providing a calmer driving experience."

[0930] The user inputs a natural language command into the in-car device, such as "Play relaxing music every day at 3 p.m." The device sends this command to the server, which not only analyzes the command but also recognizes the user's emotional state using the in-car camera and microphone. If the server determines that the user is feeling stressed, it calculates a relaxing route and instructs the entertainment system to play classical music, allowing the user to enjoy a gentler, more relaxed driving experience.

[0931] The above is a specific embodiment for carrying out the present invention.

[0932] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0933] Step 1:

[0934] The user inputs a command in natural language. The user uses the input interface of the device to input a command such as "Play relaxing music every day at 3 pm."

[0935] Input: Natural language instructions ("Play relaxing music every day at 3pm")

[0936] Output: Natural language instruction data entered into the terminal

[0937] Step 2:

[0938] The device sends input natural language instructions to the server, usually using an HTTP request.

[0939] Input: Natural language instruction data entered into the terminal

[0940] Output: HTTP request sent to the server

[0941] Step 3:

[0942] The server analyzes the received instructions and extracts their components, using a generative AI model to break down the instructions into their components.

[0943] Input: Natural language instructions sent to the server

[0944] Data processing: Generative AI models analyze instructions and extract components

[0945] Output: Component data such as "everyday at 3pm" and "relaxing music"

[0946] Step 4:

[0947] The server activates the emotion engine to recognize the user's emotional state, analyzing the user's facial expressions and tone of voice using data from the in-car camera and microphone.

[0948] Input: In-car camera and microphone data (video and audio)

[0949] Data Computation: Emotion engine analyzes video and audio data to identify emotional states

[0950] Output: User's emotional state data (e.g., stress)

[0951] Step 5:

[0952] The server generates robotic process automation (RPA) code based on the extracted components, which performs a specified action (e.g., playing relaxing music).

[0953] Input: Component data ("every day at 3 pm", "relaxing music")

[0954] Data processing: Automatically generate code using the RPA code generation module

[0955] Output: RPA code (e.g. music playback script)

[0956] Step 6:

[0957] The server adjusts the generation process and display content based on the user's emotional state as recognized by the emotion engine. For example, if the user is feeling stressed, the server provides an interface with detailed explanations.

[0958] Input: Emotional state data (e.g., stress)

[0959] Data processing: Modify and optimize the generation process and display content based on emotional state

[0960] Output: Coordinated production process and display content

[0961] Step 7:

[0962] The server sends the generated RPA code and adjustment details to the terminal, allowing the user to check the generated code and the adjusted display details.

[0963] Input: RPA code, tailored generated processes and display content

[0964] Output: RPA code sent to the terminal and its display content

[0965] Step 8:

[0966] The terminal displays the received RPA code and its contents to the user, who can visually check the code and edit it if necessary.

[0967] Input: RPA code sent from the server and display content

[0968] Output: The RPA code and generated content displayed to the user

[0969] Step 9:

[0970] The user checks the generated code and edits it as necessary. After making a final check, the modified code is deployed to the execution environment and verified to work correctly.

[0971] Input: The displayed RPA code and the generated content

[0972] Output: Reviewed and edited RPA code, and post-deployment verification

[0973] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0974] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0975] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0976] [Third embodiment]

[0977] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0978] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0979] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0980] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0981] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0982] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0983] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0984] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0985] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0986] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0987] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0988] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0989] In order to implement the present invention, the server, the terminal, and the user each play a specific role. Specific embodiments for implementing the present invention are described below.

[0990] overview

[0991] This system automatically generates code for RPA (Robotic Process Automation) robots by inputting instructions in natural language from the user. This system is realized by linking a server and terminals.

[0992] Program Operation

[0993] 1. User input

[0994] The user inputs instructions in natural language through the input interface of the terminal.

[0995] For example: "Back up all files in a specific folder to another folder every Monday at 9 AM."

[0996] 2. Sending instructions

[0997] The terminal transmits the input instructions to the server.

[0998] This allows the user's requests to be communicated throughout the system.

[0999] 3. Parsing the instructions

[1000] The server analyzes the received instruction in an instruction analysis module.

[1001] Specifically, it uses a generative AI model to break down input natural language instructions into their component parts.

[1002] In this example, the elements "every Monday at 9 AM," "specific folder," "all files," and "backup to another folder" are extracted.

[1003] 4. Generate RPA code

[1004] Based on the analysis results, the server generates appropriate code using the RPA robot generation module.

[1005] Based on the extracted elements, an RPA script is automatically generated in a programming language such as Python.

[1006] As an example result, code is generated to back up all files in a specified folder at a specified time.

[1007] 5. Sending the product

[1008] The generated code is sent from the server to the device.

[1009] This is so that the user can verify this code.

[1010] 6. Viewing and editing the generated output

[1011] The terminal displays the received code to the user.

[1012] The user can review the displayed code and edit it if necessary.

[1013] For example, you can manually change the path of the backup folder, change the backup time, etc.

[1014] Specific examples

[1015] Example 1: Backing up files periodically

[1016] User prompts:

[1017] The user enters "Copy files in the working folder to the backup folder every day at 5 p.m." into the terminal's input interface.

[1018] Parsing instructions:

[1019] The server parses the instructions and extracts the components "every day at 5pm", "working folder", and "copy to backup folder".

[1020] RPA code generation:

[1021] The server automatically generates a Python script based on the extracted information, which backs up files in a specified folder every day at 5 PM.

[1022] View and edit the artifact:

[1023] The terminal will display the generated script to the user, who can edit the folder path and time, and give a final confirmation.

[1024] Example 2: Frequent email checks

[1025] User prompts:

[1026] The user types into the terminal, "Check email every hour and notify me if there is new email."

[1027] Parsing instructions:

[1028] The server analyzes the components "every hour," "check email," and "notify if new email arrives."

[1029] RPA code generation:

[1030] The server generates Python code that checks for emails every specified time interval and notifies you if new emails arrive.

[1031] View and edit the artifact:

[1032] The terminal displays the generated code for the user to review and make any necessary corrections.

[1033] This allows users to efficiently create and operate RPA robots without advanced programming knowledge. By using generative AI models, it is possible to perform advanced analysis of natural language and provide accurate automation code.

[1034] The processing flow will be explained below.

[1035] Step 1:

[1036] The user uses the input interface of the device to input instructions in natural language, for example, "Copy files in a specific folder to a backup folder every day at 3 p.m."

[1037] Step 2:

[1038] The device sends the input natural language instructions to the server, usually using an HTTP request.

[1039] Step 3:

[1040] To parse the received instructions, the server invokes an instruction parsing module, which uses a generative AI model to break the instructions down into their component parts.

[1041] Step 4:

[1042] The server's instruction analysis module breaks down the natural language instruction into its components, in this case extracting elements such as "every day at 3 p.m.", "specific folder," and "copy to backup folder."

[1043] Step 5:

[1044] The RPA robot generation module on the server automatically generates RPA scripts, such as Python code, based on the extracted elements. The generated code executes the specified actions at the specified time.

[1045] Step 6:

[1046] The server sends the generated RPA code to the terminal, allowing the user to check the generated code.

[1047] Step 7:

[1048] The terminal displays the received RPA code to the user, who can visually check the code and edit it if necessary.

[1049] Step 8:

[1050] Users can edit the generated code as needed, for example, to change the folder path or backup time.

[1051] Step 9:

[1052] The user performs a final check and deploys the modified code to a runtime environment. Testing may also be performed to ensure the generated code works correctly.

[1053] This series of steps allows users to easily create and operate RPA robots based on instructions entered in natural language.

[1054] Example 1

[1055] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1056] Conventional robotic process automation systems require advanced programming knowledge and are difficult for average users to operate. In addition, it is difficult to input instructions in natural language, which makes it difficult to efficiently analyze input content and generate appropriate automation code.

[1057] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1058] In this invention, the server includes a means for a user to input instructions in natural language, a means for analyzing the input instructions using a generative AI model and extracting components, and a means for generating robotic process automation code based on the extracted components, thereby enabling users without advanced programming knowledge to efficiently execute automation tasks through instructions in natural language, improving the operability and convenience of the entire system.

[1059] "User" refers to an entity that uses the system to input instructions in natural language.

[1060] A "terminal" refers to a device through which a user inputs instructions and transmits the instructions to a server.

[1061] "Server" refers to a central control device that analyzes received instructions, generates robotic process automation code, and sends it to the terminal.

[1062] A "generative AI model" refers to an artificial intelligence algorithm that analyzes input natural language instructions and extracts components.

[1063] "Instructions in natural language" refers to instructions input by the user using a language that humans use on a daily basis (e.g., Japanese or English).

[1064] "Parsing instructions" refers to the process of understanding received natural language instructions and breaking them down into actionable components.

[1065] "Component extraction" refers to the process of extracting the elements needed for an automation task from the parsed natural language instructions.

[1066] "Robotic process automation code" refers to program code for performing automated tasks based on extracted components.

[1067] "Generating code" refers to the process of writing out the extracted components as program code.

[1068] "Sending the code" refers to the process of sending the generated program code from the server to the terminal.

[1069] "Displaying the code" refers to the process of presenting the generated program code to the user in a visible form on the terminal screen.

[1070] "Editing the code" refers to the process by which a user adjusts, adds to, or modifies the content of the generated program code.

[1071] The present invention provides a system for automatically generating robotic process automation (RPA) code based on instructions input by a user in natural language. Specific embodiments for carrying out the invention are described below.

[1072] overview

[1073] The user inputs instructions in natural language through the terminal, and the input instructions are analyzed by the server, which generates the automation code. This system is realized mainly through the cooperation between the terminal and the server.

[1074] Hardware and software used

[1075] Devices: PCs, smartphones, tablets, etc.

[1076] Server: Cloud server, on-premise server

[1077] Generative AI models: AI models used for natural language analysis (e.g., GPT-4)

[1078] Programming language: Python

[1079] Data processing and calculation

[1080] 1. Input interface:

[1081] The user inputs instructions in natural language through the device's input interface. Example: "Copy the files in the working folder to the backup folder every day at 5 p.m."

[1082] The instructions are converted into formatted data (e.g., JSON) and sent to the server.

[1083] 2. Parsing the instructions:

[1084] The server analyzes the received instructions using a generative AI model, and in an example of extracting components, the elements extracted are "every day at 5 p.m.", "working folder," and "copy to backup folder."

[1085] 3. Generate RPA code:

[1086] The server generates a Python script based on the extracted components. Based on the extracted elements, code is generated to back up files in a specific folder at a specified time.

[1087] 4. Send and display the code:

[1088] The generated code is sent from the server to the terminal, which displays the code to the user, who can then view and edit the displayed code.

[1089] Specific examples

[1090] Example 1: Backing up files periodically

[1091] Example of user input:

[1092] The user enters "Copy files in the working folder to the backup folder every day at 5 p.m." into the terminal's input interface.

[1093] Parsing instructions:

[1094] The server extracts the components "every day at 5pm", "working folder", and "copy to backup folder".

[1095] RPA code generation:

[1096] The server automatically generates a Python script based on the extracted information, which backs up files in a specified folder every day at 5 PM.

[1097] Send and display the artifact:

[1098] The terminal will display the generated script to the user, who can edit the folder path and time, and give a final confirmation.

[1099] Example 2: Frequent email checks

[1100] Example of user input:

[1101] The user types into the terminal, "Check email every hour and notify me if there is new email."

[1102] Parsing instructions:

[1103] The server analyzes the components "every hour," "check email," and "notify if new email arrives."

[1104] RPA code generation:

[1105] The server generates Python code that checks for emails every specified time interval and notifies you if new emails arrive.

[1106] Send and display the artifact:

[1107] The terminal displays the generated code for the user to review and make any necessary corrections.

[1108] The system allows users to efficiently execute automation tasks by inputting instructions in natural language without advanced programming knowledge, and uses a generative AI model to intelligently analyze natural language and generate accurate and efficient automation code.

[1109] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1110] Step 1:

[1111] User instruction input

[1112] Input: The user types instructions into the terminal in natural language. Example: "Copy the files in my working folder to the backup folder every day at 5pm."

[1113] Data processing and data calculation: The device converts input instructions into formatted data (e.g., JSON).

[1114] Output: Formatted data is produced.

[1115] How it works: The user opens a dedicated app on a device such as a PC or smartphone and enters instructions. The device converts the input into JSON format data.

[1116] Step 2:

[1117] Sending instructions

[1118] Input: Formatted data (e.g. JSON)

[1119] Data processing and data calculation: The device sends formatted data to the server using an HTTPS request.

[1120] Output: Data is sent to the server.

[1121] Specific behavior: The device sends the input natural language instruction to the server via an HTTPS request. The data is sent as a POST request to the endpoint (e.g., https: / / api.example.com / parse_instruction).

[1122] Step 3:

[1123] Parsing instructions

[1124] Input: Formatted data received by the server (e.g. JSON)

[1125] Data processing and data calculation: The server invokes the generative AI model, parses the natural language instructions, and extracts the components.

[1126] Output: Extracted components (e.g. "Every day at 5 PM", "Working folder", "Copy to backup folder")

[1127] What happens: The server uses a generative AI model (e.g., the GPT-4 model) to analyze the received data. As a result of the analysis, instructions are broken down into their component parts.

[1128] Step 4:

[1129] RPA code generation

[1130] Input: Extracted components

[1131] Data processing and data calculation: The server generates a Python script using the RPA robot generation module.

[1132] Output: Generated Python script

[1133] Specific operation: Based on the parsed components, the server automatically generates a Python script that executes the specified action, such as backing up files in a specific folder at a specified time.

[1134] Step 5:

[1135] Sending artifacts

[1136] Input: Generated Python script

[1137] Data processing and data calculation: The server sends the generated script to the terminal as JSON format data.

[1138] Output: Submitted output (Python script)

[1139] Specific operation: The server sends the generated script to the terminal. The generated script is returned as part of JSON in response to the endpoint (e.g. https: / / api.example.com / get_generated_code).

[1140] Step 6:

[1141] Viewing and editing the generated output

[1142] Input: Submitted product (Python script)

[1143] Data processing and data calculation: The terminal displays the received script to the user and provides an interface for the user to edit it.

[1144] Output: Edited Python script (if needed)

[1145] Specific operation: The terminal displays the received script to the user. The user can check the displayed script and make any necessary modifications. For example, the user can check the contents of the script in the GUI and edit the backup folder path or execution time. When the user saves the edited content, the new instructions are resent to the server, and a new script reflecting the necessary modifications is generated.

[1146] (Application example 1)

[1147] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1148] In recent years, there has been a growing need for inventory management and work efficiency in brick-and-mortar stores. However, traditional inventory management methods require a lot of manual work, which takes time and effort, making efficient business operations difficult. In particular, periodic tasks such as replenishing stock and managing unsold items place a heavy burden on store staff. To solve this problem, a system is needed that can automate a series of inventory management tasks simply by inputting instructions in natural language.

[1149] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1150] In this invention, the server includes a means for inputting instructions in natural language, a means for analyzing the input instructions and extracting components, a means for generating robotic process automation code based on the extracted components, a means for executing the process at a specific time to automate inventory management tasks in a physical store, and a means for displaying the generated code to a user. This allows store clerks to automate inventory management tasks simply by inputting instructions in natural language, thereby significantly reducing time and labor. Furthermore, the input instructions can be accurately analyzed using a generative AI model, enabling an efficient automation process.

[1151] "Natural language" refers to the language that humans use on a daily basis, and is used to communicate intentions through dialogue, etc.

[1152] An "instruction" is a command or order given to perform a particular action or process.

[1153] "Analysis" is the process of breaking down complex data or information to reveal individual elements and their relationships.

[1154] A "component" is an individual part or element that makes up a whole.

[1155] "Robotic process automation code" refers to a program written to automatically execute a specific business process.

[1156] A "brick and mortar store" is a retail establishment that has a physical location and offers goods and services.

[1157] "Inventory management work" refers to the work of managing the receipt, dispatch, storage, and movement of goods and materials.

[1158] The term "specific time" refers to a specific designated time or date.

[1159] A "generative AI model" is an algorithm that uses artificial intelligence to analyze natural language and generate code or data tailored to a specific purpose.

[1160] "User" means any individual or entity that uses the System or Services.

[1161] A "server" is a computer system that processes data and provides services over a network.

[1162] The present invention provides a system for automating inventory management tasks in a physical store. Specific embodiments of the invention will be described below.

[1163] First, a user uses a smartphone application to input instructions in natural language, such as "Replenish the shelves in the cosmetics section every day at 3 p.m." This instruction is then sent from the smartphone to the server.

[1164] The server uses a generative AI model to analyze the received natural language instructions and break them down into their component parts. The software used is a natural language processing tool powered by a generative AI model. By analyzing the instructions, elements such as the time, target section, and task to be performed are extracted.

[1165] The server then generates robotic process automation code based on the extracted components to automate inventory management tasks in physical stores. This code is written in a programming language such as Python and performs tasks such as replenishing specific stock at a specified time. The code generation uses the Python libraries "schedule" and "shutill."

[1166] The generated code is sent from the server to the smartphone and displayed to the user. The user can check the generated code and edit it if necessary. This allows the user to change, for example, the path of the backup folder or the processing time.

[1167] In the present invention, the following specific examples are given.

[1168] For example, if a store associate types in "Restock the shelves in the cosmetics section every day at 3 PM," this instruction is parsed and Python code is generated that will automatically replenish the shelves from a specific folder every day at 3 PM.

[1169] An example of a prompt sentence to input to the generative AI model is as follows:

[1170] "Please parse the instructions entered by the user appropriately and generate the following RPA robot code:

[1171] Instructions: Restock the shelves in the cosmetics section every day at 3pm.

[1172] Required tasks:

[1173] Copy the files in the specified folder to the backup folder.

[1174] In this way, users can implement efficient automated processes tailored to their own business operations without needing advanced programming knowledge, which will significantly reduce the workload of inventory management in physical stores.

[1175] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1176] Step 1:

[1177] A user launches a smartphone application and inputs instructions in natural language, such as "Replenish the shelves in the cosmetics section every day at 3:00 PM." This instruction is entered into the smartphone and sent to the server.

[1178] Input: User's natural language instructions

[1179] Output: Send instructions to the server

[1180] Step 2:

[1181] The server retrieves the received instructions and parses them using a generative AI model. The instruction parsing module analyzes the natural language instructions and breaks them down into their component parts: "every day at 3 p.m.", "cosmetics section," and "replenish."

[1182] Input: User's natural language instructions

[1183] Data processing: natural language analysis and component extraction

[1184] Output: Decomposed components

[1185] Step 3:

[1186] The server generates robotic process automation code based on the analyzed components. Specifically, the generative AI model generates Python code that restocks the shelves in the cosmetics section every day at 3:00 PM.

[1187] Input: Parsed components

[1188] Data transformation: component-based code generation

[1189] Output: Generated Python code

[1190] Step 4:

[1191] The generated Python code is sent from the server to the smartphone, and is displayed on the application screen so that the user can easily check the generated code.

[1192] Input: Generated Python code

[1193] Output: Sends and displays code on smartphone

[1194] Step 5:

[1195] Users can check the generated code on their smartphone and edit it if necessary, for example, to change the backup folder path or processing time.

[1196] Input: Generated Python code

[1197] Output: User can review and edit the code

[1198] Step 6:

[1199] Once the user has finished editing the Python code, it is sent back to the server, which receives it and stores it for use the next time the task is run.

[1200] Input: User-edited Python code

[1201] Output: Send the code to the server and save it

[1202] Step 7:

[1203] The server executes the Python code at the specified time according to the schedule, which then executes the process of replenishing inventory from the specified folder, automating the process.

[1204] Input: Edited Python code

[1205] Output: Automatic execution of inventory replenishment work

[1206] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1207] A specific embodiment of the present invention is a system in which a server, a terminal, a user, and an emotion engine work in cooperation with each other, as will be described in detail below.

[1208] overview

[1209] This system automatically generates code for RPA (Robotic Process Automation) robots based on instructions entered by the user in natural language, and further adjusts the generation process and display content based on the user's emotional state. The system is composed of a server, terminals, users, and an emotion engine.

[1210] Program Operation

[1211] 1. User input

[1212] The user uses the input interface of the device to input instructions in natural language, for example, "Copy files in a specific folder to a backup folder every day at 3 p.m."

[1213] 2. Sending instructions

[1214] The device sends the input natural language instructions to the server, usually using an HTTP request.

[1215] 3. Parsing the instructions

[1216] The server parses the received instructions in an instruction parsing module, which uses a generative AI model to break the instructions down into their component parts.

[1217] 4. Emotion recognition

[1218] In parallel, the server activates an emotion engine to recognize the user's emotional state, which identifies emotions from the user's input, facial expressions, and tone of voice.

[1219] 5. Extracting Components

[1220] The server's instruction analysis module breaks down the natural language instruction into its components, in this case extracting elements such as "every day at 3 p.m.", "specific folder," and "copy to backup folder."

[1221] 6. RPA Code Generation

[1222] The RPA robot generation module on the server automatically generates RPA scripts, such as Python code, based on the extracted elements. The generated code executes the specified actions at the specified time.

[1223] 7. Emotion-Based Adjustment

[1224] The emotion engine can adjust the generation process and display content based on the user's emotions. For example, if the user is feeling stressed, the engine can provide a detailed explanation of the generated code or a simplified interface.

[1225] 8. Sending the Product

[1226] The server sends the generated RPA code to the terminal, allowing the user to check the generated code.

[1227] 9. Viewing and editing the generated output

[1228] The terminal displays the received RPA code to the user, who can visually check the code and edit it if necessary.

[1229] 10. User Verification and Execution

[1230] The user edits the generated code as needed, performs a final check, and deploys the modified code to the execution environment to verify that it works correctly.

[1231] Specific examples

[1232] Example 1: Backing up files periodically

[1233] User prompts:

[1234] The user types into the terminal's input interface, "Copy the files in the working folder to the backup folder every day at 5 p.m."

[1235] Instruction analysis and emotion recognition:

[1236] The server analyzes the instruction and extracts the components "every day at 5 PM," "work folder," and "copy to backup folder." At the same time, the emotion engine analyzes the user's emotions and recognizes that they are feeling stressed.

[1237] RPA code generation and emotion-based adjustment:

[1238] The server generates a backup script based on the extracted information. Because the user is stressed, the emotion engine provides an interface with detailed explanations.

[1239] View and edit the artifact:

[1240] The terminal will display the generated script to the user, and the user can edit the folder path and time, with easy editing options to reduce frustration.

[1241] Example 2: Frequent email checks

[1242] User prompts:

[1243] The user types into the terminal, "Check email every hour and notify me if there is new email."

[1244] Instruction analysis and emotion recognition:

[1245] The server analyzes the components "every hour," "check email," and "notify if new email arrives," and the emotion engine recognizes that the user is excited.

[1246] RPA code generation and emotion-based adjustment:

[1247] The server generates the email checking script and adjusts the emotion engine to provide a simple interface.

[1248] View and edit the artifact:

[1249] The terminal displays the generated code for the user to review and make any necessary corrections.

[1250] In this way, by combining the user's natural language input with emotion recognition functionality, it is possible to create an RPA robot that is easier to use and meets individual needs. By linking the generative AI model with the emotion engine, it is possible to provide appropriate support according to the user's condition.

[1251] The processing flow will be explained below.

[1252] Step 1:

[1253] The user uses the input interface of the terminal to input instructions in natural language, for example, "Copy files in a specific folder to a backup folder every day at 3 p.m."

[1254] Step 2:

[1255] The device sends the input natural language instructions to the server, typically via an HTTP request.

[1256] Step 3:

[1257] The server invokes an instruction parsing module to parse the received instruction, using a generative AI model to break the instruction down into its component parts.

[1258] Step 4:

[1259] The server's instruction analysis module breaks down the natural language instruction into its component parts, in this case extracting elements such as "every day at 3pm," "specific folder," and "copy to backup folder."

[1260] Step 5:

[1261] In parallel, the server activates an emotion engine to recognize the user's emotional state, which identifies emotions from the user's input, facial expressions, and tone of voice.

[1262] Step 6:

[1263] The emotion engine recognizes the user's emotions and feeds that information back to the instruction analysis module. For example, if it determines that the user is feeling stressed, that information is incorporated into the analysis process.

[1264] Step 7:

[1265] The server's RPA robot generation module automatically generates RPA scripts such as Python code based on the extracted elements and emotion information. If the user is feeling stressed, the module adjusts the script to generate detailed comments and easy-to-understand code.

[1266] Step 8:

[1267] The server sends the generated RPA code to the terminal, allowing the user to check the generated code.

[1268] Step 9:

[1269] The terminal displays the received RPA code to the user, who can visually check the code and edit it if necessary.

[1270] Step 10:

[1271] Users can edit the generated code as needed, for example, to change the folder path or backup time. The emotion engine continuously monitors the user's emotions and adjusts the interface as needed.

[1272] Step 11:

[1273] The user performs a final check and deploys the modified code to the execution environment. The server also provides a function to support testing to ensure that the generated code works correctly.

[1274] This specific processing flow allows users to input instructions in natural language and easily create, edit, and execute the RPA robot code generated based on those instructions. Furthermore, the introduction of an emotion engine makes it possible to provide flexible support according to the user's emotional state.

[1275] Example 2

[1276] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1277] Systems that generate robotic process automation (RPA) code are required to allow users to easily input instructions using natural language, and to adapt the generated code and interface to suit the user's emotional state. Conventional systems have struggled to accurately analyze user input and generate appropriate RPA code, often resulting in frustration for users. Another problem is that the generated code is not intuitive and difficult to edit.

[1278] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1279] In this invention, the server includes means for analyzing instructions and extracting components, means for recognizing the user's emotional state, and means for generating robotic process automation code based on the extracted components and the user's emotional state. This allows the user to simply input instructions in natural language, and automatically generates appropriate RPA code that takes the user's emotional state into consideration, providing an intuitive and easy-to-edit interface.

[1280] "Means for inputting instructions in natural language" refers to technology that provides an interface for users to input instructions and commands to a system using natural language, which is a human language.

[1281] The "means for transmitting input instructions to a server" refers to a technology that provides a communication protocol or mechanism for a terminal to send instructions input by a user to a server.

[1282] "Means for analyzing instructions received by the server and extracting components" refers to a technology that analyzes natural language instructions received by the server using an artificial intelligence model or natural language processing technology, and extracts the elements that make up the meaning of the instructions.

[1283] "Means for the server to recognize the user's emotional state" refers to a technology in which the server analyzes the user's input and other indicators (facial expressions, tone of voice, etc.) to determine the user's emotions and mood.

[1284] The "means for generating robotic process automation code based on extracted components and emotional state" is a technology that automatically generates appropriate robotic process automation code based on the components of instructions extracted through analysis and the user's current emotional state.

[1285] "Means for adjusting the generation process and display content based on the user's emotional state" refers to technology that adjusts the code generation process and the screen and interface that the user sees to optimal forms depending on the user's emotional state.

[1286] "Means for transmitting the generated code to the user's terminal" refers to a communication technology for transmitting the robotic process automation code generated by the server back to the user's device.

[1287] "Means for displaying the generated code to the user and making it editable" refers to technology that visually displays the generated code to the user on the terminal and provides an interface that allows the user to edit and modify the code.

[1288] A "generative AI model" is an artificial intelligence model used to analyze natural language instructions entered by a user and extract components, including, for example, machine learning and deep learning algorithms.

[1289] A "prompt" is an input text given to a generative AI model, containing instructions and information that enable the model to understand and parse natural language.

[1290] As a specific embodiment for carrying out the present invention, a system is used in which a server, a terminal, a user, and an emotion engine work in cooperation with each other, as will be described in detail below.

[1291] System configuration

[1292] This system automatically generates robotic process automation (RPA) code based on instructions entered by the user in natural language, and further adjusts the generation process and display content based on the user's emotional state. This system mainly consists of the following elements:

[1293] 1. User input

[1294] The user inputs instructions in natural language using the input interface of the device. The device can be a general computer, smartphone, tablet, or other device. For example, the user might input "Copy files in a specific folder to a backup folder every day at 3:00 PM."

[1295] 2. Sending instructions

[1296] The terminal sends the input instructions in natural language to the server, which receives the instructions using a specified communication protocol (e.g., HTTP).

[1297] 3. Parsing the instructions

[1298] The server analyzes the received instructions and extracts the components through an instruction analysis module. This analysis uses a generative AI model (e.g., OpenAI's GPT-3). An example of a prompt is, "I will understand your instructions and break them down into specific steps. The information required for these steps is listed below."

[1299] 4. Emotion recognition

[1300] The server simultaneously launches an emotion engine to recognize the user's emotional state. The emotion engine utilizes services such as Microsoft's Emotion API to determine emotions from the user's input, facial expressions, and tone of voice.

[1301] 5. Generate RPA code

[1302] The RPA robot generation module on the server generates RPA code based on the extracted components and the user's emotional state. This RPA code is a script written in a programming language such as Python to perform a specified action at a predetermined time.

[1303] 6. Emotion-Based Regulation

[1304] The emotion engine can adjust the generation process and display content based on the user's emotions. For example, if the user is feeling stressed, the engine can provide detailed explanations of the generated code or a simplified interface.

[1305] 7. Sending the Product

[1306] The server sends the generated RPA code to the terminal, which then displays the received code to the user in a visually easy-to-verify format.

[1307] 8. Viewing and editing the generated output

[1308] The terminal displays the received RPA code to the user, who can then review the code and edit it if necessary. The editing interface is intuitive and designed to allow users to easily make corrections.

[1309] 9. User confirmation and execution

[1310] The user performs a final check of the generated code, makes any necessary corrections, and then deploys it to the execution environment to check whether it works correctly.

[1311] Specific examples

[1312] Example 1: Backing up files periodically

[1313] User input: The user inputs "Copy files in the working folder to the backup folder every day at 5 PM" into the terminal's input interface.

[1314] Instruction analysis and emotion recognition: The server analyzes the instruction and extracts the components "every day at 5 PM," "work folder," and "copy to backup folder." At the same time, the emotion engine analyzes the user's emotions and recognizes that the user is feeling stressed.

[1315] RPA code generation and emotion-based adjustment: The server generates a backup script based on the extracted information. Since the user is stressed, the emotion engine provides an interface with detailed explanations.

[1316] View and edit the generated script: A terminal will display the generated script to the user, allowing the user to edit the folder path and time, and convenient editing options are provided to reduce frustration.

[1317] Example 2: Frequent email checks

[1318] User input: The user types into the device, "Check email every hour and notify me if new email arrives."

[1319] Instruction analysis and emotion recognition: The server analyzes the components "every hour," "check email," and "notify when new email arrives," and the emotion engine recognizes that the user is excited.

[1320] RPA code generation and emotion-based adjustment: The server generates the email checking script and the emotion engine adjusts it to provide a simple interface.

[1321] View and edit the generated code: The terminal displays the generated code, allowing you to review and make any necessary modifications.

[1322] In this way, by combining the user's natural language instructions with emotion recognition functionality, it is possible to generate RPA robots that are easier to use and meet individual needs. By linking the generative AI model with the emotion engine, it is possible to provide appropriate support according to the user's condition.

[1323] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1324] Step 1:

[1325] The user uses the input interface of the terminal to input instructions in natural language.

[1326] Specific operation: For example, the user enters "Copy files in a specific folder to a backup folder every day at 3 PM."

[1327] Input: Natural language instructions.

[1328] Output: Input data displayed on the terminal.

[1329] Step 2:

[1330] The terminal transmits the input natural language instruction to the server.

[1331] Specific operation: The device sends the user's input data to the server using an HTTP request.

[1332] Input: Data entered into a terminal as natural language instructions.

[1333] Output: An HTTP request containing instructions is sent to the server.

[1334] Step 3:

[1335] The server analyzes the received instruction in an instruction analysis module.

[1336] Specific operation: The server provides the received data to a generative AI model (e.g., GPT-3) as a prompt sentence, which breaks down the instructions into their component parts.

[1337] Input: Natural language instructions sent to the server.

[1338] Output: Parsed component data.

[1339] Step 4:

[1340] The server concurrently launches an emotion engine to recognize the user's emotional state.

[1341] Specific operation: The server calls an emotion engine (e.g., Microsoft Emotion API) and analyzes the user's input, facial expressions, tone of voice, etc.

[1342] Input: Emotional data such as what the user types, facial expressions, and tone of voice.

[1343] Output: User's emotional state (e.g., stressed, excited, relaxed).

[1344] Step 5:

[1345] An instruction parsing module in the server breaks down the natural language instruction into its constituent components.

[1346] Specific operation: The server extracts important components (e.g., "every day at 3 p.m.", "specific folder," "copy to backup folder," etc.) from the output of the generated AI model.

[1347] Input: The data parsed by the generative AI model.

[1348] Output: The extracted component data.

[1349] Step 6:

[1350] The server's RPA robot generation module automatically generates an RPA script based on the extracted elements.

[1351] What it does: The server uses a programming language such as Python code to generate an RPA script based on the extracted components.

[1352] Input: Extracted component data.

[1353] Output: The generated RPA code.

[1354] Step 7:

[1355] The generation process and display content are adjusted based on the user's emotions recognized by the emotion engine.

[1356] Specific behavior: The server adjusts the generated code and interface depending on the user's emotional state (e.g., if the user is feeling stressed, it provides an interface with detailed explanations).

[1357] Input: The user's emotional state.

[1358] Output: The coordinated production process or display content.

[1359] Step 8:

[1360] The server sends the generated RPA code to the terminal.

[1361] Specific operation: The server sends the generated code to the device as an HTTP response.

[1362] Input: The generated RPA code.

[1363] Output: The RPA code sent to the terminal.

[1364] Step 9:

[1365] The terminal displays the received RPA code to the user and provides an interface that allows editing.

[1366] What it does: The device displays the generated code to the user and provides an interface for editing the code.

[1367] Input: The RPA code sent by the server.

[1368] Output: The RPA code that is visible to the user and can be edited.

[1369] Step 10:

[1370] The user checks the generated code, edits it if necessary, and finally deploys it to the execution environment.

[1371] Specific behavior: After the user checks the code and makes any necessary corrections, they press the run button to deploy the script to the execution environment and check its behavior.

[1372] Input: RPA code reviewed and edited by the user.

[1373] Output: RPA code that runs in the execution environment.

[1374] (Application example 2)

[1375] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1376] Existing self-driving vehicles do not adjust navigation or the in-car environment to take the user's emotional state into consideration, resulting in a lack of quality in the riding experience. Furthermore, there is no mechanism for optimizing the vehicle's driving experience based on the user's emotional state, so safety and comfort are not sufficiently ensured. Therefore, there is a need to provide a system that adjusts navigation and the in-car environment according to the user's emotional state.

[1377] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting instructions in natural language, means for analyzing the input instructions and extracting components, means for generating robot process automation code based on the extracted components, means for displaying the generated code to the user, emotion recognition means for recognizing the user's emotional state, means for adjusting the generation process and display content based on the emotional state, and means for adjusting the vehicle navigation and in-vehicle environment based on the user's emotional state. This makes it possible to provide optimal navigation and a comfortable in-vehicle environment according to the user's emotional state.

[1378] A "means for inputting instructions in natural language" is an interface that allows a user to convey commands or wishes to a system using words and sentences that they use on a daily basis.

[1379] "Means for analyzing input instructions and extracting components" refers to technology that understands natural language commands received from a user and breaks them down into specific actions and elements.

[1380] The "means for generating robotic process automation code based on the extracted components" is a function for creating RPA code for automatically executing tasks from the analyzed components.

[1381] "Means for displaying the generated code to the user" refers to a method for visually providing the automatically generated RPA code so that the user can check it.

[1382] "Emotion recognition means for recognizing the user's emotional state" refers to technology that grasps emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1383] "Means for adjusting the generation process and display content based on emotional state" refers to a method for changing and optimizing the information and interface content provided by the system according to the recognized emotion.

[1384] "Means for adjusting vehicle navigation and in-vehicle environment based on the user's emotional state" refers to technology for optimizing the route traveled by an autonomous vehicle and environmental settings such as lighting, music, and air conditioning in the vehicle, taking into account the user's emotional state.

[1385] A specific embodiment of the present invention is a system in which a server, a terminal, a user, and an emotion engine work in cooperation with each other, as will be described in detail below.

[1386] System Configuration

[1387] The system operates inside self-driving vehicles and automatically adjusts navigation and the interior environment based on instructions entered by the user in natural language.

[1388] 1. Server

[1389] The server has the following functions:

[1390] Natural Language Input Analysis: Analyzes natural language instructions sent from the device, extracts their components, and uses generative AI models to translate the instructions into specific actions.

[1391] Emotion Recognition: The car is equipped with an emotion engine that recognizes the user's emotional state. It analyzes the user's facial expressions and tone of voice based on data sent from the in-car camera and microphone to identify emotions.

[1392] RPA Code Generation: Generates robotic process automation (RPA) code based on the extracted components.

[1393] Navigation and environmental adjustment: Generate instructions to adjust the navigation route or the in-car environment (e.g., music, lighting, and air conditioning settings) based on the perceived emotional state.

[1394] 2. Terminal

[1395] The device has the following features:

[1396] User interface: Provides an interface for users to enter instructions in natural language.

[1397] Send instructions: Sends input instructions in natural language to a server, usually using an HTTP request.

[1398] Display of results: Displays the RPA code and adjustment instructions sent from the server to the user. The user can check and edit this information.

[1399] 3. Users

[1400] The user uses the terminal to input instructions in natural language, such as "send a message every day at 3pm" or "play relaxing music."

[1401] Hardware and software used

[1402] Hardware: autonomous vehicle control units, in-car cameras, voice recognition microphones, entertainment systems

[1403] Software: Emotion recognition engines (e.g., OpenVINO, Google Cloud AI), autonomous driving systems (e.g., Waymo, Tesla AutoPilot), maps and navigation software (e.g., Google Maps API)

[1404] Specific examples

[1405] Specific examples are shown below.

[1406] Example prompt sentence:

[1407] "If the user is feeling stressed, it will suggest a relaxing route and play classical music. It will move at a gentler pace, providing a calmer driving experience."

[1408] The user inputs a natural language command into the in-car device, such as "Play relaxing music every day at 3 p.m." The device sends this command to the server, which not only analyzes the command but also recognizes the user's emotional state using the in-car camera and microphone. If the server determines that the user is feeling stressed, it calculates a relaxing route and instructs the entertainment system to play classical music, allowing the user to enjoy a gentler, more relaxed driving experience.

[1409] The above is a specific embodiment for carrying out the present invention.

[1410] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1411] Step 1:

[1412] The user inputs a command in natural language. The user uses the input interface of the device to input a command such as "Play relaxing music every day at 3 pm."

[1413] Input: Natural language instructions ("Play relaxing music every day at 3pm")

[1414] Output: Natural language instruction data entered into the terminal

[1415] Step 2:

[1416] The device sends input natural language instructions to the server, usually using an HTTP request.

[1417] Input: Natural language instruction data entered into the terminal

[1418] Output: HTTP request sent to the server

[1419] Step 3:

[1420] The server analyzes the received instructions and extracts their components, using a generative AI model to break down the instructions into their components.

[1421] Input: Natural language instructions sent to the server

[1422] Data processing: Generative AI models analyze instructions and extract components

[1423] Output: Component data such as "everyday at 3pm" and "relaxing music"

[1424] Step 4:

[1425] The server activates the emotion engine to recognize the user's emotional state, analyzing the user's facial expressions and tone of voice using data from the in-car camera and microphone.

[1426] Input: In-car camera and microphone data (video and audio)

[1427] Data Computation: Emotion engine analyzes video and audio data to identify emotional states

[1428] Output: User's emotional state data (e.g., stress)

[1429] Step 5:

[1430] The server generates robotic process automation (RPA) code based on the extracted components, which performs a specified action (e.g., playing relaxing music).

[1431] Input: Component data ("every day at 3 pm", "relaxing music")

[1432] Data processing: Automatically generate code using the RPA code generation module

[1433] Output: RPA code (e.g. music playback script)

[1434] Step 6:

[1435] The server adjusts the generation process and display content based on the user's emotional state as recognized by the emotion engine. For example, if the user is feeling stressed, the server provides an interface with detailed explanations.

[1436] Input: Emotional state data (e.g., stress)

[1437] Data processing: Modify and optimize the generation process and display content based on emotional state

[1438] Output: Coordinated production process and display content

[1439] Step 7:

[1440] The server sends the generated RPA code and adjustment details to the terminal, allowing the user to check the generated code and the adjusted display details.

[1441] Input: RPA code, tailored generated processes and display content

[1442] Output: RPA code sent to the terminal and its display content

[1443] Step 8:

[1444] The terminal displays the received RPA code and its contents to the user, who can visually check the code and edit it if necessary.

[1445] Input: RPA code sent from the server and display content

[1446] Output: The RPA code and generated content displayed to the user

[1447] Step 9:

[1448] The user checks the generated code and edits it as necessary. After making a final check, the modified code is deployed to the execution environment and verified to work correctly.

[1449] Input: The displayed RPA code and the generated content

[1450] Output: Reviewed and edited RPA code, and post-deployment verification

[1451] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1452] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1453] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1454] [Fourth embodiment]

[1455] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1456] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1457] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1458] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1459] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1460] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1461] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1462] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1463] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1464] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1465] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1466] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1467] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1468] In order to implement the present invention, the server, the terminal, and the user each play a specific role. Specific embodiments for implementing the present invention are described below.

[1469] overview

[1470] This system automatically generates code for RPA (Robotic Process Automation) robots by inputting instructions in natural language from the user. This system is realized by linking a server and terminals.

[1471] Program Operation

[1472] 1. User input

[1473] The user inputs instructions in natural language through the input interface of the terminal.

[1474] For example: "Back up all files in a specific folder to another folder every Monday at 9 AM."

[1475] 2. Sending instructions

[1476] The terminal transmits the input instructions to the server.

[1477] This allows the user's requests to be communicated throughout the system.

[1478] 3. Parsing the instructions

[1479] The server analyzes the received instruction in an instruction analysis module.

[1480] Specifically, it uses a generative AI model to break down input natural language instructions into their component parts.

[1481] In this example, the elements "every Monday at 9 AM," "specific folder," "all files," and "backup to another folder" are extracted.

[1482] 4. Generate RPA code

[1483] Based on the analysis results, the server generates appropriate code using the RPA robot generation module.

[1484] Based on the extracted elements, an RPA script is automatically generated in a programming language such as Python.

[1485] As an example result, code is generated to back up all files in a specified folder at a specified time.

[1486] 5. Sending the product

[1487] The generated code is sent from the server to the device.

[1488] This is so that the user can verify this code.

[1489] 6. Viewing and editing the generated output

[1490] The terminal displays the received code to the user.

[1491] The user can review the displayed code and edit it if necessary.

[1492] For example, you can manually change the path of the backup folder, change the backup time, etc.

[1493] Specific examples

[1494] Example 1: Backing up files periodically

[1495] User prompts:

[1496] The user enters "Copy files in the working folder to the backup folder every day at 5 p.m." into the terminal's input interface.

[1497] Parsing instructions:

[1498] The server parses the instructions and extracts the components "every day at 5pm", "working folder", and "copy to backup folder".

[1499] RPA code generation:

[1500] The server automatically generates a Python script based on the extracted information, which backs up files in a specified folder every day at 5 PM.

[1501] View and edit the artifact:

[1502] The terminal will display the generated script to the user, who can edit the folder path and time, and give a final confirmation.

[1503] Example 2: Frequent email checks

[1504] User prompts:

[1505] The user types into the terminal, "Check email every hour and notify me if there is new email."

[1506] Parsing instructions:

[1507] The server analyzes the components "every hour," "check email," and "notify if new email arrives."

[1508] RPA code generation:

[1509] The server generates Python code that checks for emails every specified time interval and notifies you if new emails arrive.

[1510] View and edit the artifact:

[1511] The terminal displays the generated code for the user to review and make any necessary corrections.

[1512] This allows users to efficiently create and operate RPA robots without advanced programming knowledge. By using generative AI models, it is possible to perform advanced analysis of natural language and provide accurate automation code.

[1513] The processing flow will be explained below.

[1514] Step 1:

[1515] The user uses the input interface of the device to input instructions in natural language, for example, "Copy files in a specific folder to a backup folder every day at 3 p.m."

[1516] Step 2:

[1517] The device sends the input natural language instructions to the server, usually using an HTTP request.

[1518] Step 3:

[1519] To parse the received instructions, the server invokes an instruction parsing module, which uses a generative AI model to break the instructions down into their component parts.

[1520] Step 4:

[1521] The server's instruction analysis module breaks down the natural language instruction into its components, in this case extracting elements such as "every day at 3 p.m.", "specific folder," and "copy to backup folder."

[1522] Step 5:

[1523] The RPA robot generation module on the server automatically generates RPA scripts, such as Python code, based on the extracted elements. The generated code executes the specified actions at the specified time.

[1524] Step 6:

[1525] The server sends the generated RPA code to the terminal, allowing the user to check the generated code.

[1526] Step 7:

[1527] The terminal displays the received RPA code to the user, who can visually check the code and edit it if necessary.

[1528] Step 8:

[1529] Users can edit the generated code as needed, for example, to change the folder path or backup time.

[1530] Step 9:

[1531] The user performs a final check and deploys the modified code to a runtime environment. Testing may also be performed to ensure the generated code works correctly.

[1532] This series of steps allows users to easily create and operate RPA robots based on instructions entered in natural language.

[1533] Example 1

[1534] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1535] Conventional robotic process automation systems require advanced programming knowledge and are difficult for average users to operate. In addition, it is difficult to input instructions in natural language, which makes it difficult to efficiently analyze input content and generate appropriate automation code.

[1536] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1537] In this invention, the server includes a means for a user to input instructions in natural language, a means for analyzing the input instructions using a generative AI model and extracting components, and a means for generating robotic process automation code based on the extracted components, thereby enabling users without advanced programming knowledge to efficiently execute automation tasks through instructions in natural language, improving the operability and convenience of the entire system.

[1538] "User" refers to an entity that uses the system to input instructions in natural language.

[1539] A "terminal" refers to a device through which a user inputs instructions and transmits the instructions to a server.

[1540] "Server" refers to a central control device that analyzes received instructions, generates robotic process automation code, and sends it to the terminal.

[1541] A "generative AI model" refers to an artificial intelligence algorithm that analyzes input natural language instructions and extracts components.

[1542] "Instructions in natural language" refers to instructions input by the user using a language that humans use on a daily basis (e.g., Japanese or English).

[1543] "Parsing instructions" refers to the process of understanding received natural language instructions and breaking them down into actionable components.

[1544] "Component extraction" refers to the process of extracting the elements needed for an automation task from the parsed natural language instructions.

[1545] "Robotic process automation code" refers to program code for performing automated tasks based on extracted components.

[1546] "Generating code" refers to the process of writing out the extracted components as program code.

[1547] "Sending the code" refers to the process of sending the generated program code from the server to the terminal.

[1548] "Displaying the code" refers to the process of presenting the generated program code to the user in a visible form on the terminal screen.

[1549] "Editing the code" refers to the process by which a user adjusts, adds to, or modifies the content of the generated program code.

[1550] The present invention provides a system for automatically generating robotic process automation (RPA) code based on instructions input by a user in natural language. Specific embodiments for carrying out the invention are described below.

[1551] overview

[1552] The user inputs instructions in natural language through the terminal, and the input instructions are analyzed by the server, which generates the automation code. This system is realized mainly through the cooperation between the terminal and the server.

[1553] Hardware and software used

[1554] Devices: PCs, smartphones, tablets, etc.

[1555] Server: Cloud server, on-premise server

[1556] Generative AI models: AI models used for natural language analysis (e.g., GPT-4)

[1557] Programming language: Python

[1558] Data processing and calculation

[1559] 1. Input interface:

[1560] The user inputs instructions in natural language through the device's input interface. Example: "Copy the files in the working folder to the backup folder every day at 5 p.m."

[1561] The instructions are converted into formatted data (e.g., JSON) and sent to the server.

[1562] 2. Parsing the instructions:

[1563] The server analyzes the received instructions using a generative AI model, and in an example of extracting components, the elements extracted are "every day at 5 p.m.", "working folder," and "copy to backup folder."

[1564] 3. Generate RPA code:

[1565] The server generates a Python script based on the extracted components. Based on the extracted elements, code is generated to back up files in a specific folder at a specified time.

[1566] 4. Send and display the code:

[1567] The generated code is sent from the server to the terminal, which displays the code to the user, who can then view and edit the displayed code.

[1568] Specific examples

[1569] Example 1: Backing up files periodically

[1570] Example of user input:

[1571] The user enters "Copy files in the working folder to the backup folder every day at 5 p.m." into the terminal's input interface.

[1572] Parsing instructions:

[1573] The server extracts the components "every day at 5pm", "working folder", and "copy to backup folder".

[1574] RPA code generation:

[1575] The server automatically generates a Python script based on the extracted information, which backs up files in a specified folder every day at 5 PM.

[1576] Send and display the artifact:

[1577] The terminal will display the generated script to the user, who can edit the folder path and time, and give a final confirmation.

[1578] Example 2: Frequent email checks

[1579] Example of user input:

[1580] The user types into the terminal, "Check email every hour and notify me if there is new email."

[1581] Parsing instructions:

[1582] The server analyzes the components "every hour," "check email," and "notify if new email arrives."

[1583] RPA code generation:

[1584] The server generates Python code that checks for emails every specified time interval and notifies you if new emails arrive.

[1585] Send and display the artifact:

[1586] The terminal displays the generated code for the user to review and make any necessary corrections.

[1587] The system allows users to efficiently execute automation tasks by inputting instructions in natural language without advanced programming knowledge, and uses a generative AI model to intelligently analyze natural language and generate accurate and efficient automation code.

[1588] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1589] Step 1:

[1590] User instruction input

[1591] Input: The user types instructions into the terminal in natural language. Example: "Copy the files in my working folder to the backup folder every day at 5pm."

[1592] Data processing and data calculation: The device converts input instructions into formatted data (e.g., JSON).

[1593] Output: Formatted data is produced.

[1594] How it works: The user opens a dedicated app on a device such as a PC or smartphone and enters instructions. The device converts the input into JSON format data.

[1595] Step 2:

[1596] Sending instructions

[1597] Input: Formatted data (e.g. JSON)

[1598] Data processing and data calculation: The device sends formatted data to the server using an HTTPS request.

[1599] Output: Data is sent to the server.

[1600] Specific behavior: The device sends the input natural language instruction to the server via an HTTPS request. The data is sent as a POST request to the endpoint (e.g., https: / / api.example.com / parse_instruction).

[1601] Step 3:

[1602] Parsing instructions

[1603] Input: Formatted data received by the server (e.g. JSON)

[1604] Data processing and data calculation: The server invokes the generative AI model, parses the natural language instructions, and extracts the components.

[1605] Output: Extracted components (e.g. "Every day at 5 PM", "Working folder", "Copy to backup folder")

[1606] What happens: The server uses a generative AI model (e.g., the GPT-4 model) to analyze the received data. As a result of the analysis, instructions are broken down into their component parts.

[1607] Step 4:

[1608] RPA code generation

[1609] Input: Extracted components

[1610] Data processing and data calculation: The server generates a Python script using the RPA robot generation module.

[1611] Output: Generated Python script

[1612] Specific operation: Based on the parsed components, the server automatically generates a Python script that executes the specified action, such as backing up files in a specific folder at a specified time.

[1613] Step 5:

[1614] Sending artifacts

[1615] Input: Generated Python script

[1616] Data processing and data calculation: The server sends the generated script to the terminal as JSON format data.

[1617] Output: Submitted output (Python script)

[1618] Specific operation: The server sends the generated script to the terminal. The generated script is returned as part of JSON in response to the endpoint (e.g. https: / / api.example.com / get_generated_code).

[1619] Step 6:

[1620] Viewing and editing the generated output

[1621] Input: Submitted product (Python script)

[1622] Data processing and data calculation: The terminal displays the received script to the user and provides an interface for the user to edit it.

[1623] Output: Edited Python script (if needed)

[1624] Specific operation: The terminal displays the received script to the user. The user can check the displayed script and make any necessary modifications. For example, the user can check the contents of the script in the GUI and edit the backup folder path or execution time. When the user saves the edited content, the new instructions are resent to the server, and a new script reflecting the necessary modifications is generated.

[1625] (Application example 1)

[1626] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1627] In recent years, there has been a growing need for inventory management and work efficiency in brick-and-mortar stores. However, traditional inventory management methods require a lot of manual work, which takes time and effort, making efficient business operations difficult. In particular, periodic tasks such as replenishing stock and managing unsold items place a heavy burden on store staff. To solve this problem, a system is needed that can automate a series of inventory management tasks simply by inputting instructions in natural language.

[1628] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1629] In this invention, the server includes a means for inputting instructions in natural language, a means for analyzing the input instructions and extracting components, a means for generating robotic process automation code based on the extracted components, a means for executing the process at a specific time to automate inventory management tasks in a physical store, and a means for displaying the generated code to a user. This allows store clerks to automate inventory management tasks simply by inputting instructions in natural language, thereby significantly reducing time and labor. Furthermore, the input instructions can be accurately analyzed using a generative AI model, enabling an efficient automation process.

[1630] "Natural language" refers to the language that humans use on a daily basis, and is used to communicate intentions through dialogue, etc.

[1631] An "instruction" is a command or order given to perform a particular action or process.

[1632] "Analysis" is the process of breaking down complex data or information to reveal individual elements and their relationships.

[1633] A "component" is an individual part or element that makes up a whole.

[1634] "Robotic process automation code" refers to a program written to automatically execute a specific business process.

[1635] A "brick and mortar store" is a retail establishment that has a physical location and offers goods and services.

[1636] "Inventory management work" refers to the work of managing the receipt, dispatch, storage, and movement of goods and materials.

[1637] The term "specific time" refers to a specific designated time or date.

[1638] A "generative AI model" is an algorithm that uses artificial intelligence to analyze natural language and generate code or data tailored to a specific purpose.

[1639] "User" means any individual or entity that uses the System or Services.

[1640] A "server" is a computer system that processes data and provides services over a network.

[1641] The present invention provides a system for automating inventory management tasks in a physical store. Specific embodiments of the invention will be described below.

[1642] First, a user uses a smartphone application to input instructions in natural language, such as "Replenish the shelves in the cosmetics section every day at 3 p.m." This instruction is then sent from the smartphone to the server.

[1643] The server uses a generative AI model to analyze the received natural language instructions and break them down into their component parts. The software used is a natural language processing tool powered by a generative AI model. By analyzing the instructions, elements such as the time, target section, and task to be performed are extracted.

[1644] The server then generates robotic process automation code based on the extracted components to automate inventory management tasks in physical stores. This code is written in a programming language such as Python and performs tasks such as replenishing specific stock at a specified time. The code generation uses the Python libraries "schedule" and "shutill."

[1645] The generated code is sent from the server to the smartphone and displayed to the user. The user can check the generated code and edit it if necessary. This allows the user to change, for example, the path of the backup folder or the processing time.

[1646] In the present invention, the following specific examples are given.

[1647] For example, if a store associate types in "Restock the shelves in the cosmetics section every day at 3 PM," this instruction is parsed and Python code is generated that will automatically replenish the shelves from a specific folder every day at 3 PM.

[1648] An example of a prompt sentence to input to the generative AI model is as follows:

[1649] "Please parse the instructions entered by the user appropriately and generate the following RPA robot code:

[1650] Instructions: Restock the shelves in the cosmetics section every day at 3pm.

[1651] Required tasks:

[1652] Copy the files in the specified folder to the backup folder.

[1653] In this way, users can implement efficient automated processes tailored to their own business operations without needing advanced programming knowledge, which will significantly reduce the workload of inventory management in physical stores.

[1654] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1655] Step 1:

[1656] A user launches a smartphone application and inputs instructions in natural language, such as "Replenish the shelves in the cosmetics section every day at 3:00 PM." This instruction is entered into the smartphone and sent to the server.

[1657] Input: User's natural language instructions

[1658] Output: Send instructions to the server

[1659] Step 2:

[1660] The server retrieves the received instructions and parses them using a generative AI model. The instruction parsing module analyzes the natural language instructions and breaks them down into their component parts: "every day at 3 p.m.", "cosmetics section," and "replenish."

[1661] Input: User's natural language instructions

[1662] Data processing: natural language analysis and component extraction

[1663] Output: Decomposed components

[1664] Step 3:

[1665] The server generates robotic process automation code based on the analyzed components. Specifically, the generative AI model generates Python code that restocks the shelves in the cosmetics section every day at 3:00 PM.

[1666] Input: Parsed components

[1667] Data transformation: component-based code generation

[1668] Output: Generated Python code

[1669] Step 4:

[1670] The generated Python code is sent from the server to the smartphone, and is displayed on the application screen so that the user can easily check the generated code.

[1671] Input: Generated Python code

[1672] Output: Sends and displays code on smartphone

[1673] Step 5:

[1674] Users can check the generated code on their smartphone and edit it if necessary, for example, to change the backup folder path or processing time.

[1675] Input: Generated Python code

[1676] Output: User can review and edit the code

[1677] Step 6:

[1678] Once the user has finished editing the Python code, it is sent back to the server, which receives it and stores it for use the next time the task is run.

[1679] Input: User-edited Python code

[1680] Output: Send the code to the server and save it

[1681] Step 7:

[1682] The server executes the Python code at the specified time according to the schedule, which then executes the process of replenishing inventory from the specified folder, automating the process.

[1683] Input: Edited Python code

[1684] Output: Automatic execution of inventory replenishment work

[1685] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1686] A specific embodiment of the present invention is a system in which a server, a terminal, a user, and an emotion engine work in cooperation with each other, as will be described in detail below.

[1687] overview

[1688] This system automatically generates code for RPA (Robotic Process Automation) robots based on instructions entered by the user in natural language, and further adjusts the generation process and display content based on the user's emotional state. The system is composed of a server, terminals, users, and an emotion engine.

[1689] Program Operation

[1690] 1. User input

[1691] The user uses the input interface of the device to input instructions in natural language, for example, "Copy files in a specific folder to a backup folder every day at 3 p.m."

[1692] 2. Sending instructions

[1693] The device sends the input natural language instructions to the server, usually using an HTTP request.

[1694] 3. Parsing the instructions

[1695] The server parses the received instructions in an instruction parsing module, which uses a generative AI model to break the instructions down into their component parts.

[1696] 4. Emotion recognition

[1697] In parallel, the server activates an emotion engine to recognize the user's emotional state, which identifies emotions from the user's input, facial expressions, and tone of voice.

[1698] 5. Extracting Components

[1699] The server's instruction analysis module breaks down the natural language instruction into its components, in this case extracting elements such as "every day at 3 p.m.", "specific folder," and "copy to backup folder."

[1700] 6. RPA Code Generation

[1701] The RPA robot generation module on the server automatically generates RPA scripts, such as Python code, based on the extracted elements. The generated code executes the specified actions at the specified time.

[1702] 7. Emotion-Based Adjustment

[1703] The emotion engine can adjust the generation process and display content based on the user's emotions. For example, if the user is feeling stressed, the engine can provide a detailed explanation of the generated code or a simplified interface.

[1704] 8. Sending the Product

[1705] The server sends the generated RPA code to the terminal, allowing the user to check the generated code.

[1706] 9. Viewing and editing the generated output

[1707] The terminal displays the received RPA code to the user, who can visually check the code and edit it if necessary.

[1708] 10. User Verification and Execution

[1709] The user edits the generated code as needed, performs a final check, and deploys the modified code to the execution environment to verify that it works correctly.

[1710] Specific examples

[1711] Example 1: Backing up files periodically

[1712] User prompts:

[1713] The user types into the terminal's input interface, "Copy the files in the working folder to the backup folder every day at 5 p.m."

[1714] Instruction analysis and emotion recognition:

[1715] The server analyzes the instruction and extracts the components "every day at 5 PM," "work folder," and "copy to backup folder." At the same time, the emotion engine analyzes the user's emotions and recognizes that they are feeling stressed.

[1716] RPA code generation and emotion-based adjustment:

[1717] The server generates a backup script based on the extracted information. Because the user is stressed, the emotion engine provides an interface with detailed explanations.

[1718] View and edit the artifact:

[1719] The terminal will display the generated script to the user, and the user can edit the folder path and time, with easy editing options to reduce frustration.

[1720] Example 2: Frequent email checks

[1721] User prompts:

[1722] The user types into the terminal, "Check email every hour and notify me if there is new email."

[1723] Instruction analysis and emotion recognition:

[1724] The server analyzes the components "every hour," "check email," and "notify if new email arrives," and the emotion engine recognizes that the user is excited.

[1725] RPA code generation and emotion-based adjustment:

[1726] The server generates the email checking script and adjusts the emotion engine to provide a simple interface.

[1727] View and edit the artifact:

[1728] The terminal displays the generated code for the user to review and make any necessary corrections.

[1729] In this way, by combining the user's natural language input with emotion recognition functionality, it is possible to create an RPA robot that is easier to use and meets individual needs. By linking the generative AI model with the emotion engine, it is possible to provide appropriate support according to the user's condition.

[1730] The processing flow will be explained below.

[1731] Step 1:

[1732] The user uses the input interface of the terminal to input instructions in natural language, for example, "Copy files in a specific folder to a backup folder every day at 3 p.m."

[1733] Step 2:

[1734] The device sends the input natural language instructions to the server, typically via an HTTP request.

[1735] Step 3:

[1736] The server invokes an instruction parsing module to parse the received instruction, using a generative AI model to break the instruction down into its component parts.

[1737] Step 4:

[1738] The server's instruction analysis module breaks down the natural language instruction into its component parts, in this case extracting elements such as "every day at 3pm," "specific folder," and "copy to backup folder."

[1739] Step 5:

[1740] In parallel, the server activates an emotion engine to recognize the user's emotional state, which identifies emotions from the user's input, facial expressions, and tone of voice.

[1741] Step 6:

[1742] The emotion engine recognizes the user's emotions and feeds that information back to the instruction analysis module. For example, if it determines that the user is feeling stressed, that information is incorporated into the analysis process.

[1743] Step 7:

[1744] The server's RPA robot generation module automatically generates RPA scripts such as Python code based on the extracted elements and emotion information. If the user is feeling stressed, the module adjusts the script to generate detailed comments and easy-to-understand code.

[1745] Step 8:

[1746] The server sends the generated RPA code to the terminal, allowing the user to check the generated code.

[1747] Step 9:

[1748] The terminal displays the received RPA code to the user, who can visually check the code and edit it if necessary.

[1749] Step 10:

[1750] Users can edit the generated code as needed, for example, to change the folder path or backup time. The emotion engine continuously monitors the user's emotions and adjusts the interface as needed.

[1751] Step 11:

[1752] The user performs a final check and deploys the modified code to the execution environment. The server also provides a function to support testing to ensure that the generated code works correctly.

[1753] This specific processing flow allows users to input instructions in natural language and easily create, edit, and execute the RPA robot code generated based on those instructions. Furthermore, the introduction of an emotion engine makes it possible to provide flexible support according to the user's emotional state.

[1754] Example 2

[1755] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1756] Systems that generate robotic process automation (RPA) code are required to allow users to easily input instructions using natural language, and to adapt the generated code and interface to suit the user's emotional state. Conventional systems have struggled to accurately analyze user input and generate appropriate RPA code, often resulting in frustration for users. Another problem is that the generated code is not intuitive and difficult to edit.

[1757] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1758] In this invention, the server includes means for analyzing instructions and extracting components, means for recognizing the user's emotional state, and means for generating robotic process automation code based on the extracted components and the user's emotional state. This allows the user to simply input instructions in natural language, and automatically generates appropriate RPA code that takes the user's emotional state into consideration, providing an intuitive and easy-to-edit interface.

[1759] "Means for inputting instructions in natural language" refers to technology that provides an interface for users to input instructions and commands to a system using natural language, which is a human language.

[1760] The "means for transmitting input instructions to a server" refers to a technology that provides a communication protocol or mechanism for a terminal to send instructions input by a user to a server.

[1761] "Means for analyzing instructions received by the server and extracting components" refers to a technology that analyzes natural language instructions received by the server using an artificial intelligence model or natural language processing technology, and extracts the elements that make up the meaning of the instructions.

[1762] "Means for the server to recognize the user's emotional state" refers to a technology in which the server analyzes the user's input and other indicators (facial expressions, tone of voice, etc.) to determine the user's emotions and mood.

[1763] The "means for generating robotic process automation code based on extracted components and emotional state" is a technology that automatically generates appropriate robotic process automation code based on the components of instructions extracted through analysis and the user's current emotional state.

[1764] "Means for adjusting the generation process and display content based on the user's emotional state" refers to technology that adjusts the code generation process and the screen and interface that the user sees to optimal forms depending on the user's emotional state.

[1765] "Means for transmitting the generated code to the user's terminal" refers to a communication technology for transmitting the robotic process automation code generated by the server back to the user's device.

[1766] "Means for displaying the generated code to the user and making it editable" refers to technology that visually displays the generated code to the user on the terminal and provides an interface that allows the user to edit and modify the code.

[1767] A "generative AI model" is an artificial intelligence model used to analyze natural language instructions entered by a user and extract components, including, for example, machine learning and deep learning algorithms.

[1768] A "prompt" is an input text given to a generative AI model, containing instructions and information that enable the model to understand and parse natural language.

[1769] As a specific embodiment for carrying out the present invention, a system is used in which a server, a terminal, a user, and an emotion engine work in cooperation with each other, as will be described in detail below.

[1770] System configuration

[1771] This system automatically generates robotic process automation (RPA) code based on instructions entered by the user in natural language, and further adjusts the generation process and display content based on the user's emotional state. This system mainly consists of the following elements:

[1772] 1. User input

[1773] The user inputs instructions in natural language using the input interface of the device. The device can be a general computer, smartphone, tablet, or other device. For example, the user might input "Copy files in a specific folder to a backup folder every day at 3:00 PM."

[1774] 2. Sending instructions

[1775] The terminal sends the input instructions in natural language to the server, which receives the instructions using a specified communication protocol (e.g., HTTP).

[1776] 3. Parsing the instructions

[1777] The server analyzes the received instructions and extracts the components through an instruction analysis module. This analysis uses a generative AI model (e.g., OpenAI's GPT-3). An example of a prompt is, "I will understand your instructions and break them down into specific steps. The information required for these steps is listed below."

[1778] 4. Emotion recognition

[1779] The server simultaneously launches an emotion engine to recognize the user's emotional state. The emotion engine utilizes services such as Microsoft's Emotion API to determine emotions from the user's input, facial expressions, and tone of voice.

[1780] 5. Generate RPA code

[1781] The RPA robot generation module on the server generates RPA code based on the extracted components and the user's emotional state. This RPA code is a script written in a programming language such as Python to perform a specified action at a predetermined time.

[1782] 6. Emotion-Based Regulation

[1783] The emotion engine can adjust the generation process and display content based on the user's emotions. For example, if the user is feeling stressed, the engine can provide detailed explanations of the generated code or a simplified interface.

[1784] 7. Sending the Product

[1785] The server sends the generated RPA code to the terminal, which then displays the received code to the user in a visually easy-to-verify format.

[1786] 8. Viewing and editing the generated output

[1787] The terminal displays the received RPA code to the user, who can then review the code and edit it if necessary. The editing interface is intuitive and designed to allow users to easily make corrections.

[1788] 9. User confirmation and execution

[1789] The user performs a final check of the generated code, makes any necessary corrections, and then deploys it to the execution environment to check whether it works correctly.

[1790] Specific examples

[1791] Example 1: Backing up files periodically

[1792] User input: The user inputs "Copy files in the working folder to the backup folder every day at 5 PM" into the terminal's input interface.

[1793] Instruction analysis and emotion recognition: The server analyzes the instruction and extracts the components "every day at 5 PM," "work folder," and "copy to backup folder." At the same time, the emotion engine analyzes the user's emotions and recognizes that the user is feeling stressed.

[1794] RPA code generation and emotion-based adjustment: The server generates a backup script based on the extracted information. Since the user is stressed, the emotion engine provides an interface with detailed explanations.

[1795] View and edit the generated script: A terminal will display the generated script to the user, allowing the user to edit the folder path and time, and convenient editing options are provided to reduce frustration.

[1796] Example 2: Frequent email checks

[1797] User input: The user types into the device, "Check email every hour and notify me if new email arrives."

[1798] Instruction analysis and emotion recognition: The server analyzes the components "every hour," "check email," and "notify when new email arrives," and the emotion engine recognizes that the user is excited.

[1799] RPA code generation and emotion-based adjustment: The server generates the email checking script and the emotion engine adjusts it to provide a simple interface.

[1800] View and edit the generated code: The terminal displays the generated code, allowing you to review and make any necessary modifications.

[1801] In this way, by combining the user's natural language instructions with emotion recognition functionality, it is possible to generate RPA robots that are easier to use and meet individual needs. By linking the generative AI model with the emotion engine, it is possible to provide appropriate support according to the user's condition.

[1802] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1803] Step 1:

[1804] The user uses the input interface of the terminal to input instructions in natural language.

[1805] Specific operation: For example, the user enters "Copy files in a specific folder to a backup folder every day at 3 PM."

[1806] Input: Natural language instructions.

[1807] Output: Input data displayed on the terminal.

[1808] Step 2:

[1809] The terminal transmits the input natural language instruction to the server.

[1810] Specific operation: The device sends the user's input data to the server using an HTTP request.

[1811] Input: Data entered into a terminal as natural language instructions.

[1812] Output: An HTTP request containing instructions is sent to the server.

[1813] Step 3:

[1814] The server analyzes the received instruction in an instruction analysis module.

[1815] Specific operation: The server provides the received data to a generative AI model (e.g., GPT-3) as a prompt sentence, which breaks down the instructions into their component parts.

[1816] Input: Natural language instructions sent to the server.

[1817] Output: Parsed component data.

[1818] Step 4:

[1819] The server concurrently launches an emotion engine to recognize the user's emotional state.

[1820] Specific operation: The server calls an emotion engine (e.g., Microsoft Emotion API) and analyzes the user's input, facial expressions, tone of voice, etc.

[1821] Input: Emotional data such as what the user types, facial expressions, and tone of voice.

[1822] Output: User's emotional state (e.g., stressed, excited, relaxed).

[1823] Step 5:

[1824] An instruction parsing module in the server breaks down the natural language instruction into its constituent components.

[1825] Specific operation: The server extracts important components (e.g., "every day at 3 p.m.", "specific folder," "copy to backup folder," etc.) from the output of the generated AI model.

[1826] Input: The data parsed by the generative AI model.

[1827] Output: The extracted component data.

[1828] Step 6:

[1829] The server's RPA robot generation module automatically generates an RPA script based on the extracted elements.

[1830] What it does: The server uses a programming language such as Python code to generate an RPA script based on the extracted components.

[1831] Input: Extracted component data.

[1832] Output: The generated RPA code.

[1833] Step 7:

[1834] The generation process and display content are adjusted based on the user's emotions recognized by the emotion engine.

[1835] Specific behavior: The server adjusts the generated code and interface depending on the user's emotional state (e.g., if the user is feeling stressed, it provides an interface with detailed explanations).

[1836] Input: The user's emotional state.

[1837] Output: The coordinated production process or display content.

[1838] Step 8:

[1839] The server sends the generated RPA code to the terminal.

[1840] Specific operation: The server sends the generated code to the device as an HTTP response.

[1841] Input: The generated RPA code.

[1842] Output: The RPA code sent to the terminal.

[1843] Step 9:

[1844] The terminal displays the received RPA code to the user and provides an interface that allows editing.

[1845] What it does: The device displays the generated code to the user and provides an interface for editing the code.

[1846] Input: The RPA code sent by the server.

[1847] Output: The RPA code that is visible to the user and can be edited.

[1848] Step 10:

[1849] The user checks the generated code, edits it if necessary, and finally deploys it to the execution environment.

[1850] Specific behavior: After the user checks the code and makes any necessary corrections, they press the run button to deploy the script to the execution environment and check its behavior.

[1851] Input: RPA code reviewed and edited by the user.

[1852] Output: RPA code that runs in the execution environment.

[1853] (Application example 2)

[1854] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1855] Existing self-driving vehicles do not adjust navigation or the in-car environment to take the user's emotional state into consideration, resulting in a lack of quality in the riding experience. Furthermore, there is no mechanism for optimizing the vehicle's driving experience based on the user's emotional state, so safety and comfort are not sufficiently ensured. Therefore, there is a need to provide a system that adjusts navigation and the in-car environment according to the user's emotional state.

[1856] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting instructions in natural language, means for analyzing the input instructions and extracting components, means for generating robot process automation code based on the extracted components, means for displaying the generated code to the user, emotion recognition means for recognizing the user's emotional state, means for adjusting the generation process and display content based on the emotional state, and means for adjusting the vehicle navigation and in-vehicle environment based on the user's emotional state. This makes it possible to provide optimal navigation and a comfortable in-vehicle environment according to the user's emotional state.

[1857] A "means for inputting instructions in natural language" is an interface that allows a user to convey commands or wishes to a system using words and sentences that they use on a daily basis.

[1858] "Means for analyzing input instructions and extracting components" refers to technology that understands natural language commands received from a user and breaks them down into specific actions and elements.

[1859] The "means for generating robotic process automation code based on the extracted components" is a function for creating RPA code for automatically executing tasks from the analyzed components.

[1860] "Means for displaying the generated code to the user" refers to a method for visually providing the automatically generated RPA code so that the user can check it.

[1861] "Emotion recognition means for recognizing the user's emotional state" refers to technology that grasps emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1862] "Means for adjusting the generation process and display content based on emotional state" refers to a method for changing and optimizing the information and interface content provided by the system according to the recognized emotion.

[1863] "Means for adjusting vehicle navigation and in-vehicle environment based on the user's emotional state" refers to technology for optimizing the route traveled by an autonomous vehicle and environmental settings such as lighting, music, and air conditioning in the vehicle, taking into account the user's emotional state.

[1864] A specific embodiment of the present invention is a system in which a server, a terminal, a user, and an emotion engine work in cooperation with each other, as will be described in detail below.

[1865] System Configuration

[1866] The system operates inside self-driving vehicles and automatically adjusts navigation and the interior environment based on instructions entered by the user in natural language.

[1867] 1. Server

[1868] The server has the following functions:

[1869] Natural Language Input Analysis: Analyzes natural language instructions sent from the device, extracts their components, and uses generative AI models to translate the instructions into specific actions.

[1870] Emotion Recognition: The car is equipped with an emotion engine that recognizes the user's emotional state. It analyzes the user's facial expressions and tone of voice based on data sent from the in-car camera and microphone to identify emotions.

[1871] RPA Code Generation: Generates robotic process automation (RPA) code based on the extracted components.

[1872] Navigation and environmental adjustment: Generate instructions to adjust the navigation route or the in-car environment (e.g., music, lighting, and air conditioning settings) based on the perceived emotional state.

[1873] 2. Terminal

[1874] The device has the following features:

[1875] User interface: Provides an interface for users to enter instructions in natural language.

[1876] Send instructions: Sends input instructions in natural language to a server, usually using an HTTP request.

[1877] Display of results: Displays the RPA code and adjustment instructions sent from the server to the user. The user can check and edit this information.

[1878] 3. Users

[1879] The user uses the terminal to input instructions in natural language, such as "send a message every day at 3pm" or "play relaxing music."

[1880] Hardware and software used

[1881] Hardware: autonomous vehicle control units, in-car cameras, voice recognition microphones, entertainment systems

[1882] Software: Emotion recognition engines (e.g., OpenVINO, Google Cloud AI), autonomous driving systems (e.g., Waymo, Tesla AutoPilot), maps and navigation software (e.g., Google Maps API)

[1883] Specific examples

[1884] Specific examples are shown below.

[1885] Example prompt sentence:

[1886] "If the user is feeling stressed, it will suggest a relaxing route and play classical music. It will move at a gentler pace, providing a calmer driving experience."

[1887] The user inputs a natural language command into the in-car device, such as "Play relaxing music every day at 3 p.m." The device sends this command to the server, which not only analyzes the command but also recognizes the user's emotional state using the in-car camera and microphone. If the server determines that the user is feeling stressed, it calculates a relaxing route and instructs the entertainment system to play classical music, allowing the user to enjoy a gentler, more relaxed driving experience.

[1888] The above is a specific embodiment for carrying out the present invention.

[1889] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1890] Step 1:

[1891] The user inputs a command in natural language. The user uses the input interface of the device to input a command such as "Play relaxing music every day at 3 pm."

[1892] Input: Natural language instructions ("Play relaxing music every day at 3pm")

[1893] Output: Natural language instruction data entered into the terminal

[1894] Step 2:

[1895] The device sends input natural language instructions to the server, usually using an HTTP request.

[1896] Input: Natural language instruction data entered into the terminal

[1897] Output: HTTP request sent to the server

[1898] Step 3:

[1899] The server analyzes the received instructions and extracts their components, using a generative AI model to break down the instructions into their components.

[1900] Input: Natural language instructions sent to the server

[1901] Data processing: Generative AI models analyze instructions and extract components

[1902] Output: Component data such as "everyday at 3pm" and "relaxing music"

[1903] Step 4:

[1904] The server activates the emotion engine to recognize the user's emotional state, analyzing the user's facial expressions and tone of voice using data from the in-car camera and microphone.

[1905] Input: In-car camera and microphone data (video and audio)

[1906] Data Computation: Emotion engine analyzes video and audio data to identify emotional states

[1907] Output: User's emotional state data (e.g., stress)

[1908] Step 5:

[1909] The server generates robotic process automation (RPA) code based on the extracted components, which performs a specified action (e.g., playing relaxing music).

[1910] Input: Component data ("every day at 3 pm", "relaxing music")

[1911] Data processing: Automatically generate code using the RPA code generation module

[1912] Output: RPA code (e.g. music playback script)

[1913] Step 6:

[1914] The server adjusts the generation process and display content based on the user's emotional state as recognized by the emotion engine. For example, if the user is feeling stressed, the server provides an interface with detailed explanations.

[1915] Input: Emotional state data (e.g., stress)

[1916] Data processing: Modify and optimize the generation process and display content based on emotional state

[1917] Output: Coordinated production process and display content

[1918] Step 7:

[1919] The server sends the generated RPA code and adjustment details to the terminal, allowing the user to check the generated code and the adjusted display details.

[1920] Input: RPA code, tailored generated processes and display content

[1921] Output: RPA code sent to the terminal and its display content

[1922] Step 8:

[1923] The terminal displays the received RPA code and its contents to the user, who can visually check the code and edit it if necessary.

[1924] Input: RPA code sent from the server and display content

[1925] Output: The RPA code and generated content displayed to the user

[1926] Step 9:

[1927] The user checks the generated code and edits it as necessary. After making a final check, the modified code is deployed to the execution environment and verified to work correctly.

[1928] Input: The displayed RPA code and the generated content

[1929] Output: Reviewed and edited RPA code, and post-deployment verification

[1930] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1931] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1932] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1933] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1934] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1935] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1936] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1937] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1938] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1939] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1940] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1941] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1942] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1943] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1944] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1945] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1946] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1947] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1948] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1949] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1950] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1951] The following is further disclosed regarding the above embodiment.

[1952] (Claim 1)

[1953] a means for inputting instructions in natural language;

[1954] means for analyzing input instructions and extracting components;

[1955] means for generating robotic process automation code based on the extracted components;

[1956] a means for displaying the generated code to the user;

[1957] A system including:

[1958] (Claim 2)

[1959] 10. The system of claim 1, wherein the analysis includes using a generative AI model.

[1960] (Claim 3)

[1961] 10. The system of claim 1, further comprising means for enabling a user to edit the generated robotic process automation code.

[1962] "Example 1"

[1963] (Claim 1)

[1964] a means for a user to input instructions in natural language;

[1965] means for the terminal to transmit input instructions to a server;

[1966] A means for the server to analyze the input instructions using the generative AI model and extract components;

[1967] means for the server to generate robotic process automation code based on the extracted components;

[1968] means for transmitting the generated code from the server to the terminal;

[1969] means for the terminal to display the generated code to the user and to enable the user to edit the generated code;

[1970] A system including:

[1971] (Claim 2)

[1972] The system of claim 1, wherein the analysis uses a generative AI model.

[1973] (Claim 3)

[1974] 10. The system of claim 1, wherein the generated robotic process automation code is user editable.

[1975] "Application Example 1"

[1976] (Claim 1)

[1977] a means for inputting instructions in natural language;

[1978] means for analyzing input instructions and extracting components;

[1979] means for generating robotic process automation code based on the extracted components;

[1980] A means of executing processes at specific times to automate inventory management tasks in physical stores;

[1981] a means for displaying the generated code to the user;

[1982] A system including:

[1983] (Claim 2)

[1984] 10. The system of claim 1, further comprising means for analyzing the instructions using the generative AI model.

[1985] (Claim 3)

[1986] 10. The system of claim 1, further comprising means for enabling a user to edit the generated robotic process automation code.

[1987] "Example 2: Combining Emotion Engines"

[1988] (Claim 1)

[1989] a means for inputting instructions in natural language;

[1990] means for transmitting the input instructions to a server;

[1991] means for parsing the received instructions by the server to extract components;

[1992] means for the server to recognize the emotional state of the user;

[1993] means for generating robotic process automation code based on the extracted components and emotional states;

[1994] means for adjusting the generation process and / or the displayed content based on the emotional state of the user;

[1995] means for transmitting the generated code to a user's terminal;

[1996] a means for displaying and allowing the generated code to be edited by the user;

[1997] A system including:

[1998] (Claim 2)

[1999] 10. The system of claim 1, wherein the analysis includes using a generative AI model.

[2000] (Claim 3)

[2001] 10. The system of claim 1, further comprising means for using an emotion recognition engine to recognize the emotional state.

[2002] "Application example 2 when combining emotion engines"

[2003] (Claim 1)

[2004] a means for inputting instructions in natural language;

[2005] means for analyzing input instructions and extracting components;

[2006] means for generating robotic process automation code based on the extracted components;

[2007] a means for displaying the generated code to the user;

[2008] emotion recognition means for recognizing an emotional state of a user;

[2009] a means for adjusting the generation process and / or display content based on the emotional state;

[2010] means for adjusting the vehicle's navigation and in-vehicle environment based on the user's emotional state;

[2011] A system including:

[2012] (Claim 2)

[2013] 10. The system of claim 1, wherein the analysis includes using a generative AI model.

[2014] (Claim 3)

[2015] 10. The system of claim 1, further comprising means for enabling a user to edit the generated robotic process automation code. [Explanation of symbols]

[2016] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for inputting instructions in natural language; means for analyzing input instructions and extracting components; means for generating robotic process automation code based on the extracted components; a means for displaying the generated code to the user; A system including:

2. 10. The system of claim 1, further comprising means for using a generative AI model for the analysis.

3. 10. The system of claim 1, further comprising means for enabling a user to edit the generated robotic process automation code.

Images