System

Abstract

A system is provided.SOLUTION: The system includes a means for acquiring voice input from a terminal, a means for converting the acquired voice data into text data, a means for analyzing a task, an object, an amount, and a date and time from the text data, a means for registering the analyzed information in a database, a means for executing the task based on the preset date and time, a means for automatically accessing a target system and performing input based on the analyzed information, and a means for notifying a user of an execution result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention relates to a system that allows a user to register a task through voice input and automatically executes the task at a specified date and time, thereby eliminating manual user operations and improving efficiency. However, existing voice recognition and task management systems often have difficulty properly interpreting user instructions and often lack accuracy in the execution of tasks. Therefore, there is a need for a method that accurately executes pre-registered tasks at a date and time specified by the user. The present invention aims to solve these problems. [Means for solving the problem]

[0005] The present invention provides a means for receiving voice input from a terminal and converting the voice data into text data. It then includes a means for analyzing the task, target, quantity, and date and time from the text data. The analyzed information is registered in a database, and a means for executing the task based on a preset date and time is provided. It also includes a means for automatically accessing the target system and making input based on the analyzed information. The present invention also includes a means for notifying the user of the execution results. This significantly reduces the user's manual operations and enables the task to be executed accurately at a specific date and time.

[0006] "Terminal" refers to a computing device that allows a user to provide voice input and that displays and manipulates data.

[0007] "Voice input" refers to instructions or commands given by voice to a terminal by a user.

[0008] "Voice data" refers to a digital representation of a user's voice input.

[0009] "Text data" refers to data in the form of a string of characters obtained by converting voice data.

[0010] A "task" refers to a specific action or operation instructed by a user, such as transferring money to a bank account.

[0011] "Subject" refers to any entity, such as a particular recipient or location, relevant to the performance of a task.

[0012] "Amount" refers to the numerical information required to execute a task, such as the amount to be transferred.

[0013] "Date and time" refers to the specific time or date when the task should be performed.

[0014] "Database" refers to a structured data management system for storing analyzed information.

[0015] The "pre-set date and time" refers to the scheduled execution date and time of the task specified by the user.

[0016] "Automatic access" means that the system is programmed to connect to and operate other systems or services without user intervention.

[0017] "Means of input" refers to the process of programmatically inputting required data into forms or fields in an automatically accessed system or service.

[0018] "Execution result" refers to the result or status obtained after the task is completed, such as success or failure of the transfer.

[0019] "Notification" refers to a message or alert that notifies the user of the execution result. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention is a system that analyzes voice input and automatically executes tasks set for a specific date. This system is composed of a series of processes: voice recognition, natural language processing, database management, automatic task execution, and notification. Each process is described in detail below.

[0042] Voice input and recognition

[0043] The user gives voice instructions to the terminal, such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0044] The device uses a microphone to capture audio data and transmits it to the server.

[0045] The server uses a speech recognition engine to convert the received voice data into text data. At this stage, a speech recognition service such as Google® Cloud Speech-to-Text API is used.

[0046] Natural Language Processing and Data Analysis

[0047] The server processes the converted text data and analyzes it using a natural language processing engine (e.g., spaCy or NLTK). As a result of this analysis, the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) are extracted.

[0048] Data registration and management

[0049] The server registers the analyzed information in a database, where the task, target, amount, and date and time are associated and saved as a record. The server also uses a scheduling system (e.g., a cron job) to set tasks to run at pre-determined dates and times.

[0050] Preparing and running a task

[0051] When the due date arrives, the server starts the process to execute the task. For example, in the case of a bank transfer, the following steps are taken:

[0052] 1. The server accesses the bank's online form.

[0053] 2. The server uses an image recognition engine (such as Tesseract OCR) to identify the fields on the transfer form.

[0054] 3. The server uses an automated script to enter the transfer destination (Yamada Taro) and amount (5,000 yen).

[0055] 4. The server will check the entered information, and if it is correct, press the transfer button to execute the transfer.

[0056] 5. The server captures the transfer success message and confirmation screen and uses image recognition to confirm the transfer completion.

[0057] Result notification

[0058] When the task is successfully executed, the server summarizes the results and notifies the user, using push notifications or email.

[0059] Specific examples

[0060] The user speaks, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0061] The device captures the audio and sends it to the server.

[0062] The server converts the speech into text using a speech recognition engine, and extracts "Yamada Taro," "5,000 yen," and "25th of this month."

[0063] The server registers the information in a database and schedules the action on a scheduled date.

[0064] On the 25th, the server automatically accesses the bank's online form and executes the transfer using image recognition and an automated script.

[0065] Once the transfer is complete, the server will summarize the results and notify the user.

[0066] The user checks the notification and finds out that the transfer has been completed.

[0067] This system allows users to easily set tasks via voice input, and the tasks are automatically executed at the specified date and time, significantly reducing the user's workload.

[0068] The processing flow will be explained below.

[0069] Step 1:

[0070] The user speaks into the terminal, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0071] Step 2:

[0072] The device uses the microphone to capture the user's voice input.

[0073] Step 3:

[0074] The device sends the captured audio data to the server.

[0075] Step 4:

[0076] The server receives the audio data.

[0077] Step 5:

[0078] The server calls a voice recognition engine and converts the voice data into text data.

[0079] Step 6:

[0080] The server retrieves the converted text data and recognizes it as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0081] Step 7:

[0082] The server sends the text data to the natural language processing engine and begins analysis.

[0083] Step 8:

[0084] The server uses a natural language processing engine (e.g., spaCy or NLTK) to extract the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) from the text data.

[0085] Step 9:

[0086] The server checks the extracted information and formats it as needed, for example converting date information to a standard format (e.g. 2023-10-25).

[0087] Step 10:

[0088] The server connects to the database and formats the information to create a new record, which includes the task, target, quantity, and date and time.

[0089] Step 11:

[0090] The server periodically monitors the database and checks the due dates of registered records.

[0091] Step 12:

[0092] The server uses a scheduling system (e.g. a cron job) to schedule the task to run on a specified day.

[0093] Step 13:

[0094] When the server reaches the scheduled date (e.g., the 25th), it starts a process to execute the configured task.

[0095] Step 14:

[0096] The server accesses the bank's online transfer form.

[0097] Step 15:

[0098] The server uses an image recognition engine (e.g., Tesseract OCR) to identify each field on the transfer form.

[0099] Step 16:

[0100] The server uses an automated script to enter the transfer destination (Yamada Taro) and amount (5,000 yen) into the specified fields.

[0101] Step 17:

[0102] The server checks the input information, and if there are no problems, clicks the transfer button to execute the transfer.

[0103] Step 18:

[0104] The server captures the transfer success message and confirmation screen and verifies it using image recognition.

[0105] Step 19:

[0106] After the server confirms that the transfer was successful, it summarizes the results.

[0107] Step 20:

[0108] The server converts the results into a format that can be sent to the user's device (e.g., push notification, email).

[0109] Step 21:

[0110] The terminal receives the notification from the server and notifies the user that the transfer has been completed.

[0111] Step 22:

[0112] The user checks the notification and confirms that the transfer was successful.

[0113] Example 1

[0114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] Conventional task schedule management systems require users to manually set dates and times and execute tasks at the appropriate times, resulting in inefficiencies and a high likelihood of errors. Furthermore, the inability to set tasks via voice input hinders user convenience. Furthermore, when automatically executing specific tasks, such as payment processing, security and reliability issues remain.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0117] In this invention, the server includes means for acquiring voice input from a terminal, means for converting the acquired voice data into text data, means for analyzing the task, target, amount, and date and time from the text data, means for registering the analyzed information in a database, means for executing the task based on a preset date and time, means for automatically accessing the target information processing system and making input based on the analyzed information, and means for notifying the user of the execution result. This allows the user to easily set a task via voice input, and the task is automatically executed at the specified date and time, significantly reducing the user's effort and enabling the task to be completed efficiently and securely.

[0118] "Terminal" refers to an electronic device for receiving voice input and transmitting data to a server.

[0119] "Voice input" refers to the use of voice to provide instructions and information to a system.

[0120] "Server" refers to a central processing unit for processing received voice data and managing the automated execution of tasks.

[0121] "Voice data" refers to data that is a digital representation of a voice signal captured by a device.

[0122] "Text data" refers to data in the form of a character string converted by speech recognition.

[0123] A "task" refers to an action or process that is to be performed at a specific date and time.

[0124] "Target" refers to the recipient or location involved in the execution of a task.

[0125] "Quantity" refers to the numerical value or amount required to perform a task.

[0126] "Date and time" refers to the date and time the task should be performed.

[0127] "Natural language processing" refers to the technology of analyzing meaning and extracting information from voice or text data.

[0128] "Database" refers to a data storage device for storing and managing analyzed information.

[0129] "Scheduling system" refers to a system that manages and executes tasks based on specified dates and times.

[0130] "Information processing system" refers to a system that is the subject of automated task execution (e.g., an online banking system).

[0131] "Image recognition" refers to technology for extracting specific information or patterns from visual data.

[0132] "Automatic input" refers to the process of automatically inputting the necessary data into a system or application based on analyzed information.

[0133] "Notification" refers to the means of informing the user of the results of task execution (e.g., push notification or email).

[0134] "Payment processing" refers to the task of transferring a specific amount of money to a specified account.

[0135] This invention is a system that analyzes voice input and automatically executes tasks set for a specific date. This system is composed of a series of processes: voice recognition, natural language processing, database management, automatic task execution, and notification. Each process is described in detail below.

[0136] Voice input and recognition

[0137] The user speaks to the device, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device uses a microphone to capture the voice data and sends it to the server. The server then uses a speech recognition engine such as Google Cloud Speech-to-Text API to convert the received voice data into text data. This allows the user to easily set tasks with voice instructions, without having to perform complex operations.

[0138] Natural Language Processing and Data Analysis

[0139] The server then analyzes the converted text data using a natural language processing engine (e.g., spaCy). This analysis extracts important information such as the task (transfer), target (e.g., Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month). This process clearly analyzes the user's voice instructions and extracts the necessary information.

[0140] Data registration and management

[0141] The server registers the analyzed information in a database (e.g., PostgreSQL). It also uses a scheduling system (e.g., a cron job) to set up tasks to be executed at pre-set dates and times. At this stage, data relating tasks, targets, amounts, and dates and times is efficiently managed.

[0142] Preparing and running a task

[0143] When the scheduled date arrives, the server automatically executes the task. For example, in the case of a transfer task, the server uses an automation tool such as Selenium to access the bank's online form, identifies the form fields using Tesseract OCR, and enters the required information using an automated script. Next, the server checks the input and, if correct, presses the transfer button to execute the process. Finally, it captures a success message and confirms the completion of the transfer.

[0144] Result notification

[0145] When the task is successfully executed, the server summarizes the results and notifies the user via push notification or email, allowing the user to keep track of the progress and completion of the task in real time.

[0146] Specific examples

[0147] The user voice-inputs, "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device captures the voice and sends it to the server. The server converts the voice to text using the Google Cloud Speech-to-Text API, extracting "Yamada Taro," "5,000 yen," and "25th of this month." The server registers the information in a database and schedules the action for the scheduled date. On the 25th, the server uses Selenium to access the bank's online form and executes the transfer using Tesseract OCR and an automated script. Once the transfer is complete, the server summarizes the results and notifies the user via SMTP. This allows the user to fully manage their tasks without any hassle.

[0148] Prompt Sentence Examples

[0149] An example of a prompt sentence to be input to the generative AI model is as follows.

[0150] "This system allows you to set tasks via voice input, and then automatically executes the tasks at the specified date and time. For example, if you say, 'Transfer 5,000 yen to Yamada Taro on the 25th of this month,' the system will automatically transfer the money on that date. Could you please tell me the specific processing steps of the program?"

[0151] This prompt allows you to obtain detailed explanations and necessary steps from the generative AI model.

[0152] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0153] Step 1: Capturing Audio Input

[0154] The user gives a voice command such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device uses a microphone to capture the voice data and converts it into a digital format. This digital voice data is then sent directly to the server as an HTTP request. The input to the device is a voice signal, and the output is the converted digital voice data.

[0155] Step 2: Sending and Recognizing Voice Data

[0156] The server receives the digital audio data and sends it to the Google Cloud Speech-to-Text API, which converts the audio data into text. The server's input is digital audio data and its output is text data.

[0157] Step 3: Analyzing the text data

[0158] The text data received by the server is analyzed using a natural language processing engine (for example, spaCy). The analysis process extracts information such as the task content (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month). This process uses natural language processing technology to understand the text data and identify the necessary information. The server inputs text data and outputs analyzed information (task, target, amount, date and time).

[0159] Step 4: Registering and scheduling data

[0160] The server registers the analyzed information in a database. This information includes the task content, target, amount, and date and time. It also uses a scheduling system (for example, a cron job) to set up the task to run based on the analyzed date and time. The server's input is the analyzed information, and its output is confirmation of registration in the database and task scheduling settings.

[0161] Step 5: Automate Task Execution

[0162] At the scheduled time, the server will start a process to execute the scheduled task, which includes the following actions:

[0163] 1. The server uses an automation tool such as Selenium to access the bank's online form.

[0164] 2. The server uses Tesseract OCR to identify the fields on the transfer form.

[0165] 3. The server uses an automated script to enter the transfer destination (Yamada Taro) and amount (5,000 yen) into the form.

[0166] 4. The server checks the input information, and if it is correct, press the send button to execute the transfer.

[0167] 5. The server captures the successful transfer message and uses Tesseract OCR to confirm the transfer was successful.

[0168] The server's input is the scheduled task information, and its output is the result of the transfer execution.

[0169] Step 6: Notification of results

[0170] When the task is completed successfully, the server aggregates the execution results and notifies the user. Notification methods include push notification and email. Specifically, the server sends an email using the SMTP protocol or a push notification using Firebase Cloud Messaging (FCM). The input to the server is the task execution result, and the output is a notification to the user.

[0171] Each step of this system allows users to easily set tasks via voice input and have them executed automatically at the specified date and time, saving time and effort and enabling tasks to be completed efficiently.

[0172] (Application example 1)

[0173] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0174] In modern life, manually scheduling and executing everyday tasks, such as making payments, can be extremely cumbersome and time-consuming. There is a particular need for systems that can easily set up and automatically execute these tasks using voice commands. However, current systems are often complex to operate and unreliable. Therefore, there is an urgent need to improve the accuracy and convenience of task automation systems that use voice commands.

[0175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0176] In this invention, the server includes means for acquiring voice input from a terminal, means for converting the acquired voice data into text data, means for analyzing the task, target, amount, and date and time from the text data, means for registering the analyzed information in a database, means for executing the task based on a preset date and time, means for automatically accessing the target system and making input based on the analyzed information, means for notifying the execution result, a script for processing the scheduled task, means for starting an automatic script for automatically executing a payment task at a specified date and time, and means for analyzing information extracted from the voice input using a generative AI model. This enables highly accurate and efficient task setting and automatic execution using voice instructions.

[0177] A "terminal" is a device that allows a user to input voice, such as a smartphone or smart glasses.

[0178] The "means for acquiring voice input" is a function for capturing voice data using the microphone of the terminal and transmitting the voice data to the system.

[0179] The term "means for converting voice data into text data" refers to a process of converting voice data into character string data using a voice recognition engine.

[0180] "Means of analyzing tasks, targets, quantities, and dates" refers to the process of using a natural language processing engine to extract important information from text data.

[0181] "Means for registering in a database" is a function that stores analyzed information in a database so that it can be used later.

[0182] "Means for executing a task based on a preset date and time" refers to a process for automatically executing a task at a specified date and time using a scheduling system.

[0183] "Means for automatically accessing and inputting information into the target system" refers to a function that uses a script or API to automatically access the target system and input the necessary information.

[0184] "Means for notifying the execution results" refers to push notification or email sending functions for notifying the user of the execution results of the task.

[0185] "Script for processing scheduled tasks" refers to program code that is executed to automatically process pre-planned tasks.

[0186] The "means for starting an automatic script for automatically executing a payment task at a specified date and time" is a function for starting a script for automatically executing a payment task at a specified date and time.

[0187] "Means for analyzing information extracted from voice input using a generative AI model" refers to the process of using a generative AI model to perform detailed analysis of data obtained from voice input and extract necessary information.

[0188] This invention is a system that allows a user to set a task using voice input through a terminal and automatically executes the task at a specified date and time. This system is constructed by combining multiple specific hardware and software.

[0189] First, users use a smartphone or smart glasses as hardware for voice input. The device's microphone is used to capture voice input, and speech recognition software such as the Google Cloud Speech-to-Text API is used to convert the speech into text data.

[0190] Next, the server analyzes the received text data using a natural language processing engine (e.g., spaCy, NLTK), extracting important information such as the task, target, amount, and date and time. This analyzed information is then stored in an SQL database.

[0191] Based on the extracted information, the server uses a scheduling system (e.g., cron job, APScheduler) to set up a process to automatically execute the task at a pre-set date and time. Specifically, the server starts a script to automatically execute the payment task, and uses an image recognition engine (e.g., Tesseract OCR) to identify input fields and input information using the automatic script.

[0192] Furthermore, the information extracted from the voice input is analyzed with high accuracy using a generative AI model, which improves task execution accuracy and reduces the burden on the user.

[0193] Finally, if the task is successfully executed, the server summarizes the execution results and notifies the user. This notification is done using push notification or an email sending service (e.g., Amazon SES).

[0194] As a concrete example, if a user gives a voice command such as "Pay the water bill of 3,000 yen on the 25th of this month," this voice is captured by the device and converted into text data by a speech recognition engine. Then, a natural language processing engine extracts information such as "payment task," "water bill," "3,000 yen," and "25th of this month," and registers it in a database. The server then automatically executes the payment procedure at the specified date and time, and notifies the user of the results.

[0195] An example of a prompt sentence is, "Pay the water bill of 3,000 yen on the 25th of this month," and extract the task, target, amount, and date and time from the voice command.

[0196] This system allows users to easily and efficiently set tasks using voice instructions, and the tasks are then automatically executed at the specified date and time, significantly reducing the amount of work required.

[0197] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0198] Step 1:

[0199] The user speaks into the device

[0200] Specific operation: The user gives voice instructions to their smartphone or smart glasses, such as "Pay the water bill of 3,000 yen on the 25th of this month."

[0201] Input: Audio data

[0202] Output: Audio data

[0203] Step 2:

[0204] The device captures the audio data and sends it to the server.

[0205] Specific operation: The device's microphone captures audio and sends the audio data to the Google Cloud Speech-to-Text API.

[0206] Input: Audio data

[0207] Output: Text data

[0208] Step 3:

[0209] The server uses a speech recognition engine to convert the voice data into text data.

[0210] Specific operation: The Google Cloud Speech-to-Text API converts the audio data into text data and sends it back to the server.

[0211] Input: Audio data

[0212] Output: Text data

[0213] Step 4:

[0214] The server analyzes the text data and extracts the task, target, amount, and date and time.

[0215] Specific operation: The server uses a natural language processing engine such as spacy or NLTK to analyze the text data and extract important information (task, target, quantity, date and time).

[0216] Input: Text data

[0217] Output: Task, target, amount, date and time

[0218] Step 5:

[0219] The server registers the analyzed information in a database

[0220] Specific operation: The server stores the extracted information in an SQL database.

[0221] Input: Task, Target, Amount, Date and Time

[0222] Output: Database records

[0223] Step 6:

[0224] The server starts the scheduling system to set up scheduled tasks.

[0225] Specific operation: The server will use APScheduler or a cron job to configure the task to run at the parsed date and time.

[0226] Input: Task, Target, Amount, Date and Time

[0227] Output: Scheduled Job

[0228] Step 7:

[0229] The server will automatically execute the payment task at the specified date and time.

[0230] What happens: At the specified date and time, the server launches an automated script that contacts the payment system to perform the task, using an image recognition engine (such as Tesseract OCR) to identify input fields, enter target and quantity information, and perform verification.

[0231] Input: Task, Target, Amount, Date and Time

[0232] Output: Payment made

[0233] Step 8:

[0234] The server notifies the user of the execution result.

[0235] Specific operation: When the payment is successfully completed, the server notifies the user of the result using a push notification or email sending service (such as Amazon SES).

[0236] Input: Payment execution result

[0237] Output: Notification message

[0238] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0239] The present invention combines a system that analyzes voice input and automatically executes tasks set for specific dates with an emotion engine. In addition to speech recognition, natural language processing, database management, automatic task execution, and notifications, the system also includes a process that recognizes the user's emotions and adjusts the priority and execution method of tasks based on those emotions. Each process is described in detail below.

[0240] Voice input and recognition

[0241] The user gives voice instructions to the terminal, such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0242] The device uses a microphone to capture audio data and transmits it to the server.

[0243] The server uses a speech recognition engine to convert the received voice data into text data. At this stage, speech recognition services such as Google Cloud Speech-to-Text API are used.

[0244] Natural Language Processing and Data Analysis

[0245] The server processes the converted text data and analyzes it using a natural language processing engine (e.g., spaCy or NLTK). As a result of this analysis, the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) are extracted.

[0246] emotion recognition

[0247] The server uses an emotion engine to analyze the user's emotions from the voice input, using factors such as voice tone, speed, and intonation, for example, to analyze the level of stress or tension in the user's voice.

[0248] Data registration and management

[0249] The server registers the analyzed information in a database, where the task, target, amount, date and time, and emotion information are stored as associated records. The server also uses a scheduling system (e.g., a cron job) to set tasks to be executed at pre-set dates and times.

[0250] Preparing and running a task

[0251] When the due date arrives, the server starts the process to execute the task. For example, in the case of a bank transfer, the following steps are taken:

[0252] 1. The server accesses the bank's online form.

[0253] 2. The server uses an image recognition engine (e.g., Tesseract OCR) to identify the fields on the transfer form.

[0254] 3. The server uses an automatic script to input the transfer destination (Yamada Taro) and the amount (5,000 yen). For users with high stress, the server automatically checks their balance.

[0255] 4. The server will check the entered information, and if it is correct, press the transfer button to execute the transfer.

[0256] 5. The server captures the transfer success message and confirmation screen and uses image recognition to confirm the transfer completion.

[0257] Result notification

[0258] When the task is successfully completed, the server summarizes the results and notifies the user. This notification includes an appropriate message based on the analysis results of the emotion engine. For example, a user experiencing high stress may receive a positive, relaxing message.

[0259] Specific examples

[0260] The user speaks, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0261] The device captures the audio and sends it to the server.

[0262] The server converts the speech into text using a speech recognition engine, and extracts "Yamada Taro," "5,000 yen," and "25th of this month."

[0263] The server uses an emotion engine to analyze the stress the user is feeling.

[0264] The server registers the information in a database and schedules the action on a scheduled date.

[0265] On the 25th, the server automatically accesses the bank's online form and executes the transfer using image recognition and an automated script.

[0266] Once the transfer is complete, the server compiles the results and notifies the user with a stress-reducing message.

[0267] The user checks the notification and finds out that the transfer was successful.

[0268] This system allows users to easily set tasks via voice input, and the tasks are automatically executed at the specified date and time, significantly reducing the effort required for operation. In addition, by combining it with an emotion engine, it is possible to respond flexibly to the user's emotional state, improving user satisfaction.

[0269] The processing flow will be explained below.

[0270] Step 1:

[0271] The user speaks into the terminal, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0272] Step 2:

[0273] The device uses the microphone to capture the user's voice input.

[0274] Step 3:

[0275] The device sends the captured audio data to the server.

[0276] Step 4:

[0277] The server receives the audio data.

[0278] Step 5:

[0279] The server calls a voice recognition engine and converts the voice data into text data.

[0280] Step 6:

[0281] The server retrieves the converted text data and recognizes it as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0282] Step 7:

[0283] The server sends the text data to the natural language processing engine and begins analysis.

[0284] Step 8:

[0285] The server uses a natural language processing engine (e.g., spaCy or NLTK) to extract the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) from the text data.

[0286] Step 9:

[0287] The server uses an emotion engine to analyze the user's emotions from the voice input, using voice tone, speed, intonation, etc. For example, analyzing the user's voice for stress or tension levels.

[0288] Step 10:

[0289] The server formats the parsed information and converts date information into a standard format (e.g., 2023-10-25).

[0290] Step 11:

[0291] The server connects to the database and registers the formatted information as a new record, which includes the task, target, amount, date and time, and emotional information.

[0292] Step 12:

[0293] The server periodically monitors the database for the due dates of new records.

[0294] Step 13:

[0295] The server uses a scheduling system (e.g. a cron job) to schedule the task to run on a specified day.

[0296] Step 14:

[0297] When the server reaches the scheduled date (e.g., the 25th), it starts a process to execute the configured task.

[0298] Step 15:

[0299] The server accesses the bank's online transfer form.

[0300] Step 16:

[0301] The server uses an image recognition engine (e.g., Tesseract OCR) to identify each field on the transfer form.

[0302] Step 17:

[0303] The server uses an automated script to enter the transfer recipient (Yamada Taro) and amount (5,000 yen) into the specified fields.

[0304] Step 18:

[0305] The server checks the input information, and if there are no problems, clicks the transfer button to execute the transfer.

[0306] Step 19:

[0307] The server captures the transfer success message and confirmation screen and verifies it using image recognition.

[0308] Step 20:

[0309] The server compiles the results and generates appropriate messages based on the user's emotional information. For example, it selects a message with a relaxing effect for a user experiencing high stress.

[0310] Step 21:

[0311] The server converts the results into a format that can be sent to the user's device (e.g., push notification, email).

[0312] Step 22:

[0313] The terminal receives a notification from the server and notifies the user that the transfer has been completed. The notification includes a message that reflects the user's emotions.

[0314] Step 23:

[0315] The user checks the notification and finds out that the transfer was successful. The notification message is tailored to the user's emotions, increasing user satisfaction.

[0316] Example 2

[0317] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0318] Conventional voice input systems have the drawback of causing a lot of effort and stress when users execute tasks set for a specific date. For important tasks such as bank transfers and reservations, manual operation errors and emotional stress are particularly problematic. Furthermore, the system lacks the flexibility to respond to the user's emotional state, leaving room for improvement in user satisfaction.

[0319] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring voice input, a means for converting the acquired voice data into text data, a means for analyzing the text data to identify the task, target, quantity, and date and time, a means for registering the analyzed information in a database, a means for executing the task based on a preset date and time, a means for automatically accessing the target system and performing input based on the analyzed information, a means for notifying the user of the execution result, and a means for analyzing emotional information from the voice data and adjusting the priority and execution method of the task. This allows the user to easily set a task via voice input, and the task is automatically executed at the specified date and time, significantly reducing the effort required for operation. Furthermore, by combining an emotion engine, flexible responses based on the user's emotional state are possible, thereby improving user satisfaction.

[0320] The "means for acquiring voice input" is a function for capturing voice uttered by a user as a digital signal through the device's microphone and transmitting it to a subsequent processing step.

[0321] The "means for converting captured voice data into text data" is a function that includes a process for recognizing captured voice data and converting it into a corresponding text format.

[0322] "Means for analyzing tasks, targets, quantities, and dates and times from text data" refers to a function for analyzing text data using natural language processing technology and extracting information on the specified tasks, targets, quantities, and dates and times.

[0323] The "means for registering analyzed information in a database" is a function for storing extracted task-related information in a database and managing it so that it can be used later.

[0324] The "means for executing a task based on a preset date and time" is a function that automatically starts and executes a task at a specified date and time based on task information stored in a database.

[0325] "Means for automatically accessing the target system and entering data based on analyzed information" refers to a function that automates access to an external system required to execute a task and enters appropriate data based on analyzed information.

[0326] The "means for notifying the user of the execution result" is a function for sending a notification to inform the user of the execution status and result of the task.

[0327] "Means for analyzing emotional information from voice data and adjusting task priority and execution method" is a function that analyzes the user's emotional state at the time of voice input and adjusts task priority and execution method according to the user's emotions.

[0328] The present invention combines an emotion engine with a system that analyzes voice input and automatically executes tasks set for specific dates. This system is configured using the following hardware and software.

[0329] Acquiring and converting voice input

[0330] The user speaks to the terminal, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The terminal uses a microphone to capture the voice data and sends it to the server.

[0331] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API, which provides highly accurate speech recognition and enables real-time conversion.

[0332] Natural Language Processing and Data Analysis

[0333] The server analyzes the converted text data using a natural language processing engine (e.g., spaCy or NLTK). Through this analysis, the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) are extracted.

[0334] Emotion analysis

[0335] The server uses an emotion engine to analyze the user's emotions from the voice input. This emotion analysis uses the tone, speed, and intonation of the voice. For example, to analyze the level of stress or tension from the voice, Google Cloud Natural Language API and IBM Watson (registered trademark) Tone Analyzer are used.

[0336] Data registration and management

[0337] The server registers the analyzed information in a database, which includes the task, target, amount, date and time, and emotion information, using a database management system such as MySQL (registered trademark) or PostgreSQL. The server also uses a scheduling system (e.g., cron job) to set tasks to be executed at predetermined dates and times.

[0338] Executing a task

[0339] When the specified date and time arrives, the server will start executing the task. For example, in the case of a bank transfer, the following steps are taken:

[0340] 1. The server uses the Selenium library in Python to access the bank's online form.

[0341] 2. The server uses an image recognition engine (e.g., Tesseract OCR) to identify the fields in the transfer form, and then uses an automatic script to enter the transfer recipient (Yamada Taro) and the amount (5,000 yen). A setting to automatically check the balance is added, especially for users with high stress levels.

[0342] 3. The server will check the entered information, and if it is correct, press the transfer button to execute the transfer.

[0343] 4. The server captures the transfer success message and confirmation screen and uses image recognition to confirm the transfer completion.

[0344] Notification of results

[0345] After the server completes the task, it notifies the user of the result. The notification includes an appropriate message based on the results of the sentiment analysis. For example, a user experiencing high stress might receive a positive, relaxing message such as, "The transfer has been successfully completed. Relax and have a wonderful day."

[0346] Specific examples

[0347] The user gives a voice command such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device captures the voice and sends it to the server. The server converts the voice data into text using the Google Cloud Speech-to-Text API. The server analyzes the text using a natural language processing engine and extracts "Yamada Taro," "5,000 yen," and "the 25th of this month." The server uses an emotion engine to analyze the user's stress level. The server registers the information in a database and schedules the necessary actions for the specified date and time. On the 25th, the server automatically accesses the bank's online form and performs the transfer using Selenium and Tesseract OCR. When the transfer is complete, the server summarizes the results and notifies the user. The user checks the notification and learns that the transfer was successful.

[0348] By implementing this system, users can easily set tasks via voice commands, and the tasks will be automatically executed at the specified date and time, significantly reducing the amount of work required. In addition, by combining it with an emotion engine, flexible responses according to the user's emotional state become possible, improving user satisfaction.

[0349] An example of a generative AI model and prompts to use is ChatGPT (registered trademark)-4, and examples of prompts include:

[0350] "Please explain in detail the process of your new system that combines an emotion engine with a system that automatically executes tasks based on voice input."

[0351] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0352] Processing flow

[0353] Step 1: Getting voice input

[0354] The user gives a voice command, and the device uses the microphone to capture the voice data and transmits it to the server.

[0355] Input: User's voice command (e.g., "Transfer 5,000 yen to Yamada Taro on the 25th of this month.")

[0356] Output: Captured audio data (digital format)

[0357] Specific operation: The user speaks into the microphone. The device's microphone captures the voice and generates real-time audio data, which is then sent to the server via the network.

[0358] Step 2: Convert audio data to text

[0359] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API.

[0360] Input: Captured audio data

[0361] Output: Converted text data (e.g. "Transfer 5,000 yen to Yamada Taro on the 25th of this month")

[0362] What happens: The server processes the audio data via the Google Cloud Speech-to-Text API and generates corresponding text data, which is passed on to the next processing step.

[0363] Step 3: Analyzing the text data

[0364] The server uses a natural language processing engine (e.g., spaCy or NLTK) to parse the text data and extract important information such as task, subject, amount, date and time.

[0365] Input: Converted text data

[0366] Output: Extracted information (Task = Transfer, Target = Yamada Taro, Amount = 5000 yen, Date and Time = 25th of this month)

[0367] How it works: The server passes the text data to a natural language processing engine, which performs grammatical and semantic analysis, extracting specific keywords (task, subject, quantity, date and time).

[0368] Step 4: Sentiment Analysis

[0369] The server uses an emotion engine to analyze the user's emotion from the voice input, including the tone, rate, and intonation of the voice.

[0370] Input: converted audio data, analyzed text data

[0371] Output: Emotional information (e.g., stress level = high)

[0372] Specific operation: The server passes voice and text data to the emotion engine, which analyzes emotions from these data. It analyzes the voice characteristics and determines the user's emotional state.

[0373] Step 5: Registering and scheduling data

[0374] The server registers the parsed information in a database and schedules the task to be executed using a scheduling system (e.g., a cron job).

[0375] Input: Extracted information, emotional information

[0376] Output: Task information registered in the database, set schedule

[0377] Specific operation: The server stores the analyzed task information and emotion information in a database and schedules the task to be executed at the specified date and time using a database management system and a scheduling tool.

[0378] Step 6: Task execution

[0379] When the set date and time arrives, the server starts the task execution process. For example, in the case of a transfer task, the server accesses the bank's online system and inputs the necessary information.

[0380] Input: Task information and emotion information registered in the database

[0381] Output: Task executed, confirmation message

[0382] What it does: The server uses Selenium to access the bank's online transfer form, uses Tesseract OCR to identify the required fields, and then uses an automated script to enter the transfer details and amount, verify the details, and execute the transfer.

[0383] Step 7: Notification of results

[0384] Once the task is completed, the server notifies the user of the results, including an appropriate message based on the emotion information.

[0385] Input: Results of the executed task, emotional information

[0386] Output: Notification message sent to the user

[0387] Specific operation: The server confirms the success of the task execution and generates a notification message based on that. It also takes into account emotional information and sends a message that corresponds to the user's state. For example, it could say, "The transfer has been successfully completed. Please relax and have a wonderful day."

[0388] (Application example 2)

[0389] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0390] Conventional automated task execution systems using voice input simply execute tasks without considering the user's emotional state. As a result, depending on the user's situation or stress level, appropriate responses are not made, and satisfaction cannot be sufficiently improved. Furthermore, flexible adjustments based on emotions are required for the execution of payment tasks, but no technology that can achieve this has existed.

[0391] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring voice input from the terminal, means for converting the acquired voice data into text data, means for analyzing the task, target, quantity, and date and time from the text data, means for registering the analyzed information in a database, means for executing the task based on a preset date and time, means for automatically accessing the target system and making input based on the analyzed information, means for analyzing the user's emotions using an emotion engine, means for adjusting the priority and execution method of the task based on the user's emotional state, and means for notifying the user of the execution result. This allows the execution of payment tasks via voice input to be flexibly adjusted according to the user's emotional state, thereby improving user satisfaction.

[0392] A "terminal" is an electronic device for obtaining voice input.

[0393] "Voice input" refers to voice data uttered by the user, and includes instructions and commands to the system.

[0394] "Audio Data" means a digital recording of audio input captured by a Device.

[0395] "Text data" is character string data obtained by analyzing voice data using a voice recognition engine.

[0396] A "task" refers to a specific action or operation that is executed based on a user's voice instruction.

[0397] A "subject" is an individual, system, or physical or digital entity involved in the performance of a task.

[0398] "Quantity" refers to the numerical values ​​or metrics required to perform a task.

[0399] "Date and time" refers to a specific date and time when a task is scheduled to be performed.

[0400] A "database" is a structured data storage system for storing and managing analyzed information.

[0401] "Automatic access" means that the system connects to online services or forms based on a pre-defined process without manual intervention.

[0402] "Entering" means automatically writing the analyzed information into the target system.

[0403] An "emotion engine" is a software or hardware module that analyzes a user's emotions from their voice input and identifies their emotional state.

[0404] "Emotional state" refers to the psychological state or stress level experienced by a user.

[0405] "Priority" refers to the execution order and importance of the analyzed task.

[0406] An "execution method" is a means for determining how a task should proceed depending on the user's emotional state.

[0407] "Notification" is the act of informing a user of the results or status of an executed task.

[0408] The embodiments of the present invention will now be described in detail.

[0409] System configuration

[0410] The system consists of the following major hardware and software components:

[0411] Device: Smartphone (iOS / ANDROID (registered trademark))

[0412] Server: Speech recognition engine (Google Cloud Speech-to-Text API), natural language processing engine (spaCy), emotion analysis engine (e.g., voice tone analysis software), database (SQLite), task scheduling system

[0413] Network: The internet connection that connects your device to the server.

[0414] Main features of the program

[0415] The program has the function of automatically executing tasks through voice input, and adjusts task priority and execution method according to the user's emotional state. To achieve this, a complex system is constructed that goes through steps such as voice capture, voice recognition, natural language processing, emotion analysis, database registration, task scheduling, execution, and result notification.

[0416] Processing flow and technologies used

[0417] 1. Audio Capture

[0418] The user inputs voice using the microphone of the device.

[0419] The device captures the audio data and sends it to the server.

[0420] 2. Voice Recognition

[0421] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data.

[0422] For example, if the user instructs "Pay the electricity bill of 5,000 yen next Monday," the voice data is converted into text "Pay the electricity bill of 5,000 yen next Monday."

[0423] 3. Natural Language Processing

[0424] The converted text data is analyzed using spaCy to extract task details (date, amount, target).

[0425] In this example, "electricity bill," "5,000 yen," and "next Monday" are extracted.

[0426] 4. Emotion analysis

[0427] The server analyzes the emotional state from the voice input, analyzing the tone, speed, intonation, etc. with an emotion analysis engine.

[0428] For example, if the user's voice indicates that he or she is stressed, the user is recorded as "stressed."

[0429] 5. Database registration

[0430] The analyzed information is registered in a database, and the task, subject, amount, date and time, and emotional information are stored as associated records.

[0431] 6. Task Scheduling

[0432] Based on the registered information, the task is scheduled to be executed at a preset date and time.

[0433] For example, a task to automatically pay the electricity bill is scheduled for "next Monday."

[0434] 7. Task execution

[0435] At the specified date and time, the server will automatically make the online payment.

[0436] Based on emotional state, stressed users may be asked to check their balances in advance.

[0437] 8. Result notification

[0438] When the task is successfully completed, the user is notified of the result and a message is sent according to the user's emotional state, such as a relaxing message like "Thank you for your hard work, your electricity bill has been paid."

[0439] Examples of concrete examples and prompts

[0440] For example, if the user speaks something like this:

[0441] Today, I want to make sure I don't forget to transfer my rent, so please set it up so that the rent of 70,000 yen is automatically transferred to a specified account on the first day of each month. I'm a little nervous about asking for rent payments.

[0442] This system allows users to easily set and execute tasks, and enables flexible responses according to the user's emotional state.

[0443] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0444] Step 1:

[0445] The user uses the microphone on the device to input voice. For example, the user might say, "Please pay the electricity bill of 5,000 yen next Monday." The input to the device is voice data, which the device captures. The output here is the captured voice data.

[0446] Step 2:

[0447] The device sends the captured audio data to the server. The input is the audio data captured on the device, and the output to the server is the transmission of audio data. Once the server receives the audio data, it is ready to proceed to the next step.

[0448] Step 3:

[0449] The server uses the Google Cloud Speech-to-Text API to convert the voice data into text data. The input is the voice data, and the output is the converted text data. The server calls the speech recognition engine, analyzes the voice data, and converts it into text.

[0450] Step 4:

[0451] The server uses a natural language processing engine (spaCy) to analyze task details (date, amount, target) from text data. The input is text data, and the output is detailed task information (e.g., "electricity bill," "5,000 yen," "next Monday"). The server analyzes the text data and extracts important information.

[0452] Step 5:

[0453] The server uses an emotion analysis engine to analyze the user's emotional state from the voice data. The input is the initial voice data or text data, and the output is the emotional state (e.g., "stressed"). The server analyzes the voice tone, speed, and intonation to identify the emotional state.

[0454] Step 6:

[0455] The server registers the analyzed information in a database. The input is the analyzed task details and emotional state, and the output is a record saved in the database. The server stores the extracted information in the database as structured data.

[0456] Step 7:

[0457] The server schedules tasks to be executed based on a pre-set date and time. The input is the task details, execution date and time, and the output is the scheduled task. The server uses the task scheduling system to set up the task to be executed automatically on the specified date and time.

[0458] Step 8:

[0459] At the scheduled time, the server automatically executes the online payment. The inputs are task details, emotional state, and schedule information, and the output is confirmation of payment completion. Specifically, the server automatically accesses the target payment system, enters the necessary information (payee, amount) into the input form, and completes the payment.

[0460] Step 9:

[0461] When the task is successfully completed, the server notifies the user of the result. The input is the task execution status and emotional state, and the output is a notification message sent to the user. The server generates an appropriate message depending on the emotional state, notifying the user, for example, "Thank you for your hard work. Your electricity bill has been paid."

[0462] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0463] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0464] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0465] [Second embodiment]

[0466] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0467] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0468] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0469] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0470] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0471] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0472] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0473] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0474] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0475] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0476] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0477] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0478] The present invention is a system that analyzes voice input and automatically executes tasks set for a specific date. This system is composed of a series of processes: voice recognition, natural language processing, database management, automatic task execution, and notification. Each process is described in detail below.

[0479] Voice input and recognition

[0480] The user gives voice instructions to the terminal, such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0481] The device uses a microphone to capture audio data and transmits it to the server.

[0482] The server uses a speech recognition engine to convert the received voice data into text data. At this stage, speech recognition services such as Google Cloud Speech-to-Text API are used.

[0483] Natural Language Processing and Data Analysis

[0484] The server processes the converted text data and analyzes it using a natural language processing engine (e.g., spaCy or NLTK). As a result of this analysis, the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) are extracted.

[0485] Data registration and management

[0486] The server registers the analyzed information in a database, where the task, target, amount, and date and time are associated and saved as a record. The server also uses a scheduling system (e.g., a cron job) to set tasks to run at pre-determined dates and times.

[0487] Preparing and running a task

[0488] When the due date arrives, the server starts the process to execute the task. For example, in the case of a bank transfer, the following steps are taken:

[0489] 1. The server accesses the bank's online form.

[0490] 2. The server uses an image recognition engine (such as Tesseract OCR) to identify the fields on the transfer form.

[0491] 3. The server uses an automated script to enter the transfer destination (Yamada Taro) and amount (5,000 yen).

[0492] 4. The server will check the entered information, and if it is correct, press the transfer button to execute the transfer.

[0493] 5. The server captures the transfer success message and confirmation screen and uses image recognition to confirm the transfer completion.

[0494] Result notification

[0495] When the task is successfully executed, the server summarizes the results and notifies the user, using push notifications or email.

[0496] Specific examples

[0497] The user speaks, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0498] The device captures the audio and sends it to the server.

[0499] The server converts the speech into text using a speech recognition engine, and extracts "Yamada Taro," "5,000 yen," and "25th of this month."

[0500] The server registers the information in a database and schedules the action on a scheduled date.

[0501] On the 25th, the server automatically accesses the bank's online form and executes the transfer using image recognition and an automated script.

[0502] Once the transfer is complete, the server will summarize the results and notify the user.

[0503] The user checks the notification and finds out that the transfer has been completed.

[0504] This system allows users to easily set tasks via voice input, and the tasks are automatically executed at the specified date and time, significantly reducing the user's workload.

[0505] The processing flow will be explained below.

[0506] Step 1:

[0507] The user speaks into the terminal, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0508] Step 2:

[0509] The device uses the microphone to capture the user's voice input.

[0510] Step 3:

[0511] The device sends the captured audio data to the server.

[0512] Step 4:

[0513] The server receives the audio data.

[0514] Step 5:

[0515] The server calls a voice recognition engine and converts the voice data into text data.

[0516] Step 6:

[0517] The server retrieves the converted text data and recognizes it as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0518] Step 7:

[0519] The server sends the text data to the natural language processing engine and begins analysis.

[0520] Step 8:

[0521] The server uses a natural language processing engine (e.g., spaCy or NLTK) to extract the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) from the text data.

[0522] Step 9:

[0523] The server checks the extracted information and formats it as needed, for example converting date information to a standard format (e.g. 2023-10-25).

[0524] Step 10:

[0525] The server connects to the database and formats the information to create a new record, which includes the task, target, quantity, and date and time.

[0526] Step 11:

[0527] The server periodically monitors the database and checks the due dates of registered records.

[0528] Step 12:

[0529] The server uses a scheduling system (e.g. a cron job) to schedule the task to run on a specified day.

[0530] Step 13:

[0531] When the server reaches the scheduled date (e.g., the 25th), it starts a process to execute the configured task.

[0532] Step 14:

[0533] The server accesses the bank's online transfer form.

[0534] Step 15:

[0535] The server uses an image recognition engine (e.g., Tesseract OCR) to identify each field on the transfer form.

[0536] Step 16:

[0537] The server uses an automated script to enter the transfer destination (Yamada Taro) and amount (5,000 yen) into the specified fields.

[0538] Step 17:

[0539] The server checks the input information, and if there are no problems, clicks the transfer button to execute the transfer.

[0540] Step 18:

[0541] The server captures the transfer success message and confirmation screen and verifies it using image recognition.

[0542] Step 19:

[0543] After the server confirms that the transfer was successful, it summarizes the results.

[0544] Step 20:

[0545] The server converts the results into a format that can be sent to the user's device (e.g., push notification, email).

[0546] Step 21:

[0547] The terminal receives the notification from the server and notifies the user that the transfer has been completed.

[0548] Step 22:

[0549] The user checks the notification and confirms that the transfer was successful.

[0550] Example 1

[0551] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0552] Conventional task schedule management systems require users to manually set dates and times and execute tasks at the appropriate times, resulting in inefficiencies and a high likelihood of errors. Furthermore, the inability to set tasks via voice input hinders user convenience. Furthermore, when automatically executing specific tasks, such as payment processing, security and reliability issues remain.

[0553] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0554] In this invention, the server includes means for acquiring voice input from a terminal, means for converting the acquired voice data into text data, means for analyzing the task, target, amount, and date and time from the text data, means for registering the analyzed information in a database, means for executing the task based on a preset date and time, means for automatically accessing the target information processing system and making input based on the analyzed information, and means for notifying the user of the execution result. This allows the user to easily set a task via voice input, and the task is automatically executed at the specified date and time, significantly reducing the user's effort and enabling the task to be completed efficiently and securely.

[0555] "Terminal" refers to an electronic device for receiving voice input and transmitting data to a server.

[0556] "Voice input" refers to the use of voice to provide instructions and information to a system.

[0557] "Server" refers to a central processing unit for processing received voice data and managing the automated execution of tasks.

[0558] "Voice data" refers to data that is a digital representation of a voice signal captured by a device.

[0559] "Text data" refers to data in the form of a character string converted by speech recognition.

[0560] A "task" refers to an action or process that is to be performed at a specific date and time.

[0561] "Target" refers to the recipient or location involved in the execution of a task.

[0562] "Quantity" refers to the numerical value or amount required to perform a task.

[0563] "Date and time" refers to the date and time the task should be performed.

[0564] "Natural language processing" refers to the technology of analyzing meaning and extracting information from voice or text data.

[0565] "Database" refers to a data storage device for storing and managing analyzed information.

[0566] "Scheduling system" refers to a system that manages and executes tasks based on specified dates and times.

[0567] "Information processing system" refers to a system that is the subject of automated task execution (e.g., an online banking system).

[0568] "Image recognition" refers to technology for extracting specific information or patterns from visual data.

[0569] "Automatic input" refers to the process of automatically inputting the necessary data into a system or application based on analyzed information.

[0570] "Notification" refers to the means of informing the user of the results of task execution (e.g., push notification or email).

[0571] "Payment processing" refers to the task of transferring a specific amount of money to a specified account.

[0572] This invention is a system that analyzes voice input and automatically executes tasks set for a specific date. This system is composed of a series of processes: voice recognition, natural language processing, database management, automatic task execution, and notification. Each process is described in detail below.

[0573] Voice input and recognition

[0574] The user speaks to the device, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device uses a microphone to capture the voice data and sends it to the server. The server then uses a speech recognition engine such as Google Cloud Speech-to-Text API to convert the received voice data into text data. This allows the user to easily set tasks with voice instructions, without having to perform complex operations.

[0575] Natural Language Processing and Data Analysis

[0576] The server then analyzes the converted text data using a natural language processing engine (e.g., spaCy). This analysis extracts important information such as the task (transfer), target (e.g., Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month). This process clearly analyzes the user's voice instructions and extracts the necessary information.

[0577] Data registration and management

[0578] The server registers the analyzed information in a database (e.g., PostgreSQL). It also uses a scheduling system (e.g., a cron job) to set up tasks to be executed at pre-set dates and times. At this stage, data relating tasks, targets, amounts, and dates and times is efficiently managed.

[0579] Preparing and running a task

[0580] When the scheduled date arrives, the server automatically executes the task. For example, in the case of a transfer task, the server uses an automation tool such as Selenium to access the bank's online form, identifies the form fields using Tesseract OCR, and enters the required information using an automated script. Next, the server checks the input and, if correct, presses the transfer button to execute the process. Finally, it captures a success message and confirms the completion of the transfer.

[0581] Result notification

[0582] When the task is successfully executed, the server summarizes the results and notifies the user via push notification or email, allowing the user to keep track of the progress and completion of the task in real time.

[0583] Specific examples

[0584] The user voice-inputs, "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device captures the voice and sends it to the server. The server converts the voice to text using the Google Cloud Speech-to-Text API, extracting "Yamada Taro," "5,000 yen," and "25th of this month." The server registers the information in a database and schedules the action for the scheduled date. On the 25th, the server uses Selenium to access the bank's online form and executes the transfer using Tesseract OCR and an automated script. Once the transfer is complete, the server summarizes the results and notifies the user via SMTP. This allows the user to fully manage their tasks without any hassle.

[0585] Prompt Sentence Examples

[0586] An example of a prompt sentence to be input to the generative AI model is as follows.

[0587] "This system allows you to set tasks via voice input, and then automatically executes the tasks at the specified date and time. For example, if you say, 'Transfer 5,000 yen to Yamada Taro on the 25th of this month,' the system will automatically transfer the money on that date. Could you please tell me the specific processing steps of the program?"

[0588] This prompt allows you to obtain detailed explanations and necessary steps from the generative AI model.

[0589] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0590] Step 1: Capturing Audio Input

[0591] The user gives a voice command such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device uses a microphone to capture the voice data and converts it into a digital format. This digital voice data is then sent directly to the server as an HTTP request. The input to the device is a voice signal, and the output is the converted digital voice data.

[0592] Step 2: Sending and Recognizing Voice Data

[0593] The server receives the digital audio data and sends it to the Google Cloud Speech-to-Text API, which converts the audio data into text. The server's input is digital audio data and its output is text data.

[0594] Step 3: Analyzing the text data

[0595] The text data received by the server is analyzed using a natural language processing engine (for example, spaCy). The analysis process extracts information such as the task content (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month). This process uses natural language processing technology to understand the text data and identify the necessary information. The server inputs text data and outputs analyzed information (task, target, amount, date and time).

[0596] Step 4: Registering and scheduling data

[0597] The server registers the analyzed information in a database. This information includes the task content, target, amount, and date and time. It also uses a scheduling system (for example, a cron job) to set up the task to run based on the analyzed date and time. The server's input is the analyzed information, and its output is confirmation of registration in the database and task scheduling settings.

[0598] Step 5: Automate Task Execution

[0599] At the scheduled time, the server will start a process to execute the scheduled task, which includes the following actions:

[0600] 1. The server uses an automation tool such as Selenium to access the bank's online form.

[0601] 2. The server uses Tesseract OCR to identify the fields on the transfer form.

[0602] 3. The server uses an automated script to enter the transfer destination (Yamada Taro) and amount (5,000 yen) into the form.

[0603] 4. The server checks the input information, and if it is correct, press the send button to execute the transfer.

[0604] 5. The server captures the successful transfer message and uses Tesseract OCR to confirm the transfer was successful.

[0605] The server's input is the scheduled task information, and its output is the result of the transfer execution.

[0606] Step 6: Notification of results

[0607] When the task is completed successfully, the server aggregates the execution results and notifies the user. Notification methods include push notification and email. Specifically, the server sends an email using the SMTP protocol or a push notification using Firebase Cloud Messaging (FCM). The input to the server is the task execution result, and the output is a notification to the user.

[0608] Each step of this system allows users to easily set tasks via voice input and have them executed automatically at the specified date and time, saving time and effort and enabling tasks to be completed efficiently.

[0609] (Application example 1)

[0610] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0611] In modern life, manually scheduling and executing everyday tasks, such as making payments, can be extremely cumbersome and time-consuming. There is a particular need for systems that can easily set up and automatically execute these tasks using voice commands. However, current systems are often complex to operate and unreliable. Therefore, there is an urgent need to improve the accuracy and convenience of task automation systems that use voice commands.

[0612] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0613] In this invention, the server includes means for acquiring voice input from a terminal, means for converting the acquired voice data into text data, means for analyzing the task, target, amount, and date and time from the text data, means for registering the analyzed information in a database, means for executing the task based on a preset date and time, means for automatically accessing the target system and making input based on the analyzed information, means for notifying the execution result, a script for processing the scheduled task, means for starting an automatic script for automatically executing a payment task at a specified date and time, and means for analyzing information extracted from the voice input using a generative AI model. This enables highly accurate and efficient task setting and automatic execution using voice instructions.

[0614] A "terminal" is a device that allows a user to input voice, such as a smartphone or smart glasses.

[0615] The "means for acquiring voice input" is a function for capturing voice data using the microphone of the terminal and transmitting the voice data to the system.

[0616] The term "means for converting voice data into text data" refers to a process of converting voice data into character string data using a voice recognition engine.

[0617] "Means of analyzing tasks, targets, quantities, and dates" refers to the process of using a natural language processing engine to extract important information from text data.

[0618] "Means for registering in a database" is a function that stores analyzed information in a database so that it can be used later.

[0619] "Means for executing a task based on a preset date and time" refers to a process for automatically executing a task at a specified date and time using a scheduling system.

[0620] "Means for automatically accessing and inputting information into the target system" refers to a function that uses a script or API to automatically access the target system and input the necessary information.

[0621] "Means for notifying the execution results" refers to push notification or email sending functions for notifying the user of the execution results of the task.

[0622] "Script for processing scheduled tasks" refers to program code that is executed to automatically process pre-planned tasks.

[0623] The "means for starting an automatic script for automatically executing a payment task at a specified date and time" is a function for starting a script for automatically executing a payment task at a specified date and time.

[0624] "Means for analyzing information extracted from voice input using a generative AI model" refers to the process of using a generative AI model to perform detailed analysis of data obtained from voice input and extract necessary information.

[0625] This invention is a system that allows a user to set a task using voice input through a terminal and automatically executes the task at a specified date and time. This system is constructed by combining multiple specific hardware and software.

[0626] First, users use a smartphone or smart glasses as hardware for voice input. The device's microphone is used to capture voice input, and speech recognition software such as the Google Cloud Speech-to-Text API is used to convert the speech into text data.

[0627] Next, the server analyzes the received text data using a natural language processing engine (e.g., spaCy, NLTK), extracting important information such as the task, target, amount, and date and time. This analyzed information is then stored in an SQL database.

[0628] Based on the extracted information, the server uses a scheduling system (e.g., cron job, APScheduler) to set up a process to automatically execute the task at a pre-set date and time. Specifically, the server starts a script to automatically execute the payment task, and uses an image recognition engine (e.g., Tesseract OCR) to identify input fields and input information using the automatic script.

[0629] Furthermore, the information extracted from the voice input is analyzed with high accuracy using a generative AI model, which improves task execution accuracy and reduces the burden on the user.

[0630] Finally, if the task is successfully executed, the server summarizes the execution results and notifies the user. This notification is done using push notification or an email sending service (e.g., Amazon SES).

[0631] As a concrete example, if a user gives a voice command such as "Pay the water bill of 3,000 yen on the 25th of this month," this voice is captured by the device and converted into text data by a speech recognition engine. Then, a natural language processing engine extracts information such as "payment task," "water bill," "3,000 yen," and "25th of this month," and registers it in a database. The server then automatically executes the payment procedure at the specified date and time, and notifies the user of the results.

[0632] An example of a prompt sentence is, "Pay the water bill of 3,000 yen on the 25th of this month," and extract the task, target, amount, and date and time from the voice command.

[0633] This system allows users to easily and efficiently set tasks using voice instructions, and the tasks are then automatically executed at the specified date and time, significantly reducing the amount of work required.

[0634] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0635] Step 1:

[0636] The user speaks into the device

[0637] Specific operation: The user gives voice instructions to their smartphone or smart glasses, such as "Pay the water bill of 3,000 yen on the 25th of this month."

[0638] Input: Audio data

[0639] Output: Audio data

[0640] Step 2:

[0641] The device captures the audio data and sends it to the server.

[0642] Specific operation: The device's microphone captures audio and sends the audio data to the Google Cloud Speech-to-Text API.

[0643] Input: Audio data

[0644] Output: Text data

[0645] Step 3:

[0646] The server uses a speech recognition engine to convert the voice data into text data.

[0647] Specific operation: The Google Cloud Speech-to-Text API converts the audio data into text data and sends it back to the server.

[0648] Input: Audio data

[0649] Output: Text data

[0650] Step 4:

[0651] The server analyzes the text data and extracts the task, target, amount, and date and time.

[0652] Specific operation: The server uses a natural language processing engine such as spacy or NLTK to analyze the text data and extract important information (task, target, quantity, date and time).

[0653] Input: Text data

[0654] Output: Task, target, amount, date and time

[0655] Step 5:

[0656] The server registers the analyzed information in a database

[0657] Specific operation: The server stores the extracted information in an SQL database.

[0658] Input: Task, Target, Amount, Date and Time

[0659] Output: Database records

[0660] Step 6:

[0661] The server starts the scheduling system to set up scheduled tasks.

[0662] Specific operation: The server will use APScheduler or a cron job to configure the task to run at the parsed date and time.

[0663] Input: Task, Target, Amount, Date and Time

[0664] Output: Scheduled Job

[0665] Step 7:

[0666] The server will automatically execute the payment task at the specified date and time.

[0667] What happens: At the specified date and time, the server launches an automated script that contacts the payment system to perform the task, using an image recognition engine (such as Tesseract OCR) to identify input fields, enter target and quantity information, and perform verification.

[0668] Input: Task, Target, Amount, Date and Time

[0669] Output: Payment made

[0670] Step 8:

[0671] The server notifies the user of the execution result.

[0672] Specific operation: When the payment is successfully completed, the server notifies the user of the result using a push notification or email sending service (such as Amazon SES).

[0673] Input: Payment execution result

[0674] Output: Notification message

[0675] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0676] The present invention combines a system that analyzes voice input and automatically executes tasks set for specific dates with an emotion engine. In addition to speech recognition, natural language processing, database management, automatic task execution, and notifications, the system also includes a process that recognizes the user's emotions and adjusts the priority and execution method of tasks based on those emotions. Each process is described in detail below.

[0677] Voice input and recognition

[0678] The user gives voice instructions to the terminal, such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0679] The device uses a microphone to capture audio data and transmits it to the server.

[0680] The server uses a speech recognition engine to convert the received voice data into text data. At this stage, speech recognition services such as Google Cloud Speech-to-Text API are used.

[0681] Natural Language Processing and Data Analysis

[0682] The server processes the converted text data and analyzes it using a natural language processing engine (e.g., spaCy or NLTK). As a result of this analysis, the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) are extracted.

[0683] emotion recognition

[0684] The server uses an emotion engine to analyze the user's emotions from the voice input, using factors such as voice tone, speed, and intonation, for example, to analyze the level of stress or tension in the user's voice.

[0685] Data registration and management

[0686] The server registers the analyzed information in a database, where the task, target, amount, date and time, and emotion information are stored as associated records. The server also uses a scheduling system (e.g., a cron job) to set tasks to be executed at pre-set dates and times.

[0687] Preparing and running a task

[0688] When the due date arrives, the server starts the process to execute the task. For example, in the case of a bank transfer, the following steps are taken:

[0689] 1. The server accesses the bank's online form.

[0690] 2. The server uses an image recognition engine (e.g., Tesseract OCR) to identify the fields on the transfer form.

[0691] 3. The server uses an automatic script to input the transfer destination (Yamada Taro) and the amount (5,000 yen). For users with high stress, the server automatically checks their balance.

[0692] 4. The server will check the entered information, and if it is correct, press the transfer button to execute the transfer.

[0693] 5. The server captures the transfer success message and confirmation screen and uses image recognition to confirm the transfer completion.

[0694] Result notification

[0695] When the task is successfully completed, the server summarizes the results and notifies the user. This notification includes an appropriate message based on the analysis results of the emotion engine. For example, a user experiencing high stress may receive a positive, relaxing message.

[0696] Specific examples

[0697] The user speaks, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0698] The device captures the audio and sends it to the server.

[0699] The server converts the speech into text using a speech recognition engine, and extracts "Yamada Taro," "5,000 yen," and "25th of this month."

[0700] The server uses an emotion engine to analyze the stress the user is feeling.

[0701] The server registers the information in a database and schedules the action on a scheduled date.

[0702] On the 25th, the server automatically accesses the bank's online form and executes the transfer using image recognition and an automated script.

[0703] Once the transfer is complete, the server compiles the results and notifies the user with a stress-reducing message.

[0704] The user checks the notification and finds out that the transfer was successful.

[0705] This system allows users to easily set tasks via voice input, and the tasks are automatically executed at the specified date and time, significantly reducing the effort required for operation. In addition, by combining it with an emotion engine, it is possible to respond flexibly to the user's emotional state, improving user satisfaction.

[0706] The processing flow will be explained below.

[0707] Step 1:

[0708] The user speaks into the terminal, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0709] Step 2:

[0710] The device uses the microphone to capture the user's voice input.

[0711] Step 3:

[0712] The device sends the captured audio data to the server.

[0713] Step 4:

[0714] The server receives the audio data.

[0715] Step 5:

[0716] The server calls a voice recognition engine and converts the voice data into text data.

[0717] Step 6:

[0718] The server retrieves the converted text data and recognizes it as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0719] Step 7:

[0720] The server sends the text data to the natural language processing engine and begins analysis.

[0721] Step 8:

[0722] The server uses a natural language processing engine (e.g., spaCy or NLTK) to extract the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) from the text data.

[0723] Step 9:

[0724] The server uses an emotion engine to analyze the user's emotions from the voice input, using voice tone, speed, intonation, etc. For example, analyzing the user's voice for stress or tension levels.

[0725] Step 10:

[0726] The server formats the parsed information and converts date information into a standard format (e.g., 2023-10-25).

[0727] Step 11:

[0728] The server connects to the database and registers the formatted information as a new record, which includes the task, target, amount, date and time, and emotional information.

[0729] Step 12:

[0730] The server periodically monitors the database for the due dates of new records.

[0731] Step 13:

[0732] The server uses a scheduling system (e.g. a cron job) to schedule the task to run on a specified day.

[0733] Step 14:

[0734] When the server reaches the scheduled date (e.g., the 25th), it starts a process to execute the configured task.

[0735] Step 15:

[0736] The server accesses the bank's online transfer form.

[0737] Step 16:

[0738] The server uses an image recognition engine (e.g., Tesseract OCR) to identify each field on the transfer form.

[0739] Step 17:

[0740] The server uses an automated script to enter the transfer recipient (Yamada Taro) and amount (5,000 yen) into the specified fields.

[0741] Step 18:

[0742] The server checks the input information, and if there are no problems, clicks the transfer button to execute the transfer.

[0743] Step 19:

[0744] The server captures the transfer success message and confirmation screen and verifies it using image recognition.

[0745] Step 20:

[0746] The server compiles the results and generates appropriate messages based on the user's emotional information. For example, it selects a message with a relaxing effect for a user experiencing high stress.

[0747] Step 21:

[0748] The server converts the results into a format that can be sent to the user's device (e.g., push notification, email).

[0749] Step 22:

[0750] The terminal receives a notification from the server and notifies the user that the transfer has been completed. The notification includes a message that reflects the user's emotions.

[0751] Step 23:

[0752] The user checks the notification and finds out that the transfer was successful. The notification message is tailored to the user's emotions, increasing user satisfaction.

[0753] Example 2

[0754] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0755] Conventional voice input systems have the drawback of causing a lot of effort and stress when users execute tasks set for a specific date. For important tasks such as bank transfers and reservations, manual operation errors and emotional stress are particularly problematic. Furthermore, the system lacks the flexibility to respond to the user's emotional state, leaving room for improvement in user satisfaction.

[0756] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring voice input, a means for converting the acquired voice data into text data, a means for analyzing the text data to identify the task, target, quantity, and date and time, a means for registering the analyzed information in a database, a means for executing the task based on a preset date and time, a means for automatically accessing the target system and performing input based on the analyzed information, a means for notifying the user of the execution result, and a means for analyzing emotional information from the voice data and adjusting the priority and execution method of the task. This allows the user to easily set a task via voice input, and the task is automatically executed at the specified date and time, significantly reducing the effort required for operation. Furthermore, by combining an emotion engine, flexible responses based on the user's emotional state are possible, thereby improving user satisfaction.

[0757] The "means for acquiring voice input" is a function for capturing voice uttered by a user as a digital signal through the device's microphone and transmitting it to a subsequent processing step.

[0758] The "means for converting captured voice data into text data" is a function that includes a process for recognizing captured voice data and converting it into a corresponding text format.

[0759] "Means for analyzing tasks, targets, quantities, and dates and times from text data" refers to a function for analyzing text data using natural language processing technology and extracting information on the specified tasks, targets, quantities, and dates and times.

[0760] The "means for registering analyzed information in a database" is a function for storing extracted task-related information in a database and managing it so that it can be used later.

[0761] The "means for executing a task based on a preset date and time" is a function that automatically starts and executes a task at a specified date and time based on task information stored in a database.

[0762] "Means for automatically accessing the target system and entering data based on analyzed information" refers to a function that automates access to an external system required to execute a task and enters appropriate data based on analyzed information.

[0763] The "means for notifying the user of the execution result" is a function for sending a notification to inform the user of the execution status and result of the task.

[0764] "Means for analyzing emotional information from voice data and adjusting task priority and execution method" is a function that analyzes the user's emotional state at the time of voice input and adjusts task priority and execution method according to the user's emotions.

[0765] The present invention combines an emotion engine with a system that analyzes voice input and automatically executes tasks set for specific dates. This system is configured using the following hardware and software.

[0766] Acquiring and converting voice input

[0767] The user speaks to the terminal, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The terminal uses a microphone to capture the voice data and sends it to the server.

[0768] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API, which provides highly accurate speech recognition and enables real-time conversion.

[0769] Natural Language Processing and Data Analysis

[0770] The server analyzes the converted text data using a natural language processing engine (e.g., spaCy or NLTK). Through this analysis, the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) are extracted.

[0771] Emotion analysis

[0772] The server uses an emotion engine to analyze the user's emotions from the voice input. This emotion analysis uses the tone, speed, and intonation of the voice. For example, Google Cloud Natural Language API and IBM Watson Tone Analyzer are used to analyze the level of stress and tension from the voice.

[0773] Data registration and management

[0774] The server registers the analyzed information in a database, which includes the task, target, amount, date and time, and emotion information, using a database management system such as MySQL or PostgreSQL. The server also uses a scheduling system (e.g., cron job) to set tasks to be executed at predetermined dates and times.

[0775] Executing a task

[0776] When the specified date and time arrives, the server will start executing the task. For example, in the case of a bank transfer, the following steps are taken:

[0777] 1. The server uses the Selenium library in Python to access the bank's online form.

[0778] 2. The server uses an image recognition engine (e.g., Tesseract OCR) to identify the fields in the transfer form, and then uses an automatic script to enter the transfer recipient (Yamada Taro) and the amount (5,000 yen). A setting to automatically check the balance is added, especially for users with high stress levels.

[0779] 3. The server will check the entered information, and if it is correct, press the transfer button to execute the transfer.

[0780] 4. The server captures the transfer success message and confirmation screen and uses image recognition to confirm the transfer completion.

[0781] Notification of results

[0782] After the server completes the task, it notifies the user of the result. The notification includes an appropriate message based on the results of the sentiment analysis. For example, a user experiencing high stress might receive a positive, relaxing message such as, "The transfer has been successfully completed. Relax and have a wonderful day."

[0783] Specific examples

[0784] The user gives a voice command such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device captures the voice and sends it to the server. The server converts the voice data into text using the Google Cloud Speech-to-Text API. The server analyzes the text using a natural language processing engine and extracts "Yamada Taro," "5,000 yen," and "the 25th of this month." The server uses an emotion engine to analyze the user's stress level. The server registers the information in a database and schedules the necessary actions for the specified date and time. On the 25th, the server automatically accesses the bank's online form and performs the transfer using Selenium and Tesseract OCR. When the transfer is complete, the server summarizes the results and notifies the user. The user checks the notification and learns that the transfer was successful.

[0785] By implementing this system, users can easily set tasks via voice commands, and the tasks will be automatically executed at the specified date and time, significantly reducing the amount of work required. In addition, by combining it with an emotion engine, flexible responses according to the user's emotional state become possible, improving user satisfaction.

[0786] An example of a generative AI model and prompts to use is ChatGPT-4 (registered trademark), and examples of prompts include:

[0787] "Please explain in detail the process of your new system that combines an emotion engine with a system that automatically executes tasks based on voice input."

[0788] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0789] Processing flow

[0790] Step 1: Getting voice input

[0791] The user gives a voice command, and the device uses the microphone to capture the voice data and transmits it to the server.

[0792] Input: User's voice command (e.g., "Transfer 5,000 yen to Yamada Taro on the 25th of this month.")

[0793] Output: Captured audio data (digital format)

[0794] Specific operation: The user speaks into the microphone. The device's microphone captures the voice and generates real-time audio data, which is then sent to the server via the network.

[0795] Step 2: Convert audio data to text

[0796] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API.

[0797] Input: Captured audio data

[0798] Output: Converted text data (e.g. "Transfer 5,000 yen to Yamada Taro on the 25th of this month")

[0799] What happens: The server processes the audio data via the Google Cloud Speech-to-Text API and generates corresponding text data, which is passed on to the next processing step.

[0800] Step 3: Analyzing the text data

[0801] The server uses a natural language processing engine (e.g., spaCy or NLTK) to parse the text data and extract important information such as task, subject, amount, date and time.

[0802] Input: Converted text data

[0803] Output: Extracted information (Task = Transfer, Target = Yamada Taro, Amount = 5000 yen, Date and Time = 25th of this month)

[0804] How it works: The server passes the text data to a natural language processing engine, which performs grammatical and semantic analysis, extracting specific keywords (task, subject, quantity, date and time).

[0805] Step 4: Sentiment Analysis

[0806] The server uses an emotion engine to analyze the user's emotion from the voice input, including the tone, rate, and intonation of the voice.

[0807] Input: converted audio data, analyzed text data

[0808] Output: Emotional information (e.g., stress level = high)

[0809] Specific operation: The server passes voice and text data to the emotion engine, which analyzes emotions from these data. It analyzes the voice characteristics and determines the user's emotional state.

[0810] Step 5: Registering and scheduling data

[0811] The server registers the parsed information in a database and schedules the task to be executed using a scheduling system (e.g., a cron job).

[0812] Input: Extracted information, emotional information

[0813] Output: Task information registered in the database, set schedule

[0814] Specific operation: The server stores the analyzed task information and emotion information in a database and schedules the task to be executed at the specified date and time using a database management system and a scheduling tool.

[0815] Step 6: Task execution

[0816] When the set date and time arrives, the server starts the task execution process. For example, in the case of a transfer task, the server accesses the bank's online system and inputs the necessary information.

[0817] Input: Task information and emotion information registered in the database

[0818] Output: Task executed, confirmation message

[0819] What it does: The server uses Selenium to access the bank's online transfer form, uses Tesseract OCR to identify the required fields, and then uses an automated script to enter the transfer details and amount, verify the details, and execute the transfer.

[0820] Step 7: Notification of results

[0821] Once the task is completed, the server notifies the user of the results, including an appropriate message based on the emotion information.

[0822] Input: Results of the executed task, emotional information

[0823] Output: Notification message sent to the user

[0824] Specific operation: The server confirms the success of the task execution and generates a notification message based on that. It also takes into account emotional information and sends a message that corresponds to the user's state. For example, it could say, "The transfer has been successfully completed. Please relax and have a wonderful day."

[0825] (Application example 2)

[0826] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0827] Conventional automated task execution systems using voice input simply execute tasks without considering the user's emotional state. As a result, depending on the user's situation or stress level, appropriate responses are not made, and satisfaction cannot be sufficiently improved. Furthermore, flexible adjustments based on emotions are required for the execution of payment tasks, but no technology that can achieve this has existed.

[0828] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring voice input from the terminal, means for converting the acquired voice data into text data, means for analyzing the task, target, quantity, and date and time from the text data, means for registering the analyzed information in a database, means for executing the task based on a preset date and time, means for automatically accessing the target system and making input based on the analyzed information, means for analyzing the user's emotions using an emotion engine, means for adjusting the priority and execution method of the task based on the user's emotional state, and means for notifying the user of the execution result. This allows the execution of payment tasks via voice input to be flexibly adjusted according to the user's emotional state, thereby improving user satisfaction.

[0829] A "terminal" is an electronic device for obtaining voice input.

[0830] "Voice input" refers to voice data uttered by the user, and includes instructions and commands to the system.

[0831] "Audio Data" means a digital recording of audio input captured by a Device.

[0832] "Text data" is character string data obtained by analyzing voice data using a voice recognition engine.

[0833] A "task" refers to a specific action or operation that is executed based on a user's voice instruction.

[0834] A "subject" is an individual, system, or physical or digital entity involved in the performance of a task.

[0835] "Quantity" refers to the numerical values ​​or metrics required to perform a task.

[0836] "Date and time" refers to a specific date and time when a task is scheduled to be performed.

[0837] A "database" is a structured data storage system for storing and managing analyzed information.

[0838] "Automatic access" means that the system connects to online services or forms based on a pre-defined process without manual intervention.

[0839] "Entering" means automatically writing the analyzed information into the target system.

[0840] An "emotion engine" is a software or hardware module that analyzes a user's emotions from their voice input and identifies their emotional state.

[0841] "Emotional state" refers to the psychological state or stress level experienced by a user.

[0842] "Priority" refers to the execution order and importance of the analyzed task.

[0843] An "execution method" is a means for determining how a task should proceed depending on the user's emotional state.

[0844] "Notification" is the act of informing a user of the results or status of an executed task.

[0845] The embodiments of the present invention will now be described in detail.

[0846] System configuration

[0847] The system consists of the following major hardware and software components:

[0848] Device: Smartphone (iOS / Android)

[0849] Server: Speech recognition engine (Google Cloud Speech-to-Text API), natural language processing engine (spaCy), emotion analysis engine (e.g., voice tone analysis software), database (SQLite), task scheduling system

[0850] Network: The internet connection that connects your device to the server.

[0851] Main features of the program

[0852] The program has the function of automatically executing tasks through voice input, and adjusts task priority and execution method according to the user's emotional state. To achieve this, a complex system is constructed that goes through steps such as voice capture, voice recognition, natural language processing, emotion analysis, database registration, task scheduling, execution, and result notification.

[0853] Processing flow and technologies used

[0854] 1. Audio Capture

[0855] The user inputs voice using the microphone of the device.

[0856] The device captures the audio data and sends it to the server.

[0857] 2. Voice Recognition

[0858] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data.

[0859] For example, if the user instructs "Pay the electricity bill of 5,000 yen next Monday," the voice data is converted into text "Pay the electricity bill of 5,000 yen next Monday."

[0860] 3. Natural Language Processing

[0861] The converted text data is analyzed using spaCy to extract task details (date, amount, target).

[0862] In this example, "electricity bill," "5,000 yen," and "next Monday" are extracted.

[0863] 4. Emotion analysis

[0864] The server analyzes the emotional state from the voice input, analyzing the tone, speed, intonation, etc. with an emotion analysis engine.

[0865] For example, if the user's voice indicates that he or she is stressed, the user is recorded as "stressed."

[0866] 5. Database registration

[0867] The analyzed information is registered in a database, and the task, subject, amount, date and time, and emotional information are stored as associated records.

[0868] 6. Task Scheduling

[0869] Based on the registered information, the task is scheduled to be executed at a preset date and time.

[0870] For example, a task to automatically pay the electricity bill is scheduled for "next Monday."

[0871] 7. Task execution

[0872] At the specified date and time, the server will automatically make the online payment.

[0873] Based on emotional state, stressed users may be asked to check their balances in advance.

[0874] 8. Result notification

[0875] When the task is successfully completed, the user is notified of the result and a message is sent according to the user's emotional state, such as a relaxing message like "Thank you for your hard work, your electricity bill has been paid."

[0876] Examples of concrete examples and prompts

[0877] For example, if the user speaks something like this:

[0878] Today, I want to make sure I don't forget to transfer my rent, so please set it up so that the rent of 70,000 yen is automatically transferred to a specified account on the first day of each month. I'm a little nervous about asking for rent payments.

[0879] This system allows users to easily set and execute tasks, and enables flexible responses according to the user's emotional state.

[0880] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0881] Step 1:

[0882] The user uses the microphone on the device to input voice. For example, the user might say, "Please pay the electricity bill of 5,000 yen next Monday." The input to the device is voice data, which the device captures. The output here is the captured voice data.

[0883] Step 2:

[0884] The device sends the captured audio data to the server. The input is the audio data captured on the device, and the output to the server is the transmission of audio data. Once the server receives the audio data, it is ready to proceed to the next step.

[0885] Step 3:

[0886] The server uses the Google Cloud Speech-to-Text API to convert the voice data into text data. The input is the voice data, and the output is the converted text data. The server calls the speech recognition engine, analyzes the voice data, and converts it into text.

[0887] Step 4:

[0888] The server uses a natural language processing engine (spaCy) to analyze task details (date, amount, target) from text data. The input is text data, and the output is detailed task information (e.g., "electricity bill," "5,000 yen," "next Monday"). The server analyzes the text data and extracts important information.

[0889] Step 5:

[0890] The server uses an emotion analysis engine to analyze the user's emotional state from the voice data. The input is the initial voice data or text data, and the output is the emotional state (e.g., "stressed"). The server analyzes the voice tone, speed, and intonation to identify the emotional state.

[0891] Step 6:

[0892] The server registers the analyzed information in a database. The input is the analyzed task details and emotional state, and the output is a record saved in the database. The server stores the extracted information in the database as structured data.

[0893] Step 7:

[0894] The server schedules tasks to be executed based on a pre-set date and time. The input is the task details, execution date and time, and the output is the scheduled task. The server uses the task scheduling system to set up the task to be executed automatically on the specified date and time.

[0895] Step 8:

[0896] At the scheduled time, the server automatically executes the online payment. The inputs are task details, emotional state, and schedule information, and the output is confirmation of payment completion. Specifically, the server automatically accesses the target payment system, enters the necessary information (payee, amount) into the input form, and completes the payment.

[0897] Step 9:

[0898] When the task is successfully completed, the server notifies the user of the result. The input is the task execution status and emotional state, and the output is a notification message sent to the user. The server generates an appropriate message depending on the emotional state, notifying the user, for example, "Thank you for your hard work. Your electricity bill has been paid."

[0899] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0900] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0901] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0902] [Third embodiment]

[0903] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0904] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0905] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0906] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0907] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0908] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0909] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0910] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0911] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0912] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0913] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0914] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0915] The present invention is a system that analyzes voice input and automatically executes tasks set for a specific date. This system is composed of a series of processes: voice recognition, natural language processing, database management, automatic task execution, and notification. Each process is described in detail below.

[0916] Voice input and recognition

[0917] The user gives voice instructions to the terminal, such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0918] The device uses a microphone to capture audio data and transmits it to the server.

[0919] The server uses a speech recognition engine to convert the received voice data into text data. At this stage, speech recognition services such as Google Cloud Speech-to-Text API are used.

[0920] Natural Language Processing and Data Analysis

[0921] The server processes the converted text data and analyzes it using a natural language processing engine (e.g., spaCy or NLTK). As a result of this analysis, the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) are extracted.

[0922] Data registration and management

[0923] The server registers the analyzed information in a database, where the task, target, amount, and date and time are associated and saved as a record. The server also uses a scheduling system (e.g., a cron job) to set tasks to run at pre-determined dates and times.

[0924] Preparing and running a task

[0925] When the due date arrives, the server starts the process to execute the task. For example, in the case of a bank transfer, the following steps are taken:

[0926] 1. The server accesses the bank's online form.

[0927] 2. The server uses an image recognition engine (such as Tesseract OCR) to identify the fields on the transfer form.

[0928] 3. The server uses an automated script to enter the transfer destination (Yamada Taro) and amount (5,000 yen).

[0929] 4. The server will check the entered information, and if it is correct, press the transfer button to execute the transfer.

[0930] 5. The server captures the transfer success message and confirmation screen and uses image recognition to confirm the transfer completion.

[0931] Result notification

[0932] When the task is successfully executed, the server summarizes the results and notifies the user, using push notifications or email.

[0933] Specific examples

[0934] The user speaks, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0935] The device captures the audio and sends it to the server.

[0936] The server converts the speech into text using a speech recognition engine, and extracts "Yamada Taro," "5,000 yen," and "25th of this month."

[0937] The server registers the information in a database and schedules the action on a scheduled date.

[0938] On the 25th, the server automatically accesses the bank's online form and executes the transfer using image recognition and an automated script.

[0939] Once the transfer is complete, the server will summarize the results and notify the user.

[0940] The user checks the notification and finds out that the transfer has been completed.

[0941] This system allows users to easily set tasks via voice input, and the tasks are automatically executed at the specified date and time, significantly reducing the user's workload.

[0942] The processing flow will be explained below.

[0943] Step 1:

[0944] The user speaks into the terminal, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0945] Step 2:

[0946] The device uses the microphone to capture the user's voice input.

[0947] Step 3:

[0948] The device sends the captured audio data to the server.

[0949] Step 4:

[0950] The server receives the audio data.

[0951] Step 5:

[0952] The server calls a voice recognition engine and converts the voice data into text data.

[0953] Step 6:

[0954] The server retrieves the converted text data and recognizes it as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[0955] Step 7:

[0956] The server sends the text data to the natural language processing engine and begins analysis.

[0957] Step 8:

[0958] The server uses a natural language processing engine (e.g., spaCy or NLTK) to extract the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) from the text data.

[0959] Step 9:

[0960] The server checks the extracted information and formats it as needed, for example converting date information to a standard format (e.g. 2023-10-25).

[0961] Step 10:

[0962] The server connects to the database and formats the information to create a new record, which includes the task, target, quantity, and date and time.

[0963] Step 11:

[0964] The server periodically monitors the database and checks the due dates of registered records.

[0965] Step 12:

[0966] The server uses a scheduling system (e.g. a cron job) to schedule the task to run on a specified day.

[0967] Step 13:

[0968] When the server reaches the scheduled date (e.g., the 25th), it starts a process to execute the configured task.

[0969] Step 14:

[0970] The server accesses the bank's online transfer form.

[0971] Step 15:

[0972] The server uses an image recognition engine (e.g., Tesseract OCR) to identify each field on the transfer form.

[0973] Step 16:

[0974] The server uses an automated script to enter the transfer destination (Yamada Taro) and amount (5,000 yen) into the specified fields.

[0975] Step 17:

[0976] The server checks the input information, and if there are no problems, clicks the transfer button to execute the transfer.

[0977] Step 18:

[0978] The server captures the transfer success message and confirmation screen and verifies it using image recognition.

[0979] Step 19:

[0980] After the server confirms that the transfer was successful, it summarizes the results.

[0981] Step 20:

[0982] The server converts the results into a format that can be sent to the user's device (e.g., push notification, email).

[0983] Step 21:

[0984] The terminal receives the notification from the server and notifies the user that the transfer has been completed.

[0985] Step 22:

[0986] The user checks the notification and confirms that the transfer was successful.

[0987] Example 1

[0988] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0989] Conventional task schedule management systems require users to manually set dates and times and execute tasks at the appropriate times, resulting in inefficiencies and a high likelihood of errors. Furthermore, the inability to set tasks via voice input hinders user convenience. Furthermore, when automatically executing specific tasks, such as payment processing, security and reliability issues remain.

[0990] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0991] In this invention, the server includes means for acquiring voice input from a terminal, means for converting the acquired voice data into text data, means for analyzing the task, target, amount, and date and time from the text data, means for registering the analyzed information in a database, means for executing the task based on a preset date and time, means for automatically accessing the target information processing system and making input based on the analyzed information, and means for notifying the user of the execution result. This allows the user to easily set a task via voice input, and the task is automatically executed at the specified date and time, significantly reducing the user's effort and enabling the task to be completed efficiently and securely.

[0992] "Terminal" refers to an electronic device for receiving voice input and transmitting data to a server.

[0993] "Voice input" refers to the use of voice to provide instructions and information to a system.

[0994] "Server" refers to a central processing unit for processing received voice data and managing the automated execution of tasks.

[0995] "Voice data" refers to data that is a digital representation of a voice signal captured by a device.

[0996] "Text data" refers to data in the form of a character string converted by speech recognition.

[0997] A "task" refers to an action or process that is to be performed at a specific date and time.

[0998] "Target" refers to the recipient or location involved in the execution of a task.

[0999] "Quantity" refers to the numerical value or amount required to perform a task.

[1000] "Date and time" refers to the date and time the task should be performed.

[1001] "Natural language processing" refers to the technology of analyzing meaning and extracting information from voice or text data.

[1002] "Database" refers to a data storage device for storing and managing analyzed information.

[1003] "Scheduling system" refers to a system that manages and executes tasks based on specified dates and times.

[1004] "Information processing system" refers to a system that is the subject of automated task execution (e.g., an online banking system).

[1005] "Image recognition" refers to technology for extracting specific information or patterns from visual data.

[1006] "Automatic input" refers to the process of automatically inputting the necessary data into a system or application based on analyzed information.

[1007] "Notification" refers to the means of informing the user of the results of task execution (e.g., push notification or email).

[1008] "Payment processing" refers to the task of transferring a specific amount of money to a specified account.

[1009] This invention is a system that analyzes voice input and automatically executes tasks set for a specific date. This system is composed of a series of processes: voice recognition, natural language processing, database management, automatic task execution, and notification. Each process is described in detail below.

[1010] Voice input and recognition

[1011] The user speaks to the device, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device uses a microphone to capture the voice data and sends it to the server. The server then uses a speech recognition engine such as Google Cloud Speech-to-Text API to convert the received voice data into text data. This allows the user to easily set tasks with voice instructions, without having to perform complex operations.

[1012] Natural Language Processing and Data Analysis

[1013] The server then analyzes the converted text data using a natural language processing engine (e.g., spaCy). This analysis extracts important information such as the task (transfer), target (e.g., Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month). This process clearly analyzes the user's voice instructions and extracts the necessary information.

[1014] Data registration and management

[1015] The server registers the analyzed information in a database (e.g., PostgreSQL). It also uses a scheduling system (e.g., a cron job) to set up tasks to be executed at pre-set dates and times. At this stage, data relating tasks, targets, amounts, and dates and times is efficiently managed.

[1016] Preparing and running a task

[1017] When the scheduled date arrives, the server automatically executes the task. For example, in the case of a transfer task, the server uses an automation tool such as Selenium to access the bank's online form, identifies the form fields using Tesseract OCR, and enters the required information using an automated script. Next, the server checks the input and, if correct, presses the transfer button to execute the process. Finally, it captures a success message and confirms the completion of the transfer.

[1018] Result notification

[1019] When the task is successfully executed, the server summarizes the results and notifies the user via push notification or email, allowing the user to keep track of the progress and completion of the task in real time.

[1020] Specific examples

[1021] The user voice-inputs, "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device captures the voice and sends it to the server. The server converts the voice to text using the Google Cloud Speech-to-Text API, extracting "Yamada Taro," "5,000 yen," and "25th of this month." The server registers the information in a database and schedules the action for the scheduled date. On the 25th, the server uses Selenium to access the bank's online form and executes the transfer using Tesseract OCR and an automated script. Once the transfer is complete, the server summarizes the results and notifies the user via SMTP. This allows the user to fully manage their tasks without any hassle.

[1022] Prompt Sentence Examples

[1023] An example of a prompt sentence to be input to the generative AI model is as follows.

[1024] "This system allows you to set tasks via voice input, and then automatically executes the tasks at the specified date and time. For example, if you say, 'Transfer 5,000 yen to Yamada Taro on the 25th of this month,' the system will automatically transfer the money on that date. Could you please tell me the specific processing steps of the program?"

[1025] This prompt allows you to obtain detailed explanations and necessary steps from the generative AI model.

[1026] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1027] Step 1: Capturing Audio Input

[1028] The user gives a voice command such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device uses a microphone to capture the voice data and converts it into a digital format. This digital voice data is then sent directly to the server as an HTTP request. The input to the device is a voice signal, and the output is the converted digital voice data.

[1029] Step 2: Sending and Recognizing Voice Data

[1030] The server receives the digital audio data and sends it to the Google Cloud Speech-to-Text API, which converts the audio data into text. The server's input is digital audio data and its output is text data.

[1031] Step 3: Analyzing the text data

[1032] The text data received by the server is analyzed using a natural language processing engine (for example, spaCy). The analysis process extracts information such as the task content (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month). This process uses natural language processing technology to understand the text data and identify the necessary information. The server inputs text data and outputs analyzed information (task, target, amount, date and time).

[1033] Step 4: Registering and scheduling data

[1034] The server registers the analyzed information in a database. This information includes the task content, target, amount, and date and time. It also uses a scheduling system (for example, a cron job) to set up the task to run based on the analyzed date and time. The server's input is the analyzed information, and its output is confirmation of registration in the database and task scheduling settings.

[1035] Step 5: Automate Task Execution

[1036] At the scheduled time, the server will start a process to execute the scheduled task, which includes the following actions:

[1037] 1. The server uses an automation tool such as Selenium to access the bank's online form.

[1038] 2. The server uses Tesseract OCR to identify the fields on the transfer form.

[1039] 3. The server uses an automated script to enter the transfer destination (Yamada Taro) and amount (5,000 yen) into the form.

[1040] 4. The server checks the input information, and if it is correct, press the send button to execute the transfer.

[1041] 5. The server captures the successful transfer message and uses Tesseract OCR to confirm the transfer was successful.

[1042] The server's input is the scheduled task information, and its output is the result of the transfer execution.

[1043] Step 6: Notification of results

[1044] When the task is completed successfully, the server aggregates the execution results and notifies the user. Notification methods include push notification and email. Specifically, the server sends an email using the SMTP protocol or a push notification using Firebase Cloud Messaging (FCM). The input to the server is the task execution result, and the output is a notification to the user.

[1045] Each step of this system allows users to easily set tasks via voice input and have them executed automatically at the specified date and time, saving time and effort and enabling tasks to be completed efficiently.

[1046] (Application example 1)

[1047] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1048] In modern life, manually scheduling and executing everyday tasks, such as making payments, can be extremely cumbersome and time-consuming. There is a particular need for systems that can easily set up and automatically execute these tasks using voice commands. However, current systems are often complex to operate and unreliable. Therefore, there is an urgent need to improve the accuracy and convenience of task automation systems that use voice commands.

[1049] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1050] In this invention, the server includes means for acquiring voice input from a terminal, means for converting the acquired voice data into text data, means for analyzing the task, target, amount, and date and time from the text data, means for registering the analyzed information in a database, means for executing the task based on a preset date and time, means for automatically accessing the target system and making input based on the analyzed information, means for notifying the execution result, a script for processing the scheduled task, means for starting an automatic script for automatically executing a payment task at a specified date and time, and means for analyzing information extracted from the voice input using a generative AI model. This enables highly accurate and efficient task setting and automatic execution using voice instructions.

[1051] A "terminal" is a device that allows a user to input voice, such as a smartphone or smart glasses.

[1052] The "means for acquiring voice input" is a function for capturing voice data using the microphone of the terminal and transmitting the voice data to the system.

[1053] The term "means for converting voice data into text data" refers to a process of converting voice data into character string data using a voice recognition engine.

[1054] "Means of analyzing tasks, targets, quantities, and dates" refers to the process of using a natural language processing engine to extract important information from text data.

[1055] "Means for registering in a database" is a function that stores analyzed information in a database so that it can be used later.

[1056] "Means for executing a task based on a preset date and time" refers to a process for automatically executing a task at a specified date and time using a scheduling system.

[1057] "Means for automatically accessing and inputting information into the target system" refers to a function that uses a script or API to automatically access the target system and input the necessary information.

[1058] "Means for notifying the execution results" refers to push notification or email sending functions for notifying the user of the execution results of the task.

[1059] "Script for processing scheduled tasks" refers to program code that is executed to automatically process pre-planned tasks.

[1060] The "means for starting an automatic script for automatically executing a payment task at a specified date and time" is a function for starting a script for automatically executing a payment task at a specified date and time.

[1061] "Means for analyzing information extracted from voice input using a generative AI model" refers to the process of using a generative AI model to perform detailed analysis of data obtained from voice input and extract necessary information.

[1062] This invention is a system that allows a user to set a task using voice input through a terminal and automatically executes the task at a specified date and time. This system is constructed by combining multiple specific hardware and software.

[1063] First, users use a smartphone or smart glasses as hardware for voice input. The device's microphone is used to capture voice input, and speech recognition software such as the Google Cloud Speech-to-Text API is used to convert the speech into text data.

[1064] Next, the server analyzes the received text data using a natural language processing engine (e.g., spaCy, NLTK), extracting important information such as the task, target, amount, and date and time. This analyzed information is then stored in an SQL database.

[1065] Based on the extracted information, the server uses a scheduling system (e.g., cron job, APScheduler) to set up a process to automatically execute the task at a pre-set date and time. Specifically, the server starts a script to automatically execute the payment task, and uses an image recognition engine (e.g., Tesseract OCR) to identify input fields and input information using the automatic script.

[1066] Furthermore, the information extracted from the voice input is analyzed with high accuracy using a generative AI model, which improves task execution accuracy and reduces the burden on the user.

[1067] Finally, if the task is successfully executed, the server summarizes the execution results and notifies the user. This notification is done using push notification or an email sending service (e.g., Amazon SES).

[1068] As a concrete example, if a user gives a voice command such as "Pay the water bill of 3,000 yen on the 25th of this month," this voice is captured by the device and converted into text data by a speech recognition engine. Then, a natural language processing engine extracts information such as "payment task," "water bill," "3,000 yen," and "25th of this month," and registers it in a database. The server then automatically executes the payment procedure at the specified date and time, and notifies the user of the results.

[1069] An example of a prompt sentence is, "Pay the water bill of 3,000 yen on the 25th of this month," and extract the task, target, amount, and date and time from the voice command.

[1070] This system allows users to easily and efficiently set tasks using voice instructions, and the tasks are then automatically executed at the specified date and time, significantly reducing the amount of work required.

[1071] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1072] Step 1:

[1073] The user speaks into the device

[1074] Specific operation: The user gives voice instructions to their smartphone or smart glasses, such as "Pay the water bill of 3,000 yen on the 25th of this month."

[1075] Input: Audio data

[1076] Output: Audio data

[1077] Step 2:

[1078] The device captures the audio data and sends it to the server.

[1079] Specific operation: The device's microphone captures audio and sends the audio data to the Google Cloud Speech-to-Text API.

[1080] Input: Audio data

[1081] Output: Text data

[1082] Step 3:

[1083] The server uses a speech recognition engine to convert the voice data into text data.

[1084] Specific operation: The Google Cloud Speech-to-Text API converts the audio data into text data and sends it back to the server.

[1085] Input: Audio data

[1086] Output: Text data

[1087] Step 4:

[1088] The server analyzes the text data and extracts the task, target, amount, and date and time.

[1089] Specific operation: The server uses a natural language processing engine such as spacy or NLTK to analyze the text data and extract important information (task, target, quantity, date and time).

[1090] Input: Text data

[1091] Output: Task, target, amount, date and time

[1092] Step 5:

[1093] The server registers the analyzed information in a database

[1094] Specific operation: The server stores the extracted information in an SQL database.

[1095] Input: Task, Target, Amount, Date and Time

[1096] Output: Database records

[1097] Step 6:

[1098] The server starts the scheduling system to set up scheduled tasks.

[1099] Specific operation: The server will use APScheduler or a cron job to configure the task to run at the parsed date and time.

[1100] Input: Task, Target, Amount, Date and Time

[1101] Output: Scheduled Job

[1102] Step 7:

[1103] The server will automatically execute the payment task at the specified date and time.

[1104] What happens: At the specified date and time, the server launches an automated script that contacts the payment system to perform the task, using an image recognition engine (such as Tesseract OCR) to identify input fields, enter target and quantity information, and perform verification.

[1105] Input: Task, Target, Amount, Date and Time

[1106] Output: Payment made

[1107] Step 8:

[1108] The server notifies the user of the execution result.

[1109] Specific operation: When the payment is successfully completed, the server notifies the user of the result using a push notification or email sending service (such as Amazon SES).

[1110] Input: Payment execution result

[1111] Output: Notification message

[1112] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1113] The present invention combines a system that analyzes voice input and automatically executes tasks set for specific dates with an emotion engine. In addition to speech recognition, natural language processing, database management, automatic task execution, and notifications, the system also includes a process that recognizes the user's emotions and adjusts the priority and execution method of tasks based on those emotions. Each process is described in detail below.

[1114] Voice input and recognition

[1115] The user gives voice instructions to the terminal, such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[1116] The device uses a microphone to capture audio data and transmits it to the server.

[1117] The server uses a speech recognition engine to convert the received voice data into text data. At this stage, speech recognition services such as Google Cloud Speech-to-Text API are used.

[1118] Natural Language Processing and Data Analysis

[1119] The server processes the converted text data and analyzes it using a natural language processing engine (e.g., spaCy or NLTK). As a result of this analysis, the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) are extracted.

[1120] emotion recognition

[1121] The server uses an emotion engine to analyze the user's emotions from the voice input, using factors such as voice tone, speed, and intonation, for example, to analyze the level of stress or tension in the user's voice.

[1122] Data registration and management

[1123] The server registers the analyzed information in a database, where the task, target, amount, date and time, and emotion information are stored as associated records. The server also uses a scheduling system (e.g., a cron job) to set tasks to be executed at pre-set dates and times.

[1124] Preparing and running a task

[1125] When the due date arrives, the server starts the process to execute the task. For example, in the case of a bank transfer, the following steps are taken:

[1126] 1. The server accesses the bank's online form.

[1127] 2. The server uses an image recognition engine (e.g., Tesseract OCR) to identify the fields on the transfer form.

[1128] 3. The server uses an automatic script to input the transfer destination (Yamada Taro) and the amount (5,000 yen). For users with high stress, the server automatically checks their balance.

[1129] 4. The server will check the entered information, and if it is correct, press the transfer button to execute the transfer.

[1130] 5. The server captures the transfer success message and confirmation screen and uses image recognition to confirm the transfer completion.

[1131] Result notification

[1132] When the task is successfully completed, the server summarizes the results and notifies the user. This notification includes an appropriate message based on the analysis results of the emotion engine. For example, a user experiencing high stress may receive a positive, relaxing message.

[1133] Specific examples

[1134] The user speaks, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[1135] The device captures the audio and sends it to the server.

[1136] The server converts the speech into text using a speech recognition engine, and extracts "Yamada Taro," "5,000 yen," and "25th of this month."

[1137] The server uses an emotion engine to analyze the stress the user is feeling.

[1138] The server registers the information in a database and schedules the action on a scheduled date.

[1139] On the 25th, the server automatically accesses the bank's online form and executes the transfer using image recognition and an automated script.

[1140] Once the transfer is complete, the server compiles the results and notifies the user with a stress-reducing message.

[1141] The user checks the notification and finds out that the transfer was successful.

[1142] This system allows users to easily set tasks via voice input, and the tasks are automatically executed at the specified date and time, significantly reducing the effort required for operation. In addition, by combining it with an emotion engine, it is possible to respond flexibly to the user's emotional state, improving user satisfaction.

[1143] The processing flow will be explained below.

[1144] Step 1:

[1145] The user speaks into the terminal, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[1146] Step 2:

[1147] The device uses the microphone to capture the user's voice input.

[1148] Step 3:

[1149] The device sends the captured audio data to the server.

[1150] Step 4:

[1151] The server receives the audio data.

[1152] Step 5:

[1153] The server calls a voice recognition engine and converts the voice data into text data.

[1154] Step 6:

[1155] The server retrieves the converted text data and recognizes it as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[1156] Step 7:

[1157] The server sends the text data to the natural language processing engine and begins analysis.

[1158] Step 8:

[1159] The server uses a natural language processing engine (e.g., spaCy or NLTK) to extract the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) from the text data.

[1160] Step 9:

[1161] The server uses an emotion engine to analyze the user's emotions from the voice input, using voice tone, speed, intonation, etc. For example, analyzing the user's voice for stress or tension levels.

[1162] Step 10:

[1163] The server formats the parsed information and converts date information into a standard format (e.g., 2023-10-25).

[1164] Step 11:

[1165] The server connects to the database and registers the formatted information as a new record, which includes the task, target, amount, date and time, and emotional information.

[1166] Step 12:

[1167] The server periodically monitors the database for the due dates of new records.

[1168] Step 13:

[1169] The server uses a scheduling system (e.g. a cron job) to schedule the task to run on a specified day.

[1170] Step 14:

[1171] When the server reaches the scheduled date (e.g., the 25th), it starts a process to execute the configured task.

[1172] Step 15:

[1173] The server accesses the bank's online transfer form.

[1174] Step 16:

[1175] The server uses an image recognition engine (e.g., Tesseract OCR) to identify each field on the transfer form.

[1176] Step 17:

[1177] The server uses an automated script to enter the transfer recipient (Yamada Taro) and amount (5,000 yen) into the specified fields.

[1178] Step 18:

[1179] The server checks the input information, and if there are no problems, clicks the transfer button to execute the transfer.

[1180] Step 19:

[1181] The server captures the transfer success message and confirmation screen and verifies it using image recognition.

[1182] Step 20:

[1183] The server compiles the results and generates appropriate messages based on the user's emotional information. For example, it selects a message with a relaxing effect for a user experiencing high stress.

[1184] Step 21:

[1185] The server converts the results into a format that can be sent to the user's device (e.g., push notification, email).

[1186] Step 22:

[1187] The terminal receives a notification from the server and notifies the user that the transfer has been completed. The notification includes a message that reflects the user's emotions.

[1188] Step 23:

[1189] The user checks the notification and finds out that the transfer was successful. The notification message is tailored to the user's emotions, increasing user satisfaction.

[1190] Example 2

[1191] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1192] Conventional voice input systems have the drawback of causing a lot of effort and stress when users execute tasks set for a specific date. For important tasks such as bank transfers and reservations, manual operation errors and emotional stress are particularly problematic. Furthermore, the system lacks the flexibility to respond to the user's emotional state, leaving room for improvement in user satisfaction.

[1193] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring voice input, a means for converting the acquired voice data into text data, a means for analyzing the text data to identify the task, target, quantity, and date and time, a means for registering the analyzed information in a database, a means for executing the task based on a preset date and time, a means for automatically accessing the target system and performing input based on the analyzed information, a means for notifying the user of the execution result, and a means for analyzing emotional information from the voice data and adjusting the priority and execution method of the task. This allows the user to easily set a task via voice input, and the task is automatically executed at the specified date and time, significantly reducing the effort required for operation. Furthermore, by combining an emotion engine, flexible responses based on the user's emotional state are possible, thereby improving user satisfaction.

[1194] The "means for acquiring voice input" is a function for capturing voice uttered by a user as a digital signal through the device's microphone and transmitting it to a subsequent processing step.

[1195] The "means for converting captured voice data into text data" is a function that includes a process for recognizing captured voice data and converting it into a corresponding text format.

[1196] "Means for analyzing tasks, targets, quantities, and dates and times from text data" refers to a function for analyzing text data using natural language processing technology and extracting information on the specified tasks, targets, quantities, and dates and times.

[1197] The "means for registering analyzed information in a database" is a function for storing extracted task-related information in a database and managing it so that it can be used later.

[1198] The "means for executing a task based on a preset date and time" is a function that automatically starts and executes a task at a specified date and time based on task information stored in a database.

[1199] "Means for automatically accessing the target system and entering data based on analyzed information" refers to a function that automates access to an external system required to execute a task and enters appropriate data based on analyzed information.

[1200] The "means for notifying the user of the execution result" is a function for sending a notification to inform the user of the execution status and result of the task.

[1201] "Means for analyzing emotional information from voice data and adjusting task priority and execution method" is a function that analyzes the user's emotional state at the time of voice input and adjusts task priority and execution method according to the user's emotions.

[1202] The present invention combines an emotion engine with a system that analyzes voice input and automatically executes tasks set for specific dates. This system is configured using the following hardware and software.

[1203] Acquiring and converting voice input

[1204] The user speaks to the terminal, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The terminal uses a microphone to capture the voice data and sends it to the server.

[1205] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API, which provides highly accurate speech recognition and enables real-time conversion.

[1206] Natural Language Processing and Data Analysis

[1207] The server analyzes the converted text data using a natural language processing engine (e.g., spaCy or NLTK). Through this analysis, the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) are extracted.

[1208] Emotion analysis

[1209] The server uses an emotion engine to analyze the user's emotions from the voice input. This emotion analysis uses the tone, speed, and intonation of the voice. For example, Google Cloud Natural Language API and IBM Watson Tone Analyzer are used to analyze the level of stress and tension from the voice.

[1210] Data registration and management

[1211] The server registers the analyzed information in a database, which includes the task, target, amount, date and time, and emotion information, using a database management system such as MySQL or PostgreSQL. The server also uses a scheduling system (e.g., cron job) to set tasks to be executed at predetermined dates and times.

[1212] Executing a task

[1213] When the specified date and time arrives, the server will start executing the task. For example, in the case of a bank transfer, the following steps are taken:

[1214] 1. The server uses the Selenium library in Python to access the bank's online form.

[1215] 2. The server uses an image recognition engine (e.g., Tesseract OCR) to identify the fields in the transfer form, and then uses an automatic script to enter the transfer recipient (Yamada Taro) and the amount (5,000 yen). A setting to automatically check the balance is added, especially for users with high stress levels.

[1216] 3. The server will check the entered information, and if it is correct, press the transfer button to execute the transfer.

[1217] 4. The server captures the transfer success message and confirmation screen and uses image recognition to confirm the transfer completion.

[1218] Notification of results

[1219] After the server completes the task, it notifies the user of the result. The notification includes an appropriate message based on the results of the sentiment analysis. For example, a user experiencing high stress might receive a positive, relaxing message such as, "The transfer has been successfully completed. Relax and have a wonderful day."

[1220] Specific examples

[1221] The user gives a voice command such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device captures the voice and sends it to the server. The server converts the voice data into text using the Google Cloud Speech-to-Text API. The server analyzes the text using a natural language processing engine and extracts "Yamada Taro," "5,000 yen," and "the 25th of this month." The server uses an emotion engine to analyze the user's stress level. The server registers the information in a database and schedules the necessary actions for the specified date and time. On the 25th, the server automatically accesses the bank's online form and performs the transfer using Selenium and Tesseract OCR. When the transfer is complete, the server summarizes the results and notifies the user. The user checks the notification and learns that the transfer was successful.

[1222] By implementing this system, users can easily set tasks via voice commands, and the tasks will be automatically executed at the specified date and time, significantly reducing the amount of work required. In addition, by combining it with an emotion engine, flexible responses according to the user's emotional state become possible, improving user satisfaction.

[1223] An example of a generative AI model and prompts to use is ChatGPT-4, and examples of prompts include:

[1224] "Please explain in detail the process of your new system that combines an emotion engine with a system that automatically executes tasks based on voice input."

[1225] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1226] Processing flow

[1227] Step 1: Getting voice input

[1228] The user gives a voice command, and the device uses the microphone to capture the voice data and transmits it to the server.

[1229] Input: User's voice command (e.g., "Transfer 5,000 yen to Yamada Taro on the 25th of this month.")

[1230] Output: Captured audio data (digital format)

[1231] Specific operation: The user speaks into the microphone. The device's microphone captures the voice and generates real-time audio data, which is then sent to the server via the network.

[1232] Step 2: Convert audio data to text

[1233] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API.

[1234] Input: Captured audio data

[1235] Output: Converted text data (e.g. "Transfer 5,000 yen to Yamada Taro on the 25th of this month")

[1236] What happens: The server processes the audio data via the Google Cloud Speech-to-Text API and generates corresponding text data, which is passed on to the next processing step.

[1237] Step 3: Analyzing the text data

[1238] The server uses a natural language processing engine (e.g., spaCy or NLTK) to parse the text data and extract important information such as task, subject, amount, date and time.

[1239] Input: Converted text data

[1240] Output: Extracted information (Task = Transfer, Target = Yamada Taro, Amount = 5000 yen, Date and Time = 25th of this month)

[1241] How it works: The server passes the text data to a natural language processing engine, which performs grammatical and semantic analysis, extracting specific keywords (task, subject, quantity, date and time).

[1242] Step 4: Sentiment Analysis

[1243] The server uses an emotion engine to analyze the user's emotion from the voice input, including the tone, rate, and intonation of the voice.

[1244] Input: converted audio data, analyzed text data

[1245] Output: Emotional information (e.g., stress level = high)

[1246] Specific operation: The server passes voice and text data to the emotion engine, which analyzes emotions from these data. It analyzes the voice characteristics and determines the user's emotional state.

[1247] Step 5: Registering and scheduling data

[1248] The server registers the parsed information in a database and schedules the task to be executed using a scheduling system (e.g., a cron job).

[1249] Input: Extracted information, emotional information

[1250] Output: Task information registered in the database, set schedule

[1251] Specific operation: The server stores the analyzed task information and emotion information in a database and schedules the task to be executed at the specified date and time using a database management system and a scheduling tool.

[1252] Step 6: Task execution

[1253] When the set date and time arrives, the server starts the task execution process. For example, in the case of a transfer task, the server accesses the bank's online system and inputs the necessary information.

[1254] Input: Task information and emotion information registered in the database

[1255] Output: Task executed, confirmation message

[1256] What it does: The server uses Selenium to access the bank's online transfer form, uses Tesseract OCR to identify the required fields, and then uses an automated script to enter the transfer details and amount, verify the details, and execute the transfer.

[1257] Step 7: Notification of results

[1258] Once the task is completed, the server notifies the user of the results, including an appropriate message based on the emotion information.

[1259] Input: Results of the executed task, emotional information

[1260] Output: Notification message sent to the user

[1261] Specific operation: The server confirms the success of the task execution and generates a notification message based on that. It also takes into account emotional information and sends a message that corresponds to the user's state. For example, it could say, "The transfer has been successfully completed. Please relax and have a wonderful day."

[1262] (Application example 2)

[1263] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1264] Conventional automated task execution systems using voice input simply execute tasks without considering the user's emotional state. As a result, depending on the user's situation or stress level, appropriate responses are not made, and satisfaction cannot be sufficiently improved. Furthermore, flexible adjustments based on emotions are required for the execution of payment tasks, but no technology that can achieve this has existed.

[1265] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring voice input from the terminal, means for converting the acquired voice data into text data, means for analyzing the task, target, quantity, and date and time from the text data, means for registering the analyzed information in a database, means for executing the task based on a preset date and time, means for automatically accessing the target system and making input based on the analyzed information, means for analyzing the user's emotions using an emotion engine, means for adjusting the priority and execution method of the task based on the user's emotional state, and means for notifying the user of the execution result. This allows the execution of payment tasks via voice input to be flexibly adjusted according to the user's emotional state, thereby improving user satisfaction.

[1266] A "terminal" is an electronic device for obtaining voice input.

[1267] "Voice input" refers to voice data uttered by the user, and includes instructions and commands to the system.

[1268] "Audio Data" means a digital recording of audio input captured by a Device.

[1269] "Text data" is character string data obtained by analyzing voice data using a voice recognition engine.

[1270] A "task" refers to a specific action or operation that is executed based on a user's voice instruction.

[1271] A "subject" is an individual, system, or physical or digital entity involved in the performance of a task.

[1272] "Quantity" refers to the numerical values ​​or metrics required to perform a task.

[1273] "Date and time" refers to a specific date and time when a task is scheduled to be performed.

[1274] A "database" is a structured data storage system for storing and managing analyzed information.

[1275] "Automatic access" means that the system connects to online services or forms based on a pre-defined process without manual intervention.

[1276] "Entering" means automatically writing the analyzed information into the target system.

[1277] An "emotion engine" is a software or hardware module that analyzes a user's emotions from their voice input and identifies their emotional state.

[1278] "Emotional state" refers to the psychological state or stress level experienced by a user.

[1279] "Priority" refers to the execution order and importance of the analyzed task.

[1280] An "execution method" is a means for determining how a task should proceed depending on the user's emotional state.

[1281] "Notification" is the act of informing a user of the results or status of an executed task.

[1282] The embodiments of the present invention will now be described in detail.

[1283] System configuration

[1284] The system consists of the following major hardware and software components:

[1285] Device: Smartphone (iOS / Android)

[1286] Server: Speech recognition engine (Google Cloud Speech-to-Text API), natural language processing engine (spaCy), emotion analysis engine (e.g., voice tone analysis software), database (SQLite), task scheduling system

[1287] Network: The internet connection that connects your device to the server.

[1288] Main features of the program

[1289] The program has the function of automatically executing tasks through voice input, and adjusts task priority and execution method according to the user's emotional state. To achieve this, a complex system is constructed that goes through steps such as voice capture, voice recognition, natural language processing, emotion analysis, database registration, task scheduling, execution, and result notification.

[1290] Processing flow and technologies used

[1291] 1. Audio Capture

[1292] The user inputs voice using the microphone of the device.

[1293] The device captures the audio data and sends it to the server.

[1294] 2. Voice Recognition

[1295] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data.

[1296] For example, if the user instructs "Pay the electricity bill of 5,000 yen next Monday," the voice data is converted into text "Pay the electricity bill of 5,000 yen next Monday."

[1297] 3. Natural Language Processing

[1298] The converted text data is analyzed using spaCy to extract task details (date, amount, target).

[1299] In this example, "electricity bill," "5,000 yen," and "next Monday" are extracted.

[1300] 4. Emotion analysis

[1301] The server analyzes the emotional state from the voice input, analyzing the tone, speed, intonation, etc. with an emotion analysis engine.

[1302] For example, if the user's voice indicates that he or she is stressed, the user is recorded as "stressed."

[1303] 5. Database registration

[1304] The analyzed information is registered in a database, and the task, subject, amount, date and time, and emotional information are stored as associated records.

[1305] 6. Task Scheduling

[1306] Based on the registered information, the task is scheduled to be executed at a preset date and time.

[1307] For example, a task to automatically pay the electricity bill is scheduled for "next Monday."

[1308] 7. Task execution

[1309] At the specified date and time, the server will automatically make the online payment.

[1310] Based on emotional state, stressed users may be asked to check their balances in advance.

[1311] 8. Result notification

[1312] When the task is successfully completed, the user is notified of the result and a message is sent according to the user's emotional state, such as a relaxing message like "Thank you for your hard work, your electricity bill has been paid."

[1313] Examples of concrete examples and prompts

[1314] For example, if the user speaks something like this:

[1315] Today, I want to make sure I don't forget to transfer my rent, so please set it up so that the rent of 70,000 yen is automatically transferred to a specified account on the first day of each month. I'm a little nervous about asking for rent payments.

[1316] This system allows users to easily set and execute tasks, and enables flexible responses according to the user's emotional state.

[1317] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1318] Step 1:

[1319] The user uses the microphone on the device to input voice. For example, the user might say, "Please pay the electricity bill of 5,000 yen next Monday." The input to the device is voice data, which the device captures. The output here is the captured voice data.

[1320] Step 2:

[1321] The device sends the captured audio data to the server. The input is the audio data captured on the device, and the output to the server is the transmission of audio data. Once the server receives the audio data, it is ready to proceed to the next step.

[1322] Step 3:

[1323] The server uses the Google Cloud Speech-to-Text API to convert the voice data into text data. The input is the voice data, and the output is the converted text data. The server calls the speech recognition engine, analyzes the voice data, and converts it into text.

[1324] Step 4:

[1325] The server uses a natural language processing engine (spaCy) to analyze task details (date, amount, target) from text data. The input is text data, and the output is detailed task information (e.g., "electricity bill," "5,000 yen," "next Monday"). The server analyzes the text data and extracts important information.

[1326] Step 5:

[1327] The server uses an emotion analysis engine to analyze the user's emotional state from the voice data. The input is the initial voice data or text data, and the output is the emotional state (e.g., "stressed"). The server analyzes the voice tone, speed, and intonation to identify the emotional state.

[1328] Step 6:

[1329] The server registers the analyzed information in a database. The input is the analyzed task details and emotional state, and the output is a record saved in the database. The server stores the extracted information in the database as structured data.

[1330] Step 7:

[1331] The server schedules tasks to be executed based on a pre-set date and time. The input is the task details, execution date and time, and the output is the scheduled task. The server uses the task scheduling system to set up the task to be executed automatically on the specified date and time.

[1332] Step 8:

[1333] At the scheduled time, the server automatically executes the online payment. The inputs are task details, emotional state, and schedule information, and the output is confirmation of payment completion. Specifically, the server automatically accesses the target payment system, enters the necessary information (payee, amount) into the input form, and completes the payment.

[1334] Step 9:

[1335] When the task is successfully completed, the server notifies the user of the result. The input is the task execution status and emotional state, and the output is a notification message sent to the user. The server generates an appropriate message depending on the emotional state, notifying the user, for example, "Thank you for your hard work. Your electricity bill has been paid."

[1336] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1337] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1338] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1339] [Fourth embodiment]

[1340] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1341] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1342] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1343] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1344] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1345] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1346] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1347] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1348] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1349] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1350] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1351] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1352] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1353] The present invention is a system that analyzes voice input and automatically executes tasks set for a specific date. This system is composed of a series of processes: voice recognition, natural language processing, database management, automatic task execution, and notification. Each process is described in detail below.

[1354] Voice input and recognition

[1355] The user gives voice instructions to the terminal, such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[1356] The device uses a microphone to capture audio data and transmits it to the server.

[1357] The server uses a speech recognition engine to convert the received voice data into text data. At this stage, speech recognition services such as Google Cloud Speech-to-Text API are used.

[1358] Natural Language Processing and Data Analysis

[1359] The server processes the converted text data and analyzes it using a natural language processing engine (e.g., spaCy or NLTK). As a result of this analysis, the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) are extracted.

[1360] Data registration and management

[1361] The server registers the analyzed information in a database, where the task, target, amount, and date and time are associated and saved as a record. The server also uses a scheduling system (e.g., a cron job) to set tasks to run at pre-determined dates and times.

[1362] Preparing and running a task

[1363] When the due date arrives, the server starts the process to execute the task. For example, in the case of a bank transfer, the following steps are taken:

[1364] 1. The server accesses the bank's online form.

[1365] 2. The server uses an image recognition engine (such as Tesseract OCR) to identify the fields on the transfer form.

[1366] 3. The server uses an automated script to enter the transfer destination (Yamada Taro) and amount (5,000 yen).

[1367] 4. The server will check the entered information, and if it is correct, press the transfer button to execute the transfer.

[1368] 5. The server captures the transfer success message and confirmation screen and uses image recognition to confirm the transfer completion.

[1369] Result notification

[1370] When the task is successfully executed, the server summarizes the results and notifies the user, using push notifications or email.

[1371] Specific examples

[1372] The user speaks, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[1373] The device captures the audio and sends it to the server.

[1374] The server converts the speech into text using a speech recognition engine, and extracts "Yamada Taro," "5,000 yen," and "25th of this month."

[1375] The server registers the information in a database and schedules the action on a scheduled date.

[1376] On the 25th, the server automatically accesses the bank's online form and executes the transfer using image recognition and an automated script.

[1377] Once the transfer is complete, the server will summarize the results and notify the user.

[1378] The user checks the notification and finds out that the transfer has been completed.

[1379] This system allows users to easily set tasks via voice input, and the tasks are automatically executed at the specified date and time, significantly reducing the user's workload.

[1380] The processing flow will be explained below.

[1381] Step 1:

[1382] The user speaks into the terminal, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[1383] Step 2:

[1384] The device uses the microphone to capture the user's voice input.

[1385] Step 3:

[1386] The device sends the captured audio data to the server.

[1387] Step 4:

[1388] The server receives the audio data.

[1389] Step 5:

[1390] The server calls a voice recognition engine and converts the voice data into text data.

[1391] Step 6:

[1392] The server retrieves the converted text data and recognizes it as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[1393] Step 7:

[1394] The server sends the text data to the natural language processing engine and begins analysis.

[1395] Step 8:

[1396] The server uses a natural language processing engine (e.g., spaCy or NLTK) to extract the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) from the text data.

[1397] Step 9:

[1398] The server checks the extracted information and formats it as needed, for example converting date information to a standard format (e.g. 2023-10-25).

[1399] Step 10:

[1400] The server connects to the database and formats the information to create a new record, which includes the task, target, quantity, and date and time.

[1401] Step 11:

[1402] The server periodically monitors the database and checks the due dates of registered records.

[1403] Step 12:

[1404] The server uses a scheduling system (e.g. a cron job) to schedule the task to run on a specified day.

[1405] Step 13:

[1406] When the server reaches the scheduled date (e.g., the 25th), it starts a process to execute the configured task.

[1407] Step 14:

[1408] The server accesses the bank's online transfer form.

[1409] Step 15:

[1410] The server uses an image recognition engine (e.g., Tesseract OCR) to identify each field on the transfer form.

[1411] Step 16:

[1412] The server uses an automated script to enter the transfer destination (Yamada Taro) and amount (5,000 yen) into the specified fields.

[1413] Step 17:

[1414] The server checks the input information, and if there are no problems, clicks the transfer button to execute the transfer.

[1415] Step 18:

[1416] The server captures the transfer success message and confirmation screen and verifies it using image recognition.

[1417] Step 19:

[1418] After the server confirms that the transfer was successful, it summarizes the results.

[1419] Step 20:

[1420] The server converts the results into a format that can be sent to the user's device (e.g., push notification, email).

[1421] Step 21:

[1422] The terminal receives the notification from the server and notifies the user that the transfer has been completed.

[1423] Step 22:

[1424] The user checks the notification and confirms that the transfer was successful.

[1425] Example 1

[1426] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1427] Conventional task schedule management systems require users to manually set dates and times and execute tasks at the appropriate times, resulting in inefficiencies and a high likelihood of errors. Furthermore, the inability to set tasks via voice input hinders user convenience. Furthermore, when automatically executing specific tasks, such as payment processing, security and reliability issues remain.

[1428] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1429] In this invention, the server includes means for acquiring voice input from a terminal, means for converting the acquired voice data into text data, means for analyzing the task, target, amount, and date and time from the text data, means for registering the analyzed information in a database, means for executing the task based on a preset date and time, means for automatically accessing the target information processing system and making input based on the analyzed information, and means for notifying the user of the execution result. This allows the user to easily set a task via voice input, and the task is automatically executed at the specified date and time, significantly reducing the user's effort and enabling the task to be completed efficiently and securely.

[1430] "Terminal" refers to an electronic device for receiving voice input and transmitting data to a server.

[1431] "Voice input" refers to the use of voice to provide instructions and information to a system.

[1432] "Server" refers to a central processing unit for processing received voice data and managing the automated execution of tasks.

[1433] "Voice data" refers to data that is a digital representation of a voice signal captured by a device.

[1434] "Text data" refers to data in the form of a character string converted by speech recognition.

[1435] A "task" refers to an action or process that is to be performed at a specific date and time.

[1436] "Target" refers to the recipient or location involved in the execution of a task.

[1437] "Quantity" refers to the numerical value or amount required to perform a task.

[1438] "Date and time" refers to the date and time the task should be performed.

[1439] "Natural language processing" refers to the technology of analyzing meaning and extracting information from voice or text data.

[1440] "Database" refers to a data storage device for storing and managing analyzed information.

[1441] "Scheduling system" refers to a system that manages and executes tasks based on specified dates and times.

[1442] "Information processing system" refers to a system that is the subject of automated task execution (e.g., an online banking system).

[1443] "Image recognition" refers to technology for extracting specific information or patterns from visual data.

[1444] "Automatic input" refers to the process of automatically inputting the necessary data into a system or application based on analyzed information.

[1445] "Notification" refers to the means of informing the user of the results of task execution (e.g., push notification or email).

[1446] "Payment processing" refers to the task of transferring a specific amount of money to a specified account.

[1447] This invention is a system that analyzes voice input and automatically executes tasks set for a specific date. This system is composed of a series of processes: voice recognition, natural language processing, database management, automatic task execution, and notification. Each process is described in detail below.

[1448] Voice input and recognition

[1449] The user speaks to the device, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device uses a microphone to capture the voice data and sends it to the server. The server then uses a speech recognition engine such as Google Cloud Speech-to-Text API to convert the received voice data into text data. This allows the user to easily set tasks with voice instructions, without having to perform complex operations.

[1450] Natural Language Processing and Data Analysis

[1451] The server then analyzes the converted text data using a natural language processing engine (e.g., spaCy). This analysis extracts important information such as the task (transfer), target (e.g., Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month). This process clearly analyzes the user's voice instructions and extracts the necessary information.

[1452] Data registration and management

[1453] The server registers the analyzed information in a database (e.g., PostgreSQL). It also uses a scheduling system (e.g., a cron job) to set up tasks to be executed at pre-set dates and times. At this stage, data relating tasks, targets, amounts, and dates and times is efficiently managed.

[1454] Preparing and running a task

[1455] When the scheduled date arrives, the server automatically executes the task. For example, in the case of a transfer task, the server uses an automation tool such as Selenium to access the bank's online form, identifies the form fields using Tesseract OCR, and enters the required information using an automated script. Next, the server checks the input and, if correct, presses the transfer button to execute the process. Finally, it captures a success message and confirms the completion of the transfer.

[1456] Result notification

[1457] When the task is successfully executed, the server summarizes the results and notifies the user via push notification or email, allowing the user to keep track of the progress and completion of the task in real time.

[1458] Specific examples

[1459] The user voice-inputs, "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device captures the voice and sends it to the server. The server converts the voice to text using the Google Cloud Speech-to-Text API, extracting "Yamada Taro," "5,000 yen," and "25th of this month." The server registers the information in a database and schedules the action for the scheduled date. On the 25th, the server uses Selenium to access the bank's online form and executes the transfer using Tesseract OCR and an automated script. Once the transfer is complete, the server summarizes the results and notifies the user via SMTP. This allows the user to fully manage their tasks without any hassle.

[1460] Prompt Sentence Examples

[1461] An example of a prompt sentence to be input to the generative AI model is as follows.

[1462] "This system allows you to set tasks via voice input, and then automatically executes the tasks at the specified date and time. For example, if you say, 'Transfer 5,000 yen to Yamada Taro on the 25th of this month,' the system will automatically transfer the money on that date. Could you please tell me the specific processing steps of the program?"

[1463] This prompt allows you to obtain detailed explanations and necessary steps from the generative AI model.

[1464] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1465] Step 1: Capturing Audio Input

[1466] The user gives a voice command such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device uses a microphone to capture the voice data and converts it into a digital format. This digital voice data is then sent directly to the server as an HTTP request. The input to the device is a voice signal, and the output is the converted digital voice data.

[1467] Step 2: Sending and Recognizing Voice Data

[1468] The server receives the digital audio data and sends it to the Google Cloud Speech-to-Text API, which converts the audio data into text. The server's input is digital audio data and its output is text data.

[1469] Step 3: Analyzing the text data

[1470] The text data received by the server is analyzed using a natural language processing engine (for example, spaCy). The analysis process extracts information such as the task content (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month). This process uses natural language processing technology to understand the text data and identify the necessary information. The server inputs text data and outputs analyzed information (task, target, amount, date and time).

[1471] Step 4: Registering and scheduling data

[1472] The server registers the analyzed information in a database. This information includes the task content, target, amount, and date and time. It also uses a scheduling system (for example, a cron job) to set up the task to run based on the analyzed date and time. The server's input is the analyzed information, and its output is confirmation of registration in the database and task scheduling settings.

[1473] Step 5: Automate Task Execution

[1474] At the scheduled time, the server will start a process to execute the scheduled task, which includes the following actions:

[1475] 1. The server uses an automation tool such as Selenium to access the bank's online form.

[1476] 2. The server uses Tesseract OCR to identify the fields on the transfer form.

[1477] 3. The server uses an automated script to enter the transfer destination (Yamada Taro) and amount (5,000 yen) into the form.

[1478] 4. The server checks the input information, and if it is correct, press the send button to execute the transfer.

[1479] 5. The server captures the successful transfer message and uses Tesseract OCR to confirm the transfer was successful.

[1480] The server's input is the scheduled task information, and its output is the result of the transfer execution.

[1481] Step 6: Notification of results

[1482] When the task is completed successfully, the server aggregates the execution results and notifies the user. Notification methods include push notification and email. Specifically, the server sends an email using the SMTP protocol or a push notification using Firebase Cloud Messaging (FCM). The input to the server is the task execution result, and the output is a notification to the user.

[1483] Each step of this system allows users to easily set tasks via voice input and have them executed automatically at the specified date and time, saving time and effort and enabling tasks to be completed efficiently.

[1484] (Application example 1)

[1485] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1486] In modern life, manually scheduling and executing everyday tasks, such as making payments, can be extremely cumbersome and time-consuming. There is a particular need for systems that can easily set up and automatically execute these tasks using voice commands. However, current systems are often complex to operate and unreliable. Therefore, there is an urgent need to improve the accuracy and convenience of task automation systems that use voice commands.

[1487] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1488] In this invention, the server includes means for acquiring voice input from a terminal, means for converting the acquired voice data into text data, means for analyzing the task, target, amount, and date and time from the text data, means for registering the analyzed information in a database, means for executing the task based on a preset date and time, means for automatically accessing the target system and making input based on the analyzed information, means for notifying the execution result, a script for processing the scheduled task, means for starting an automatic script for automatically executing a payment task at a specified date and time, and means for analyzing information extracted from the voice input using a generative AI model. This enables highly accurate and efficient task setting and automatic execution using voice instructions.

[1489] A "terminal" is a device that allows a user to input voice, such as a smartphone or smart glasses.

[1490] The "means for acquiring voice input" is a function for capturing voice data using the microphone of the terminal and transmitting the voice data to the system.

[1491] The term "means for converting voice data into text data" refers to a process of converting voice data into character string data using a voice recognition engine.

[1492] "Means of analyzing tasks, targets, quantities, and dates" refers to the process of using a natural language processing engine to extract important information from text data.

[1493] "Means for registering in a database" is a function that stores analyzed information in a database so that it can be used later.

[1494] "Means for executing a task based on a preset date and time" refers to a process for automatically executing a task at a specified date and time using a scheduling system.

[1495] "Means for automatically accessing and inputting information into the target system" refers to a function that uses a script or API to automatically access the target system and input the necessary information.

[1496] "Means for notifying the execution results" refers to push notification or email sending functions for notifying the user of the execution results of the task.

[1497] "Script for processing scheduled tasks" refers to program code that is executed to automatically process pre-planned tasks.

[1498] The "means for starting an automatic script for automatically executing a payment task at a specified date and time" is a function for starting a script for automatically executing a payment task at a specified date and time.

[1499] "Means for analyzing information extracted from voice input using a generative AI model" refers to the process of using a generative AI model to perform detailed analysis of data obtained from voice input and extract necessary information.

[1500] This invention is a system that allows a user to set a task using voice input through a terminal and automatically executes the task at a specified date and time. This system is constructed by combining multiple specific hardware and software.

[1501] First, users use a smartphone or smart glasses as hardware for voice input. The device's microphone is used to capture voice input, and speech recognition software such as the Google Cloud Speech-to-Text API is used to convert the speech into text data.

[1502] Next, the server analyzes the received text data using a natural language processing engine (e.g., spaCy, NLTK), extracting important information such as the task, target, amount, and date and time. This analyzed information is then stored in an SQL database.

[1503] Based on the extracted information, the server uses a scheduling system (e.g., cron job, APScheduler) to set up a process to automatically execute the task at a pre-set date and time. Specifically, the server starts a script to automatically execute the payment task, and uses an image recognition engine (e.g., Tesseract OCR) to identify input fields and input information using the automatic script.

[1504] Furthermore, the information extracted from the voice input is analyzed with high accuracy using a generative AI model, which improves task execution accuracy and reduces the burden on the user.

[1505] Finally, if the task is successfully executed, the server summarizes the execution results and notifies the user. This notification is done using push notification or an email sending service (e.g., Amazon SES).

[1506] As a concrete example, if a user gives a voice command such as "Pay the water bill of 3,000 yen on the 25th of this month," this voice is captured by the device and converted into text data by a speech recognition engine. Then, a natural language processing engine extracts information such as "payment task," "water bill," "3,000 yen," and "25th of this month," and registers it in a database. The server then automatically executes the payment procedure at the specified date and time, and notifies the user of the results.

[1507] An example of a prompt sentence is, "Pay the water bill of 3,000 yen on the 25th of this month," and extract the task, target, amount, and date and time from the voice command.

[1508] This system allows users to easily and efficiently set tasks using voice instructions, and the tasks are then automatically executed at the specified date and time, significantly reducing the amount of work required.

[1509] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1510] Step 1:

[1511] The user speaks into the device

[1512] Specific operation: The user gives voice instructions to their smartphone or smart glasses, such as "Pay the water bill of 3,000 yen on the 25th of this month."

[1513] Input: Audio data

[1514] Output: Audio data

[1515] Step 2:

[1516] The device captures the audio data and sends it to the server.

[1517] Specific operation: The device's microphone captures audio and sends the audio data to the Google Cloud Speech-to-Text API.

[1518] Input: Audio data

[1519] Output: Text data

[1520] Step 3:

[1521] The server uses a speech recognition engine to convert the voice data into text data.

[1522] Specific operation: The Google Cloud Speech-to-Text API converts the audio data into text data and sends it back to the server.

[1523] Input: Audio data

[1524] Output: Text data

[1525] Step 4:

[1526] The server analyzes the text data and extracts the task, target, amount, and date and time.

[1527] Specific operation: The server uses a natural language processing engine such as spacy or NLTK to analyze the text data and extract important information (task, target, quantity, date and time).

[1528] Input: Text data

[1529] Output: Task, target, amount, date and time

[1530] Step 5:

[1531] The server registers the analyzed information in a database

[1532] Specific operation: The server stores the extracted information in an SQL database.

[1533] Input: Task, Target, Amount, Date and Time

[1534] Output: Database records

[1535] Step 6:

[1536] The server starts the scheduling system to set up scheduled tasks.

[1537] Specific operation: The server will use APScheduler or a cron job to configure the task to run at the parsed date and time.

[1538] Input: Task, Target, Amount, Date and Time

[1539] Output: Scheduled Job

[1540] Step 7:

[1541] The server will automatically execute the payment task at the specified date and time.

[1542] What happens: At the specified date and time, the server launches an automated script that contacts the payment system to perform the task, using an image recognition engine (such as Tesseract OCR) to identify input fields, enter target and quantity information, and perform verification.

[1543] Input: Task, Target, Amount, Date and Time

[1544] Output: Payment made

[1545] Step 8:

[1546] The server notifies the user of the execution result.

[1547] Specific operation: When the payment is successfully completed, the server notifies the user of the result using a push notification or email sending service (such as Amazon SES).

[1548] Input: Payment execution result

[1549] Output: Notification message

[1550] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1551] The present invention combines a system that analyzes voice input and automatically executes tasks set for specific dates with an emotion engine. In addition to speech recognition, natural language processing, database management, automatic task execution, and notifications, the system also includes a process that recognizes the user's emotions and adjusts the priority and execution method of tasks based on those emotions. Each process is described in detail below.

[1552] Voice input and recognition

[1553] The user gives voice instructions to the terminal, such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[1554] The device uses a microphone to capture audio data and transmits it to the server.

[1555] The server uses a speech recognition engine to convert the received voice data into text data. At this stage, speech recognition services such as Google Cloud Speech-to-Text API are used.

[1556] Natural Language Processing and Data Analysis

[1557] The server processes the converted text data and analyzes it using a natural language processing engine (e.g., spaCy or NLTK). As a result of this analysis, the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) are extracted.

[1558] emotion recognition

[1559] The server uses an emotion engine to analyze the user's emotions from the voice input, using factors such as voice tone, speed, and intonation, for example, to analyze the level of stress or tension in the user's voice.

[1560] Data registration and management

[1561] The server registers the analyzed information in a database, where the task, target, amount, date and time, and emotion information are stored as associated records. The server also uses a scheduling system (e.g., a cron job) to set tasks to be executed at pre-set dates and times.

[1562] Preparing and running a task

[1563] When the due date arrives, the server starts the process to execute the task. For example, in the case of a bank transfer, the following steps are taken:

[1564] 1. The server accesses the bank's online form.

[1565] 2. The server uses an image recognition engine (e.g., Tesseract OCR) to identify the fields on the transfer form.

[1566] 3. The server uses an automatic script to input the transfer destination (Yamada Taro) and the amount (5,000 yen). For users with high stress, the server automatically checks their balance.

[1567] 4. The server will check the entered information, and if it is correct, press the transfer button to execute the transfer.

[1568] 5. The server captures the transfer success message and confirmation screen and uses image recognition to confirm the transfer completion.

[1569] Result notification

[1570] When the task is successfully completed, the server summarizes the results and notifies the user. This notification includes an appropriate message based on the analysis results of the emotion engine. For example, a user experiencing high stress may receive a positive, relaxing message.

[1571] Specific examples

[1572] The user speaks, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[1573] The device captures the audio and sends it to the server.

[1574] The server converts the speech into text using a speech recognition engine, and extracts "Yamada Taro," "5,000 yen," and "25th of this month."

[1575] The server uses an emotion engine to analyze the stress the user is feeling.

[1576] The server registers the information in a database and schedules the action on a scheduled date.

[1577] On the 25th, the server automatically accesses the bank's online form and executes the transfer using image recognition and an automated script.

[1578] Once the transfer is complete, the server compiles the results and notifies the user with a stress-reducing message.

[1579] The user checks the notification and finds out that the transfer was successful.

[1580] This system allows users to easily set tasks via voice input, and the tasks are automatically executed at the specified date and time, significantly reducing the effort required for operation. In addition, by combining it with an emotion engine, it is possible to respond flexibly to the user's emotional state, improving user satisfaction.

[1581] The processing flow will be explained below.

[1582] Step 1:

[1583] The user speaks into the terminal, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[1584] Step 2:

[1585] The device uses the microphone to capture the user's voice input.

[1586] Step 3:

[1587] The device sends the captured audio data to the server.

[1588] Step 4:

[1589] The server receives the audio data.

[1590] Step 5:

[1591] The server calls a voice recognition engine and converts the voice data into text data.

[1592] Step 6:

[1593] The server retrieves the converted text data and recognizes it as "Transfer 5,000 yen to Yamada Taro on the 25th of this month."

[1594] Step 7:

[1595] The server sends the text data to the natural language processing engine and begins analysis.

[1596] Step 8:

[1597] The server uses a natural language processing engine (e.g., spaCy or NLTK) to extract the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) from the text data.

[1598] Step 9:

[1599] The server uses an emotion engine to analyze the user's emotions from the voice input, using voice tone, speed, intonation, etc. For example, analyzing the user's voice for stress or tension levels.

[1600] Step 10:

[1601] The server formats the parsed information and converts date information into a standard format (e.g., 2023-10-25).

[1602] Step 11:

[1603] The server connects to the database and registers the formatted information as a new record, which includes the task, target, amount, date and time, and emotional information.

[1604] Step 12:

[1605] The server periodically monitors the database for the due dates of new records.

[1606] Step 13:

[1607] The server uses a scheduling system (e.g. a cron job) to schedule the task to run on a specified day.

[1608] Step 14:

[1609] When the server reaches the scheduled date (e.g., the 25th), it starts a process to execute the configured task.

[1610] Step 15:

[1611] The server accesses the bank's online transfer form.

[1612] Step 16:

[1613] The server uses an image recognition engine (e.g., Tesseract OCR) to identify each field on the transfer form.

[1614] Step 17:

[1615] The server uses an automated script to enter the transfer recipient (Yamada Taro) and amount (5,000 yen) into the specified fields.

[1616] Step 18:

[1617] The server checks the input information, and if there are no problems, clicks the transfer button to execute the transfer.

[1618] Step 19:

[1619] The server captures the transfer success message and confirmation screen and verifies it using image recognition.

[1620] Step 20:

[1621] The server compiles the results and generates appropriate messages based on the user's emotional information. For example, it selects a message with a relaxing effect for a user experiencing high stress.

[1622] Step 21:

[1623] The server converts the results into a format that can be sent to the user's device (e.g., push notification, email).

[1624] Step 22:

[1625] The terminal receives a notification from the server and notifies the user that the transfer has been completed. The notification includes a message that reflects the user's emotions.

[1626] Step 23:

[1627] The user checks the notification and finds out that the transfer was successful. The notification message is tailored to the user's emotions, increasing user satisfaction.

[1628] Example 2

[1629] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1630] Conventional voice input systems have the drawback of causing a lot of effort and stress when users execute tasks set for a specific date. For important tasks such as bank transfers and reservations, manual operation errors and emotional stress are particularly problematic. Furthermore, the system lacks the flexibility to respond to the user's emotional state, leaving room for improvement in user satisfaction.

[1631] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for acquiring voice input, a means for converting the acquired voice data into text data, a means for analyzing the text data to identify the task, target, quantity, and date and time, a means for registering the analyzed information in a database, a means for executing the task based on a preset date and time, a means for automatically accessing the target system and performing input based on the analyzed information, a means for notifying the user of the execution result, and a means for analyzing emotional information from the voice data and adjusting the priority and execution method of the task. This allows the user to easily set a task via voice input, and the task is automatically executed at the specified date and time, significantly reducing the effort required for operation. Furthermore, by combining an emotion engine, flexible responses based on the user's emotional state are possible, thereby improving user satisfaction.

[1632] The "means for acquiring voice input" is a function for capturing voice uttered by a user as a digital signal through the device's microphone and transmitting it to a subsequent processing step.

[1633] The "means for converting captured voice data into text data" is a function that includes a process for recognizing captured voice data and converting it into a corresponding text format.

[1634] "Means for analyzing tasks, targets, quantities, and dates and times from text data" refers to a function for analyzing text data using natural language processing technology and extracting information on the specified tasks, targets, quantities, and dates and times.

[1635] The "means for registering analyzed information in a database" is a function for storing extracted task-related information in a database and managing it so that it can be used later.

[1636] The "means for executing a task based on a preset date and time" is a function that automatically starts and executes a task at a specified date and time based on task information stored in a database.

[1637] "Means for automatically accessing the target system and entering data based on analyzed information" refers to a function that automates access to an external system required to execute a task and enters appropriate data based on analyzed information.

[1638] The "means for notifying the user of the execution result" is a function for sending a notification to inform the user of the execution status and result of the task.

[1639] "Means for analyzing emotional information from voice data and adjusting task priority and execution method" is a function that analyzes the user's emotional state at the time of voice input and adjusts task priority and execution method according to the user's emotions.

[1640] The present invention combines an emotion engine with a system that analyzes voice input and automatically executes tasks set for specific dates. This system is configured using the following hardware and software.

[1641] Acquiring and converting voice input

[1642] The user speaks to the terminal, saying, "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The terminal uses a microphone to capture the voice data and sends it to the server.

[1643] The server converts the received voice data into text data using the Google Cloud Speech-to-Text API, which provides highly accurate speech recognition and enables real-time conversion.

[1644] Natural Language Processing and Data Analysis

[1645] The server analyzes the converted text data using a natural language processing engine (e.g., spaCy or NLTK). Through this analysis, the task (transfer), target (Yamada Taro), amount (5,000 yen), and date and time (the 25th of the current month) are extracted.

[1646] Emotion analysis

[1647] The server uses an emotion engine to analyze the user's emotions from the voice input. This emotion analysis uses the tone, speed, and intonation of the voice. For example, Google Cloud Natural Language API and IBM Watson Tone Analyzer are used to analyze the level of stress and tension from the voice.

[1648] Data registration and management

[1649] The server registers the analyzed information in a database, which includes the task, target, amount, date and time, and emotion information, using a database management system such as MySQL or PostgreSQL. The server also uses a scheduling system (e.g., cron job) to set tasks to be executed at predetermined dates and times.

[1650] Executing a task

[1651] When the specified date and time arrives, the server will start executing the task. For example, in the case of a bank transfer, the following steps are taken:

[1652] 1. The server uses the Selenium library in Python to access the bank's online form.

[1653] 2. The server uses an image recognition engine (e.g., Tesseract OCR) to identify the fields in the transfer form, and then uses an automatic script to enter the transfer recipient (Yamada Taro) and the amount (5,000 yen). A setting to automatically check the balance is added, especially for users with high stress levels.

[1654] 3. The server will check the entered information, and if it is correct, press the transfer button to execute the transfer.

[1655] 4. The server captures the transfer success message and confirmation screen and uses image recognition to confirm the transfer completion.

[1656] Notification of results

[1657] After the server completes the task, it notifies the user of the result. The notification includes an appropriate message based on the results of the sentiment analysis. For example, a user experiencing high stress might receive a positive, relaxing message such as, "The transfer has been successfully completed. Relax and have a wonderful day."

[1658] Specific examples

[1659] The user gives a voice command such as "Transfer 5,000 yen to Yamada Taro on the 25th of this month." The device captures the voice and sends it to the server. The server converts the voice data into text using the Google Cloud Speech-to-Text API. The server analyzes the text using a natural language processing engine and extracts "Yamada Taro," "5,000 yen," and "the 25th of this month." The server uses an emotion engine to analyze the user's stress level. The server registers the information in a database and schedules the necessary actions for the specified date and time. On the 25th, the server automatically accesses the bank's online form and performs the transfer using Selenium and Tesseract OCR. When the transfer is complete, the server summarizes the results and notifies the user. The user checks the notification and learns that the transfer was successful.

[1660] By implementing this system, users can easily set tasks via voice commands, and the tasks will be automatically executed at the specified date and time, significantly reducing the amount of work required. In addition, by combining it with an emotion engine, flexible responses according to the user's emotional state become possible, improving user satisfaction.

[1661] An example of a generative AI model and prompts to use is ChatGPT-4, and examples of prompts include:

[1662] "Please explain in detail the process of your new system that combines an emotion engine with a system that automatically executes tasks based on voice input."

[1663] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1664] Processing flow

[1665] Step 1: Getting voice input

[1666] The user gives a voice command, and the device uses the microphone to capture the voice data and transmits it to the server.

[1667] Input: User's voice command (e.g., "Transfer 5,000 yen to Yamada Taro on the 25th of this month.")

[1668] Output: Captured audio data (digital format)

[1669] Specific operation: The user speaks into the microphone. The device's microphone captures the voice and generates real-time audio data, which is then sent to the server via the network.

[1670] Step 2: Convert audio data to text

[1671] The server converts the received audio data into text data using the Google Cloud Speech-to-Text API.

[1672] Input: Captured audio data

[1673] Output: Converted text data (e.g. "Transfer 5,000 yen to Yamada Taro on the 25th of this month")

[1674] What happens: The server processes the audio data via the Google Cloud Speech-to-Text API and generates corresponding text data, which is passed on to the next processing step.

[1675] Step 3: Analyzing the text data

[1676] The server uses a natural language processing engine (e.g., spaCy or NLTK) to parse the text data and extract important information such as task, subject, amount, date and time.

[1677] Input: Converted text data

[1678] Output: Extracted information (Task = Transfer, Target = Yamada Taro, Amount = 5000 yen, Date and Time = 25th of this month)

[1679] How it works: The server passes the text data to a natural language processing engine, which performs grammatical and semantic analysis, extracting specific keywords (task, subject, quantity, date and time).

[1680] Step 4: Sentiment Analysis

[1681] The server uses an emotion engine to analyze the user's emotion from the voice input, including the tone, rate, and intonation of the voice.

[1682] Input: converted audio data, analyzed text data

[1683] Output: Emotional information (e.g., stress level = high)

[1684] Specific operation: The server passes voice and text data to the emotion engine, which analyzes emotions from these data. It analyzes the voice characteristics and determines the user's emotional state.

[1685] Step 5: Registering and scheduling data

[1686] The server registers the parsed information in a database and schedules the task to be executed using a scheduling system (e.g., a cron job).

[1687] Input: Extracted information, emotional information

[1688] Output: Task information registered in the database, set schedule

[1689] Specific operation: The server stores the analyzed task information and emotion information in a database and schedules the task to be executed at the specified date and time using a database management system and a scheduling tool.

[1690] Step 6: Task execution

[1691] When the set date and time arrives, the server starts the task execution process. For example, in the case of a transfer task, the server accesses the bank's online system and inputs the necessary information.

[1692] Input: Task information and emotion information registered in the database

[1693] Output: Task executed, confirmation message

[1694] What it does: The server uses Selenium to access the bank's online transfer form, uses Tesseract OCR to identify the required fields, and then uses an automated script to enter the transfer details and amount, verify the details, and execute the transfer.

[1695] Step 7: Notification of results

[1696] Once the task is completed, the server notifies the user of the results, including an appropriate message based on the emotion information.

[1697] Input: Results of the executed task, emotional information

[1698] Output: Notification message sent to the user

[1699] Specific operation: The server confirms the success of the task execution and generates a notification message based on that. It also takes into account emotional information and sends a message that corresponds to the user's state. For example, it could say, "The transfer has been successfully completed. Please relax and have a wonderful day."

[1700] (Application example 2)

[1701] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1702] Conventional automated task execution systems using voice input simply execute tasks without considering the user's emotional state. As a result, depending on the user's situation or stress level, appropriate responses are not made, and satisfaction cannot be sufficiently improved. Furthermore, flexible adjustments based on emotions are required for the execution of payment tasks, but no technology that can achieve this has existed.

[1703] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring voice input from the terminal, means for converting the acquired voice data into text data, means for analyzing the task, target, quantity, and date and time from the text data, means for registering the analyzed information in a database, means for executing the task based on a preset date and time, means for automatically accessing the target system and making input based on the analyzed information, means for analyzing the user's emotions using an emotion engine, means for adjusting the priority and execution method of the task based on the user's emotional state, and means for notifying the user of the execution result. This allows the execution of payment tasks via voice input to be flexibly adjusted according to the user's emotional state, thereby improving user satisfaction.

[1704] A "terminal" is an electronic device for obtaining voice input.

[1705] "Voice input" refers to voice data uttered by the user, and includes instructions and commands to the system.

[1706] "Audio Data" means a digital recording of audio input captured by a Device.

[1707] "Text data" is character string data obtained by analyzing voice data using a voice recognition engine.

[1708] A "task" refers to a specific action or operation that is executed based on a user's voice instruction.

[1709] A "subject" is an individual, system, or physical or digital entity involved in the performance of a task.

[1710] "Quantity" refers to the numerical values ​​or metrics required to perform a task.

[1711] "Date and time" refers to a specific date and time when a task is scheduled to be performed.

[1712] A "database" is a structured data storage system for storing and managing analyzed information.

[1713] "Automatic access" means that the system connects to online services or forms based on a pre-defined process without manual intervention.

[1714] "Entering" means automatically writing the analyzed information into the target system.

[1715] An "emotion engine" is a software or hardware module that analyzes a user's emotions from their voice input and identifies their emotional state.

[1716] "Emotional state" refers to the psychological state or stress level experienced by a user.

[1717] "Priority" refers to the execution order and importance of the analyzed task.

[1718] An "execution method" is a means for determining how a task should proceed depending on the user's emotional state.

[1719] "Notification" is the act of informing a user of the results or status of an executed task.

[1720] The embodiments of the present invention will now be described in detail.

[1721] System configuration

[1722] The system consists of the following major hardware and software components:

[1723] Device: Smartphone (iOS / Android)

[1724] Server: Speech recognition engine (Google Cloud Speech-to-Text API), natural language processing engine (spaCy), emotion analysis engine (e.g., voice tone analysis software), database (SQLite), task scheduling system

[1725] Network: The internet connection that connects your device to the server.

[1726] Main features of the program

[1727] The program has the function of automatically executing tasks through voice input, and adjusts task priority and execution method according to the user's emotional state. To achieve this, a complex system is constructed that goes through steps such as voice capture, voice recognition, natural language processing, emotion analysis, database registration, task scheduling, execution, and result notification.

[1728] Processing flow and technologies used

[1729] 1. Audio Capture

[1730] The user inputs voice using the microphone of the device.

[1731] The device captures the audio data and sends it to the server.

[1732] 2. Voice Recognition

[1733] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data.

[1734] For example, if the user instructs "Pay the electricity bill of 5,000 yen next Monday," the voice data is converted into text "Pay the electricity bill of 5,000 yen next Monday."

[1735] 3. Natural Language Processing

[1736] The converted text data is analyzed using spaCy to extract task details (date, amount, target).

[1737] In this example, "electricity bill," "5,000 yen," and "next Monday" are extracted.

[1738] 4. Emotion analysis

[1739] The server analyzes the emotional state from the voice input, analyzing the tone, speed, intonation, etc. with an emotion analysis engine.

[1740] For example, if the user's voice indicates that he or she is stressed, the user is recorded as "stressed."

[1741] 5. Database registration

[1742] The analyzed information is registered in a database, and the task, subject, amount, date and time, and emotional information are stored as associated records.

[1743] 6. Task Scheduling

[1744] Based on the registered information, the task is scheduled to be executed at a preset date and time.

[1745] For example, a task to automatically pay the electricity bill is scheduled for "next Monday."

[1746] 7. Task execution

[1747] At the specified date and time, the server will automatically make the online payment.

[1748] Based on emotional state, stressed users may be asked to check their balances in advance.

[1749] 8. Result notification

[1750] When the task is successfully completed, the user is notified of the result and a message is sent according to the user's emotional state, such as a relaxing message like "Thank you for your hard work, your electricity bill has been paid."

[1751] Examples of concrete examples and prompts

[1752] For example, if the user speaks something like this:

[1753] Today, I want to make sure I don't forget to transfer my rent, so please set it up so that the rent of 70,000 yen is automatically transferred to a specified account on the first day of each month. I'm a little nervous about asking for rent payments.

[1754] This system allows users to easily set and execute tasks, and enables flexible responses according to the user's emotional state.

[1755] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1756] Step 1:

[1757] The user uses the microphone on the device to input voice. For example, the user might say, "Please pay the electricity bill of 5,000 yen next Monday." The input to the device is voice data, which the device captures. The output here is the captured voice data.

[1758] Step 2:

[1759] The device sends the captured audio data to the server. The input is the audio data captured on the device, and the output to the server is the transmission of audio data. Once the server receives the audio data, it is ready to proceed to the next step.

[1760] Step 3:

[1761] The server uses the Google Cloud Speech-to-Text API to convert the voice data into text data. The input is the voice data, and the output is the converted text data. The server calls the speech recognition engine, analyzes the voice data, and converts it into text.

[1762] Step 4:

[1763] The server uses a natural language processing engine (spaCy) to analyze task details (date, amount, target) from text data. The input is text data, and the output is detailed task information (e.g., "electricity bill," "5,000 yen," "next Monday"). The server analyzes the text data and extracts important information.

[1764] Step 5:

[1765] The server uses an emotion analysis engine to analyze the user's emotional state from the voice data. The input is the initial voice data or text data, and the output is the emotional state (e.g., "stressed"). The server analyzes the voice tone, speed, and intonation to identify the emotional state.

[1766] Step 6:

[1767] The server registers the analyzed information in a database. The input is the analyzed task details and emotional state, and the output is a record saved in the database. The server stores the extracted information in the database as structured data.

[1768] Step 7:

[1769] The server schedules tasks to be executed based on a pre-set date and time. The input is the task details, execution date and time, and the output is the scheduled task. The server uses the task scheduling system to set up the task to be executed automatically on the specified date and time.

[1770] Step 8:

[1771] At the scheduled time, the server automatically executes the online payment. The inputs are task details, emotional state, and schedule information, and the output is confirmation of payment completion. Specifically, the server automatically accesses the target payment system, enters the necessary information (payee, amount) into the input form, and completes the payment.

[1772] Step 9:

[1773] When the task is successfully completed, the server notifies the user of the result. The input is the task execution status and emotional state, and the output is a notification message sent to the user. The server generates an appropriate message depending on the emotional state, notifying the user, for example, "Thank you for your hard work. Your electricity bill has been paid."

[1774] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1775] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1776] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1777] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1778] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1779] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1780] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1781] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1782] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1783] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1784] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1785] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1786] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1787] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1788] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1789] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1790] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1791] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1792] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1793] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1794] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1795] The following is further disclosed regarding the above embodiment.

[1796] (Claim 1)

[1797] a means for obtaining voice input from the device;

[1798] A means for converting the acquired voice data into text data;

[1799] A means of analyzing tasks, targets, quantities, and dates from text data;

[1800] A means for registering the analyzed information in a database;

[1801] means for executing a task based on a preset date and time;

[1802] A means for automatically accessing the target system and inputting information based on the analyzed information;

[1803] a means for notifying a user of the execution result;

[1804] A system including:

[1805] (Claim 2)

[1806] The system of claim 1, wherein the system executes a transfer task based on the analyzed information.

[1807] (Claim 3)

[1808] 10. The system of claim 1, wherein image recognition is used to identify input fields and automatically input information.

[1809] "Example 1"

[1810] (Claim 1)

[1811] a means for obtaining voice input from the device;

[1812] A means for converting the acquired voice data into text data;

[1813] A means of analyzing tasks, targets, quantities, and dates from text data;

[1814] A means for registering the analyzed information in a database;

[1815] means for executing a task based on a preset date and time;

[1816] means for automatically accessing a target information processing system and inputting information based on the analyzed information;

[1817] a means for notifying a user of the execution result;

[1818] A system including:

[1819] (Claim 2)

[1820] 10. The system of claim 1, further comprising: performing a payment processing task based on the analyzed information.

[1821] (Claim 3)

[1822] 10. The system of claim 1, wherein image recognition is used to identify input fields and automatically input information.

[1823] "Application Example 1"

[1824] (Claim 1)

[1825] a means for obtaining voice input from the device;

[1826] A means for converting the acquired voice data into text data;

[1827] A means of analyzing tasks, targets, quantities, and dates from text data;

[1828] A means for registering the analyzed information in a database;

[1829] means for executing a task based on a preset date and time;

[1830] A means for automatically accessing the target system and inputting information based on the analyzed information;

[1831] a means for notifying the execution result;

[1832] Scripts for handling scheduled tasks, and

[1833] A means to initiate an automated script to automatically execute payment tasks at a specified date and time;

[1834] a means for analyzing information extracted from the speech input using a generative AI model;

[1835] A system including:

[1836] (Claim 2)

[1837] The system of claim 1, further comprising: performing a payment task based on the analyzed information.

[1838] (Claim 3)

[1839] 10. The system of claim 1, wherein image recognition is used to identify input fields and automatically input information.

[1840] "Example 2: Combining Emotion Engines"

[1841] (Claim 1)

[1842] a means for obtaining a voice input;

[1843] A means for converting the acquired voice data into text data;

[1844] A means of analyzing tasks, targets, quantities, and dates from text data;

[1845] A means for registering the analyzed information in a database;

[1846] means for executing a task based on a preset date and time;

[1847] A means for automatically accessing the target system and inputting information based on the analyzed information;

[1848] a means for notifying a user of the execution result;

[1849] A method for analyzing emotional information from voice data and adjusting task priorities and execution methods;

[1850] A system including:

[1851] (Claim 2)

[1852] The system of claim 1, wherein the system executes a transfer task based on the analyzed information.

[1853] (Claim 3)

[1854] 10. The system of claim 1, wherein image recognition is used to identify input fields and automatically input information.

[1855] "Application example 2 when combining emotion engines"

[1856] (Claim 1)

[1857] a means for obtaining voice input from the device;

[1858] A means for converting the acquired voice data into text data;

[1859] A means of analyzing tasks, targets, quantities, and dates from text data;

[1860] A means for registering the analyzed information in a database;

[1861] means for executing a task based on a preset date and time;

[1862] A means for automatically accessing the target system and inputting information based on the analyzed information;

[1863] means for analyzing a user's emotions using an emotion engine;

[1864] a means for adjusting the priority and execution of tasks based on the user's emotional state;

[1865] a means for notifying a user of the execution result;

[1866] A system including:

[1867] (Claim 2)

[1868] The system of claim 1, further comprising: a payment processing unit configured to perform a payment processing task based on the analyzed information.

[1869] (Claim 3)

[1870] 10. The system of claim 1, wherein image recognition is used to identify input fields and automatically input information. [Explanation of symbols]

[1871] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for obtaining voice input from the device; A means for converting the acquired voice data into text data; A means of analyzing tasks, targets, quantities, and dates from text data; A means for registering the analyzed information in a database; means for executing a task based on a preset date and time; A means for automatically accessing the target system and inputting information based on the analyzed information; a means for notifying a user of the execution result; A system including:

2. 2. The system according to claim 1, wherein the system executes a transfer task based on the analyzed information.

3. 10. The system of claim 1, wherein image recognition is used to identify input fields and automatically input information.

Images