system

Abstract

Provide a system. [Solution] terminal means for capturing voice input; a server means for analyzing the voice data received from the terminal means and converting it into text data; a server means for analyzing the text data and identifying a corresponding task; a server means for accessing the schedule management system based on the identified task and executing necessary processing; a server means for notifying the terminal means of a processing result; a terminal means for displaying the notification; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's business environment, daily business processes are becoming more complex and diverse, necessitating efficient task management and automation. However, many small and medium-sized enterprises and team leaders find it difficult to achieve this with limited resources. Efficient task management systems that utilize voice input, in particular, have been hindered by numerous technological barriers. As a result, time-consuming manual input and the use of multiple tools significantly reduce business efficiency. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following means. A terminal device that captures voice input allows the user to easily set tasks by inputting voice data. Next, a server device is provided that analyzes the voice data received from the terminal device and converts it into text data. This server device utilizes a voice recognition engine to achieve highly accurate text conversion. Furthermore, a server device is provided that analyzes the converted text data and identifies the corresponding task. This device analyzes the text data using a natural language processing engine and can accurately understand the user's intention. Then, a server device is provided that accesses a schedule management system based on the identified task and executes the necessary processing, thereby realizing automatic processing such as reservation and cancellation. Finally, the processing result is notified to the terminal device, and the terminal device displays the notification to the user, allowing the user to immediately confirm that the processing was executed correctly. These means enable business efficiency improvement and automated task management, significantly reducing the burden on small and medium-sized enterprises and team leaders.

[0006] "Voice input" is a means by which a user inputs instructions or information through voice.

[0007] The "terminal means" is a device operated by a user, which captures voice input and displays notifications.

[0008] "Audio data" refers to data that digitally represents sound signals provided by a user through voice input.

[0009] The "server means" is a computer system for carrying out multiple processes, and is a device in charge of analyzing voice data and executing tasks.

[0010] A "voice recognition engine" is software or a service for converting voice data into text data.

[0011] "Text data" is character information converted by a voice recognition engine.

[0012] A "natural language processing engine" is software or technology that analyzes text data and understands user intent and commands.

[0013] A "task" is an operation or processing content requested by a user through voice input.

[0014] A "schedule management system" is software or a service for managing schedules and tasks, and is a system that provides calendar and schedule management functions.

[0015] "Notification" is a message that notifies the user of processing results or important information. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The system of the present invention is designed to realize task management and business automation using voice input. The following describes in detail the embodiments of the present invention.

[0038] First, a user uses a device with a dedicated application installed to input voice. This device is equipped with a microphone to capture voice input. The user presses the voice input button and inputs voice, such as "Schedule a team meeting tomorrow at 10:00 AM."

[0039] The audio data is then captured by the terminal and transmitted over the Internet to a server.

[0040] The server uses a speech recognition engine to analyze the received voice data and convert it into text data, in this case "Schedule a team meeting tomorrow at 10 AM."

[0041] The server then uses a natural language processing engine to analyze the text data and identify the user's intent, extracting the time information "tomorrow at 10:00 AM" and the task information "book a team meeting."

[0042] Based on the identified task (i.e., booking a team meeting), the server accesses a schedule management system (e.g., a calendar service) and performs an operation to add an event for the specified time slot. The server calls the appropriate API to complete the booking.

[0043] If the reservation is successful, the server generates a message to notify the terminal of the result, for example, a message saying "A team meeting has been reserved for tomorrow at 10:00 AM."

[0044] The terminal displays the received notification message to the user, and the user confirms that the reservation has been made correctly.

[0045] As a concrete example, if a user says "Schedule a team meeting tomorrow at 10 AM," the following occurs:

[0046] 1. The user speaks into the device.

[0047] 2. The device sends the audio data to the server.

[0048] 3. The server uses a speech recognition engine to convert the speech into text data.

[0049] 4. The server uses a natural language processing engine to analyze the text data and identify tasks.

[0050] 5. The server accesses the schedule management system and adds the event to the specified time slot.

[0051] 6. The server notifies the terminal of the processing results.

[0052] 7. The device displays the notification to the user.

[0053] 8. The user confirms the notification.

[0054] Through this series of processes, the system of the present invention realizes efficient task management and business automation using voice input.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] The user speaks into a device with a dedicated application installed, pressing the voice input button and saying something like, "Schedule a team meeting for tomorrow at 10:00 AM."

[0058] Step 2:

[0059] The device captures voice input from the user and converts it into digital audio data, which is temporarily stored on the device.

[0060] Step 3:

[0061] The device transmits the captured audio data to a server over the internet using a secure communication protocol such as HTTPS or WebSocket.

[0062] Step 4:

[0063] The server passes the received voice data to a speech recognition engine, which converts the digital voice into text data, which then translates to "Schedule a team meeting tomorrow at 10 AM."

[0064] Step 5:

[0065] The server receives the text data and passes it to a natural language processing engine for analysis, which identifies the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" from the text.

[0066] Step 6:

[0067] The server uses the identified task information to create an API request to the schedule management system, including the date and time to be booked and details of the event.

[0068] Step 7:

[0069] The server sends an API request to the schedule management system, for example, an HTTP POST request to the API endpoint of a calendar service.

[0070] Step 8:

[0071] The schedule management system processes the received API request, creates a new event at the specified date and time, and returns a success response to the server if the process is completed successfully.

[0072] Step 9:

[0073] The server receives the response from the schedule management system, confirms that the reservation was successful, and then generates a notification message as a result of the processing: "A team meeting has been booked for tomorrow at 10:00 AM."

[0074] Step 10:

[0075] The server then sends the generated notification message to the device, again using a secure communication protocol such as HTTPS or WebSocket.

[0076] Step 11:

[0077] The device displays the notification message received from the server to the user, for example, as a push notification or a pop-up message within the app.

[0078] Step 12:

[0079] The user will see a notification message on the terminal and will know that the reservation has been made successfully. Based on the notification, the user can consider the next action to take.

[0080] The above steps enable efficient task management and business automation through voice input.

[0081] Example 1

[0082] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0083] While there are a wide variety of solutions for task management and work efficiency today, efficient and intuitive methods using voice input have not yet been fully established. In particular, there is room for improvement in systems that automatically schedule tasks using voice and easily realize complex natural language processing. There is a need to significantly reduce user effort by providing a consistent automated process from voice input, task identification, registration in a schedule management system, and notification of the results.

[0084] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0085] In this invention, the server includes means for analyzing received voice data using a voice recognition engine and converting it into text data, means for analyzing the text data using a natural language processing engine and identifying the corresponding task, and means for accessing the schedule management system based on the identified task and executing the necessary processing. This makes it possible to consistently automate everything from voice input to task identification, automatic schedule addition, and notification of processing results.

[0086] "Voice input" is the act of conveying information or instructions to a system using voice.

[0087] The "terminal means" is a device that allows a user to input voice data and capture and transmit the data.

[0088] The "server means" is a device or program for analyzing received data and executing various processes.

[0089] "Audio Data" means a digital representation of the sound captured by an audio input.

[0090] "Text data" is character information that has been converted by analyzing voice data.

[0091] A "voice recognition engine" is software or hardware that analyzes voice data and converts it into text data.

[0092] A "natural language processing engine" is software or hardware that analyzes text data and understands and interprets human language.

[0093] A "task" is a specific activity or work undertaken to achieve a particular goal.

[0094] A "schedule management system" is software or hardware for organizing and managing tasks and events in chronological order.

[0095] "Notification" refers to a message or information presentation that conveys processing results or information to the user.

[0096] "Analysis" is the process of extracting meaning and information from data and understanding it.

[0097] The system of the present invention is designed to realize task management and business automation using voice input. This system includes a terminal means for capturing voice input, a server means for analyzing the voice data and converting it into text data, a server means for analyzing the text data and identifying tasks, a server means for accessing a schedule management system based on the identified tasks and executing processing, and a server means for notifying the terminal means of the processing results.

[0098] First, a user uses a device with a dedicated application installed to input voice data. This device is equipped with a microphone to capture voice data. When the user inputs voice data such as "Schedule a team meeting tomorrow at 10:00 AM," the voice data is captured by the device and sent to the server via the Internet.

[0099] The server uses a speech recognition engine (e.g., speech recognition software) to analyze the received voice data and convert it into text data, in this case, "Schedule a team meeting tomorrow at 10:00 AM."

[0100] The server then uses a natural language processing engine (e.g., a natural language processing API) to analyze the text data and identify the user's intent. In this specific process, the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" are extracted.

[0101] Based on the identified task, i.e., booking a team meeting, the server accesses the schedule management system (e.g., a calendar service) and performs an operation to add an event for the specified time slot. The server performs this operation by calling the appropriate API.

[0102] When the reservation is successfully completed, the server generates a message to notify the terminal of the result. For example, a message saying "A team meeting has been reserved for tomorrow at 10:00 AM" is sent to the terminal, which displays it to the user. The user confirms the notification and knows that the reservation was made successfully.

[0103] As a specific example, the processing flow when a user voice-inputs "Book a team meeting for tomorrow at 10:00 AM" will be described.

[0104] In addition, by inputting the following prompt sentence into the generative AI model, an explanation based on the system's processing flow can be obtained.

[0105] Take a step-by-step guide to how the system works, given the voice input "Schedule a team meeting tomorrow at 10am."

[0106] The prompt sentences provide detailed explanations of the specific operations of each processing step.

[0107] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0108] Step 1:

[0109] The user provides voice input. Using a device with a dedicated application installed, the user says, "Schedule a team meeting for tomorrow at 10:00 AM."

[0110] Input: User's voice commands

[0111] Output: Audio data (digital format)

[0112] Step 2:

[0113] The device sends the audio data to the server using a secure communication protocol (e.g., HTTPS).

[0114] Input: Audio data

[0115] Output: Request sent to server

[0116] Step 3:

[0117] The server converts the voice data into text data. The server uses a voice recognition engine to analyze the voice data and convert it into text data.

[0118] Input: Audio data

[0119] Output: Text data: "Schedule a team meeting tomorrow at 10 AM"

[0120] Step 4:

[0121] The server analyzes the text data and identifies the task. The server uses a natural language processing engine to extract the intent from the text data. In this case, the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" are extracted.

[0122] Input: Text data: "Schedule a team meeting tomorrow at 10 AM"

[0123] Output: Time information "Tomorrow at 10 AM" and task information "Schedule a team meeting"

[0124] Step 5:

[0125] The server sends a request to the schedule management system. Based on the identified task information, the server sends an API request to add an event to the schedule management system.

[0126] Input: Time information "Tomorrow at 10 AM" and task information "Schedule a team meeting"

[0127] Output: A request to add an event to the schedule management system.

[0128] Step 6:

[0129] The server generates a notification message of the processing result, creates a notification of successful reservation, and sends it to the terminal.

[0130] Input: Response from the schedule management system (if successful)

[0131] Output: Notification message: "Your team meeting has been scheduled for tomorrow at 10 AM."

[0132] Step 7:

[0133] The device receives the notification message and displays it to the user, either in the notification center within the application or as a pop-up message.

[0134] Input: Notification message: "Team meeting booked for tomorrow at 10am"

[0135] Output: Display notification

[0136] Step 8:

[0137] The user checks the notification. The user checks the notification on the terminal and realizes that the reservation has been made successfully.

[0138] Input: The displayed notification message

[0139] Output: Recognition (no specific output)

[0140] (Application example 1)

[0141] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0142] In modern factories, many tasks are performed manually, resulting in issues with efficiency and accuracy. Misunderstandings and delays are particularly common when communicating work instructions and executing tasks. Frequent on-site operations by workers also increase the workload. There is a need for a system that can improve this situation and increase efficiency by automating factory operations.

[0143] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0144] In this invention, the server includes terminal means for capturing voice input, means for analyzing voice data received from the terminal means and converting it into text data, means for analyzing the text data and identifying the corresponding task, means for accessing the work machine control system based on the identified task and executing the required processing, means for notifying the terminal means of the processing results, and terminal means for displaying the notification. This makes it possible to efficiently and accurately transmit work instructions and automate work using voice instructions.

[0145] The "terminal means for capturing voice input" refers to a device equipped to receive a user's voice, such as a microphone or a smartphone.

[0146] "Means for analysis" refers to the technology or process for converting voice data into text data and analyzing the user's intent from that text data.

[0147] "Text data" refers to character string data converted from speech input, which can then be analyzed.

[0148] "Means for identification" refers to the technology or algorithm that finds and recognizes the corresponding task from the analyzed text data.

[0149] A "work machine control system" is a system that controls various machines and robots used in factories and production lines to carry out tasks.

[0150] "Means for performing processing" refers to hardware or software that performs actual actions or operations based on a specified task.

[0151] "Means for notifying" refers to the method or device for notifying the user of the processing results or status.

[0152] "Terminal means" refers to a device operated by a user, and is used for voice input, display of notifications, etc.

[0153] A "schedule management system" is a system that users use to manage tasks and events, including calendar apps and scheduling software.

[0154] A "voice recognition engine" is software or algorithms that analyze voice data and convert it into text data.

[0155] A "natural language processing engine" is a technology or algorithm that analyzes text data and understands user intent and commands.

[0156] The system of the present invention is designed to automate work instructions and operations in a factory using voice input. In this system, a user gives instructions by voice, and the robot automatically executes the work by analyzing the contents of the instructions. Specific embodiments are described in detail below.

[0157] First, the user uses a device with a dedicated application installed to input voice. This device is equipped with a microphone to capture voice input. The user presses the voice input button and inputs voice, such as "Move part B to the next line."

[0158] Next, the voice data is captured by the device and sent to a server via the Internet. The server uses a voice recognition engine to analyze the received voice data and convert it into text data. In this case, it is converted into text data such as "Move part B to the next line."

[0159] Next, the server uses a natural language processing engine to analyze the text data and identify the user's intent. In this case, the server extracts the movement instruction "to the next line" and the specific target "Part B."

[0160] Based on the identified task (i.e., moving a part using a factory robot), the server accesses the work machine control system and performs operations to execute the specified task. The server calls the APIs and protocols for robot control to execute the movement instructions.

[0161] When the operation is completed successfully, the server generates a message to notify the terminal of the result. For example, a message saying "Part B has moved to the next line" is generated. The terminal displays the received notification message to the user, who can confirm that the operation was performed correctly.

[0162] As a concrete example, if the user speaks "Move part B to the next line," the following prompt sentence is generated:

[0163] "Move part B to the next line."

[0164] This prompt allows the system to understand the user's instructions and the factory robot to automatically carry out the specified work. Through this series of processes, the system of the present invention realizes efficient and accurate transmission of work instructions and automation of work using voice input.

[0165] The hardware used includes microphones and smartphones to capture voice input and factory robots to perform the tasks, while the software used includes a speech recognition engine, a natural language processing engine, and an API for controlling the robots.

[0166] As described above, the system of the present invention efficiently transmits instructions for factory work through voice input, thereby realizing automation of work.

[0167] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0168] Step 1:

[0169] The user provides voice input.

[0170] Input: User speech (e.g., "Move part B to the next line").

[0171] Output: Audio data.

[0172] Specific operation: The user presses the voice input button and speaks, and the device's microphone captures the voice.

[0173] Step 2:

[0174] The terminal transmits the voice data to the server.

[0175] Input: Audio data.

[0176] Output: The audio data sent to the server.

[0177] Specific operation: The device sends the captured audio data to a server via the Internet.

[0178] Step 3:

[0179] The server converts the voice data into text data using a voice recognition engine.

[0180] Input: Audio data.

[0181] Output: Text data (e.g. "Move part B to the next line").

[0182] Specific operation: The server passes the received voice data to a voice recognition engine and converts the data into text format.

[0183] Step 4:

[0184] The server uses a natural language processing engine to analyze the text data and identify the user's intent.

[0185] Input: Text data.

[0186] Output: Data specifying the intent (e.g., "Task to move part B to the next line").

[0187] Specific operation: The server passes the text data to a natural language processing engine for analysis to identify the user's intent and task.

[0188] Step 5:

[0189] The server accesses the work machine control system and performs the necessary processing.

[0190] Input: Identified task data.

[0191] Output: Robot control command (e.g. "Move part B from line A to line B").

[0192] Specific operation: Based on the identified task, the server calls the API of the work machine control system to give instructions to the robot.

[0193] Step 6:

[0194] The processing result is notified to the terminal.

[0195] Input: Processing result (e.g., work completion notification).

[0196] Output: Notification message to the terminal (e.g. "Part B has moved to the next line").

[0197] Specific operation: The server receives the processing result from the work machine control system, generates a notification message, and sends it to the terminal.

[0198] Step 7:

[0199] The terminal displays the notification to the user.

[0200] Input: Notification message.

[0201] Output: A user-visible notification (e.g., "Part B has moved to the next line").

[0202] Specific operation: The device receives a notification message from the server and notifies the user by displaying the message on the screen or by voice notification.

[0203] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0204] The system of the present invention is equipped with a function for recognizing a user's emotions in addition to task management and business automation using voice input. MODES FOR CARRYING OUT THE INVENTION The following describes in detail embodiments of the present invention.

[0205] First, the user speaks into a device with a dedicated application installed. The device is equipped with a microphone for capturing voice input, and while pressing the voice input button, the user speaks, "Schedule a team meeting tomorrow at 10:00 AM."

[0206] The audio data is then captured by the terminal and transmitted over the Internet to a server.

[0207] The server uses a speech recognition engine to analyze the received voice data and convert it into text data, which in this case is "Schedule a team meeting tomorrow at 10 AM."

[0208] The server then passes the text and voice data to an emotion engine to identify the user's emotion. The emotion engine determines whether the user is stressed or relaxed based on the tone of the voice and the content of the text. For example, if the user's voice sounds urgent and tense, the emotion engine will recognize the user as being in a "tense" state.

[0209] The server then uses a natural language processing engine to analyze the text data and identify the user's intent, extracting the time information "tomorrow at 10:00 AM" and the task information "book a team meeting."

[0210] Based on the identified task (i.e., booking a team meeting), the server makes an API request to the scheduling system, including the date and time to book and details of the event.

[0211] The server sends an API request to the schedule management system and performs an operation to add an event to the specified time slot. For example, it calls the API of the schedule management system to execute a reservation.

[0212] Once the reservation is successfully completed, the server generates a message to notify the device of the result. This message is customized based on the user's emotions. For example, if the user is nervous, the server generates a message saying, "Your team meeting has been booked for tomorrow at 10:00 AM. Relax and move on to the next task!"

[0213] The terminal displays the received notification message to the user, and the user confirms that the reservation has been made correctly. The content of the notification message is designed to help the user relax.

[0214] For example, if a user says "Schedule a team meeting tomorrow at 10 AM" and the emotion engine recognizes the user's nervousness, the following happens:

[0215] 1. The user speaks into the device.

[0216] 2. The device sends the audio data to the server.

[0217] 3. The server uses a speech recognition engine to convert the speech into text data.

[0218] 4. The server uses the emotion engine to recognize the user's emotions.

[0219] 5. The server uses a natural language processing engine to analyze the text data and identify tasks.

[0220] 6. The server accesses the schedule management system and adds the event to the specified time slot.

[0221] 7. The server generates a notification message of the processing result that takes into consideration the user's feelings.

[0222] 8. The server notifies the terminal of the processing results.

[0223] 9. The device displays the notification to the user.

[0224] 10. The user acknowledges the notification, relaxes, and moves on to the next task.

[0225] As a result, the system of the present invention can provide a system that not only enables efficient task management and business automation through voice input, but also takes into consideration the psychological state of the user.

[0226] The processing flow will be explained below.

[0227] Step 1:

[0228] Users speak into a device with a dedicated application installed to perform a specific task, for example, by saying, "Schedule a team meeting tomorrow at 10 AM."

[0229] Step 2:

[0230] The device captures the user's voice input through a microphone and converts it into digital audio data, which is temporarily stored on the device.

[0231] Step 3:

[0232] The device transmits the captured audio data to a server over the Internet using a secure communication protocol (e.g., HTTPS).

[0233] Step 4:

[0234] The server sends the received voice data to a speech recognition engine, which converts the voice data into text data, such as "Schedule a team meeting tomorrow at 10:00 AM."

[0235] Step 5:

[0236] The server sends the converted text and voice data to the emotion engine, which analyzes the user's emotions. The emotion engine identifies the user's emotional state from the tone of the voice and the content of the text. For example, if the voice tone sounds urgent, it is determined to be "nervous."

[0237] Step 6:

[0238] The server sends the parsed text data to a natural language processing (NLP) engine to identify the user's intent, which extracts the time information "tomorrow at 10 AM" and the task information "book a team meeting."

[0239] Step 7:

[0240] The server creates an API request to access the schedule management system based on the extracted task information. This API request includes task details (date and time, event content, etc.).

[0241] Step 8:

[0242] The server sends the created API request to the schedule management system. Specifically, it sends an HTTP POST request to the API endpoint of the schedule management system.

[0243] Step 9:

[0244] The schedule management system processes the received API request, creates a new event at the specified date and time, and returns a success response to the server when the processing is complete.

[0245] Step 10:

[0246] The server receives a response from the scheduling management system, confirms that the reservation was successful, and then generates a customized notification message based on the user's sentiment. For example, it could create a message like, "Your team meeting has been booked for tomorrow at 10 AM. Relax and move on to your next task!"

[0247] Step 11:

[0248] The server then sends the generated notification message to the terminal, again using a secure communication protocol (e.g., HTTPS).

[0249] Step 12:

[0250] The device displays the notification message received from the server to the user, for example, as a push notification or a pop-up message within the app.

[0251] Step 13:

[0252] The user will see a notification message on their device and know that the reservation has been made successfully, allowing them to move on to the next task with peace of mind.

[0253] Through the above steps, the system of the present invention can provide a system that not only enables efficient task management and business automation through voice input, but also takes into consideration the psychological state of the user.

[0254] Example 2

[0255] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0256] Conventional task management systems simply convert users' voice input into text data and identify tasks, but lack the ability to recognize users' emotions. Furthermore, they lack consideration for users, as they are unable to take their emotional state into account when managing tasks or automating tasks. Furthermore, notification messages that do not take users' emotions into account can potentially impair the user experience.

[0257] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing voice data and converting it into text data, a means for analyzing the text data and voice data and recognizing the user's emotions, and a means for analyzing the text data and identifying the corresponding task. This enables task management and business automation that take the user's emotions into consideration.

[0258] "Voice input" refers to data in a format in which the user inputs spoken words into the terminal using a microphone.

[0259] "Terminal means" refers to a device or apparatus for capturing voice input and performing the necessary processing.

[0260] "Server means" refers to a computer system that performs analysis and data processing, and manages and provides necessary information.

[0261] "Voice data" refers to digital data generated from the voice input by a user into a terminal.

[0262] "Text data" refers to data that has been analyzed and converted into textual information from audio data.

[0263] "Emotion recognition" refers to the process of analyzing voice and text data to determine a user's emotions and mental state.

[0264] "Task identification" refers to the process of analyzing text data to clarify the specific actions and instructions intended by the user.

[0265] A "schedule management system" refers to software or services for managing schedules and tasks.

[0266] "Notification" refers to messages or alerts that inform users of processing results or information.

[0267] A "natural language processing engine" refers to algorithms and technologies for analyzing text data and understanding and processing human language.

[0268] "Speech recognition engine" refers to the algorithms and technologies used to analyze voice data and convert it into corresponding text data.

[0269] The system of the present invention is equipped with a function for recognizing a user's emotions in addition to task management and business automation using voice input. MODES FOR CARRYING OUT THE INVENTION The following describes in detail embodiments of the present invention.

[0270] The user speaks into a device with a dedicated application installed. The device is equipped with a microphone to capture voice input, and while pressing the voice input button, the user speaks, "Schedule a team meeting tomorrow at 10:00 AM."

[0271] The device captures audio using a built-in microphone, temporarily stores the audio data, and then transmits it to a server via the internet. The hardware used here could be a smartphone or tablet.

[0272] The server analyzes the received voice data using a voice recognition engine (for example, a general voice recognition engine) and converts the voice data into text data. For example, text data such as "Please schedule a team meeting tomorrow at 10:00 AM" is generated.

[0273] The server then passes the converted text and voice data to an emotion engine (e.g., a general emotion analysis engine) to analyze the user's emotions. This determines whether the user is stressed or relaxed. For example, if the user's voice sounds rushed and tense, the emotion engine will recognize the user as being in a "tense" state.

[0274] Next, the server uses a natural language processing engine (e.g., a general natural language processing model) to analyze the text data in detail and identify the user's intent. In this case, the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" are extracted.

[0275] Based on the identified task (i.e., booking a team meeting), the server makes an API request to access the scheduling system (e.g., a common calendar API). This request includes the date and time to book and details of the event. The server sends this request to the scheduling system, adding the event to the specified time slot.

[0276] Once the reservation is successfully completed, the server generates a message to notify the device of the result. This message is customized based on the user's emotions. For example, if the user is nervous, the server generates a message saying, "Your team meeting has been booked for tomorrow at 10:00 AM. Relax and move on to the next task!"

[0277] The terminal displays the notification message received from the server, letting the user know that the reservation has been made successfully. The content of the notification message is designed to help the user relax.

[0278] For example, if a user says "Schedule a team meeting tomorrow at 10 AM" and the emotion engine recognizes the user's nervousness, the following happens:

[0279] 1. The user speaks into the device.

[0280] 2. The device sends the audio data to the server.

[0281] 3. The server uses a speech recognition engine to convert the speech into text data.

[0282] 4. The server uses the emotion engine to recognize the user's emotions.

[0283] 5. The server uses a natural language processing engine to analyze the text data and identify tasks.

[0284] 6. The server accesses the schedule management system and adds the event to the specified time slot.

[0285] 7. The server generates a notification message of the processing result that takes into consideration the user's feelings.

[0286] 8. The server notifies the terminal of the processing results.

[0287] 9. The device displays the notification to the user.

[0288] 10. The user acknowledges the notification, relaxes, and moves on to the next task.

[0289] This system allows users to efficiently manage tasks through voice input and automate work processes while taking into consideration their psychological state.

[0290] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0291] Step 1:

[0292] The user provides voice input.

[0293] The user speaks into a device with a dedicated application installed, presses the voice input button, and says, "Schedule a team meeting for tomorrow at 10 a.m." This voice becomes the input data.

[0294] Step 2:

[0295] The device captures and transmits the audio data.

[0296] The device captures the user's voice using a built-in microphone, temporarily stores it as digital audio data, and then transmits the audio data to a server via the Internet. The output is the digitized audio data sent to the server.

[0297] Step 3:

[0298] The server converts the voice data into text data.

[0299] The server analyzes the received voice data using a speech recognition engine and converts it into text data. The collected voice data is converted into text information, and the text output is, for example, "Schedule a team meeting tomorrow at 10:00 AM."

[0300] Step 4:

[0301] The server recognizes the user's emotions from the text data and voice data.

[0302] The server then passes the converted text and voice data to an emotion recognition engine, which analyzes the user's emotions based on the tone and speed of the voice and the content of the text. For example, if the voice is spoken quickly and with a high pitch, it is determined to be "tension," and this information is output.

[0303] Step 5:

[0304] The server analyzes the text data and identifies the user's intent.

[0305] The server uses a natural language processing engine to perform detailed analysis of the text data and extract task and time information. For example, the analysis identifies instructions such as "tomorrow at 10:00 AM" and "book a team meeting." The output is specific task instructions.

[0306] Step 6:

[0307] The server accesses the schedule management system and reserves the event.

[0308] The server creates a request to the schedule management system's API based on the extracted task information. This request includes the reservation date and time and event details. The server sends this request to the schedule management system, which adds the event to the specified time slot. The output is the reservation execution result.

[0309] Step 7:

[0310] The server generates a reservation completion notification and sends it to the terminal.

[0311] The server confirms that the reservation has been successfully completed and generates a customized message based on the user's sentiment, for example, "Your team meeting has been booked for tomorrow at 10 AM. Relax and move on to the next task!", and sends this to the terminal. As an output, a customized notification message is generated.

[0312] Step 8:

[0313] The device displays a notification message.

[0314] The device displays the notification message received from the server as a pop-up notification or an in-app message, and the user can confirm that the reservation has been successfully made.

[0315] This means:

[0316] This enables efficient task management using voice input and business automation that takes into account the user's psychological state.

[0317] (Application example 2)

[0318] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0319] Conventional voice input systems simply convert voice data into text data and execute tasks based on that text. However, because they do not take into account the user's emotional state, they have the problem of not contributing to stress reduction or motivation improvement. In the food delivery field in particular, it is an issue that they are unable to provide appropriate responses when the user is in a hurry or feeling stressed.

[0320] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0321] In this invention, the server includes means for identifying the user's emotion from the voice data and text data, means for generating customized feedback according to the identified emotion, and means for notifying the terminal means of the processing result and the emotion-considered feedback, thereby making it possible to provide a response customized according to the user's emotional state.

[0322] "Voice input" refers to the act of generating voice data by the user speaking.

[0323] The "terminal means" is a device that allows a user to input voice, and includes a microphone.

[0324] The "server means" is a computer system that analyzes and processes data received from the terminal means via a network.

[0325] "Voice data" refers to the user's voice signals captured by the terminal means.

[0326] "Text data" is a collection of character information obtained by analyzing voice data.

[0327] A "task" refers to a specific action or work that a user requests by voice input.

[0328] A "schedule management system" is a software application for completing tasks and coordinating schedules.

[0329] "Emotion identification" is the process of analyzing audio and text data to determine the user's psychological state.

[0330] "Customized feedback" refers to response messages and guidance that are individually tailored to the user's emotional state.

[0331] The system of the present invention utilizes voice input for task management and business automation, and further recognizes the user's emotions to provide customized feedback. Detailed embodiments for implementing the present invention will be described below.

[0332] Hardware and software used

[0333] Hardware

[0334] Terminal means: A smartphone is used as a device for capturing voice input. This smartphone has a built-in microphone.

[0335] Server means: A computer system is used to analyze data and perform necessary processing.

[0336] software

[0337] Speech recognition engine: Uses the speech_recognition library to convert voice data into text data.

[0338] Natural Language Processing Engine: Uses the TextBlob library to analyze text data and identify tasks and sentiments.

[0339] HTTP Request Library: The requests library is used to communicate between the server means and the schedule management system and the feedback processing system.

[0340] System processing procedure

[0341] 1. Capture voice input

[0342] The user speaks into the microphone of the smartphone. Voice input is the act of generating voice data by speaking. For example, the user may say, "I'd like one pizza, please."

[0343] 2. Converting voice data to text

[0344] The voice data captured by the terminal means is transmitted to the server means, which converts the voice data into text data using a speech_recognition library.

[0345] 3. Emotion Identification

[0346] The server identifies the user's emotion from the converted text data and voice data. It analyzes the text data using the TextBlob library to determine the user's emotional state. For example, if the text is "One pizza please," it determines whether the user is "relaxed" or "in a hurry" based on the content and tone of the voice.

[0347] 4. Task execution

[0348] The server means sends an order to the food delivery API using the requests library based on the identified task, with the text data and emotion data included in the payload.

[0349] 5. Feedback Generation and Notification

[0350] The server generates customized feedback according to the user's emotional state and notifies the terminal, thereby allowing the user to move on to the next task without feeling stressed. For example, if the user is relaxed, a message such as "Your order has been completed. Please wait a moment" is displayed.

[0351] Specific examples

[0352] For example, if a user says "I'd like one pizza, please," the system converts the order into text, recognizes the user's emotion as "relaxed," sends the order to a food delivery API, and then displays feedback such as "Your order has been completed. Please wait a moment."

[0353] Prompt Sentence Examples

[0354] "Voice input: 'One pizza please'

[0355] Emotion Recognition: Relax

[0356] Action: User is relaxed, so show 'Your order is complete. Please wait a moment.'

[0357] In this way, the system of the present invention realizes efficient task management and business automation through voice input, and also provides services that take into consideration the psychological state of the user.

[0358] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0359] Step 1:

[0360] Capturing voice input

[0361] The user inputs voice by speaking into the terminal means. For example, the user may say, "I'd like one pizza, please." The terminal means uses a microphone to capture this voice data.

[0362] Input: User's voice

[0363] Output: Captured audio data

[0364] Step 2:

[0365] Converting audio data to text

[0366] The voice data captured by the terminal means is transmitted to the server means, which analyzes the voice data using the speech_recognition library and converts it into text data.

[0367] Input: Captured audio data

[0368] Output: Text data

[0369] Step 3:

[0370] Emotion Identification

[0371] The server identifies the user's emotion from the text data and voice data. It analyzes the text data using the TextBlob library to determine the user's emotional state. For example, if the text is "One pizza please" and the voice tone is calm, it determines that the user is "relaxed."

[0372] Input: Text data and tone information of audio data

[0373] Output: User's emotional state (e.g., relaxed, rushed, stressed)

[0374] Step 4:

[0375] Task execution

[0376] The server sends an order to the food delivery API based on the identified task, with the text data and emotion data included in the payload. It then uses the requests library to create an HTTP request and process the order.

[0377] Input: Text data, user's emotional state

[0378] Output: API request response (order confirmation)

[0379] Step 5:

[0380] Feedback Generation and Notification

[0381] The server means generates a feedback message customized according to the user's emotional state and notifies the terminal means of the message. For example, if the user is relaxed, the server means generates a message saying, "Your order is complete. Please wait a moment."

[0382] Input: API request response, user emotional state

[0383] Output: Customized feedback message

[0384] Step 6:

[0385] View Feedback

[0386] The terminal means displays the feedback message received from the server means to the user, thereby confirming that the order has been processed correctly.

[0387] Input: Customized feedback message

[0388] Output: Display of feedback message

[0389] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0390] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0391] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0392] [Second embodiment]

[0393] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0394] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0395] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0396] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0397] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0398] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0399] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0400] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0401] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0402] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0403] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0404] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0405] The system of the present invention is designed to realize task management and business automation using voice input. The following describes in detail the embodiments of the present invention.

[0406] First, a user uses a device with a dedicated application installed to input voice. This device is equipped with a microphone to capture voice input. The user presses the voice input button and inputs voice, such as "Schedule a team meeting tomorrow at 10:00 AM."

[0407] The audio data is then captured by the terminal and transmitted over the Internet to a server.

[0408] The server uses a speech recognition engine to analyze the received voice data and convert it into text data, in this case "Schedule a team meeting tomorrow at 10 AM."

[0409] The server then uses a natural language processing engine to analyze the text data and identify the user's intent, extracting the time information "tomorrow at 10:00 AM" and the task information "book a team meeting."

[0410] Based on the identified task (i.e., booking a team meeting), the server accesses a schedule management system (e.g., a calendar service) and performs an operation to add an event for the specified time slot. The server calls the appropriate API to complete the booking.

[0411] If the reservation is successful, the server generates a message to notify the terminal of the result, for example, a message saying "A team meeting has been reserved for tomorrow at 10:00 AM."

[0412] The terminal displays the received notification message to the user, and the user confirms that the reservation has been made correctly.

[0413] As a concrete example, if a user says "Schedule a team meeting tomorrow at 10 AM," the following occurs:

[0414] 1. The user speaks into the device.

[0415] 2. The device sends the audio data to the server.

[0416] 3. The server uses a speech recognition engine to convert the speech into text data.

[0417] 4. The server uses a natural language processing engine to analyze the text data and identify tasks.

[0418] 5. The server accesses the schedule management system and adds the event to the specified time slot.

[0419] 6. The server notifies the terminal of the processing results.

[0420] 7. The device displays the notification to the user.

[0421] 8. The user confirms the notification.

[0422] Through this series of processes, the system of the present invention realizes efficient task management and business automation using voice input.

[0423] The processing flow will be explained below.

[0424] Step 1:

[0425] The user speaks into a device with a dedicated application installed, pressing the voice input button and saying something like, "Schedule a team meeting for tomorrow at 10:00 AM."

[0426] Step 2:

[0427] The device captures voice input from the user and converts it into digital audio data, which is temporarily stored on the device.

[0428] Step 3:

[0429] The device transmits the captured audio data to a server over the internet using a secure communication protocol such as HTTPS or WebSocket.

[0430] Step 4:

[0431] The server passes the received voice data to a speech recognition engine, which converts the digital voice into text data, which then translates to "Schedule a team meeting tomorrow at 10 AM."

[0432] Step 5:

[0433] The server receives the text data and passes it to a natural language processing engine for analysis, which identifies the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" from the text.

[0434] Step 6:

[0435] The server uses the identified task information to create an API request to the schedule management system, including the date and time to be booked and details of the event.

[0436] Step 7:

[0437] The server sends an API request to the schedule management system, for example, an HTTP POST request to the API endpoint of a calendar service.

[0438] Step 8:

[0439] The schedule management system processes the received API request, creates a new event at the specified date and time, and returns a success response to the server if the process is completed successfully.

[0440] Step 9:

[0441] The server receives the response from the schedule management system, confirms that the reservation was successful, and then generates a notification message as a result of the processing: "A team meeting has been booked for tomorrow at 10:00 AM."

[0442] Step 10:

[0443] The server then sends the generated notification message to the device, again using a secure communication protocol such as HTTPS or WebSocket.

[0444] Step 11:

[0445] The device displays the notification message received from the server to the user, for example, as a push notification or a pop-up message within the app.

[0446] Step 12:

[0447] The user will see a notification message on the terminal and will know that the reservation has been made successfully. Based on the notification, the user can consider the next action to take.

[0448] The above steps enable efficient task management and business automation through voice input.

[0449] Example 1

[0450] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0451] While there are a wide variety of solutions for task management and work efficiency today, efficient and intuitive methods using voice input have not yet been fully established. In particular, there is room for improvement in systems that automatically schedule tasks using voice and easily realize complex natural language processing. There is a need to significantly reduce user effort by providing a consistent automated process from voice input, task identification, registration in a schedule management system, and notification of the results.

[0452] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0453] In this invention, the server includes means for analyzing received voice data using a voice recognition engine and converting it into text data, means for analyzing the text data using a natural language processing engine and identifying the corresponding task, and means for accessing the schedule management system based on the identified task and executing the necessary processing. This makes it possible to consistently automate everything from voice input to task identification, automatic schedule addition, and notification of processing results.

[0454] "Voice input" is the act of conveying information or instructions to a system using voice.

[0455] The "terminal means" is a device that allows a user to input voice data and capture and transmit the data.

[0456] The "server means" is a device or program for analyzing received data and executing various processes.

[0457] "Audio Data" means a digital representation of the sound captured by an audio input.

[0458] "Text data" is character information that has been converted by analyzing voice data.

[0459] A "voice recognition engine" is software or hardware that analyzes voice data and converts it into text data.

[0460] A "natural language processing engine" is software or hardware that analyzes text data and understands and interprets human language.

[0461] A "task" is a specific activity or work undertaken to achieve a particular goal.

[0462] A "schedule management system" is software or hardware for organizing and managing tasks and events in chronological order.

[0463] "Notification" refers to a message or information presentation that conveys processing results or information to the user.

[0464] "Analysis" is the process of extracting meaning and information from data and understanding it.

[0465] The system of the present invention is designed to realize task management and business automation using voice input. This system includes a terminal means for capturing voice input, a server means for analyzing the voice data and converting it into text data, a server means for analyzing the text data and identifying tasks, a server means for accessing a schedule management system based on the identified tasks and executing processing, and a server means for notifying the terminal means of the processing results.

[0466] First, a user uses a device with a dedicated application installed to input voice data. This device is equipped with a microphone to capture voice data. When the user inputs voice data such as "Schedule a team meeting tomorrow at 10:00 AM," the voice data is captured by the device and sent to the server via the Internet.

[0467] The server uses a speech recognition engine (e.g., speech recognition software) to analyze the received voice data and convert it into text data, in this case, "Schedule a team meeting tomorrow at 10:00 AM."

[0468] The server then uses a natural language processing engine (e.g., a natural language processing API) to analyze the text data and identify the user's intent. In this specific process, the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" are extracted.

[0469] Based on the identified task, i.e., booking a team meeting, the server accesses the schedule management system (e.g., a calendar service) and performs an operation to add an event for the specified time slot. The server performs this operation by calling the appropriate API.

[0470] When the reservation is successfully completed, the server generates a message to notify the terminal of the result. For example, a message saying "A team meeting has been reserved for tomorrow at 10:00 AM" is sent to the terminal, which displays it to the user. The user confirms the notification and knows that the reservation was made successfully.

[0471] As a specific example, the processing flow when a user voice-inputs "Book a team meeting for tomorrow at 10:00 AM" will be described.

[0472] In addition, by inputting the following prompt sentence into the generative AI model, an explanation based on the system's processing flow can be obtained.

[0473] Take a step-by-step guide to how the system works, given the voice input "Schedule a team meeting tomorrow at 10am."

[0474] The prompt sentences provide detailed explanations of the specific operations of each processing step.

[0475] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0476] Step 1:

[0477] The user provides voice input. Using a device with a dedicated application installed, the user says, "Schedule a team meeting for tomorrow at 10:00 AM."

[0478] Input: User's voice commands

[0479] Output: Audio data (digital format)

[0480] Step 2:

[0481] The device sends the audio data to the server using a secure communication protocol (e.g., HTTPS).

[0482] Input: Audio data

[0483] Output: Request sent to server

[0484] Step 3:

[0485] The server converts the voice data into text data. The server uses a voice recognition engine to analyze the voice data and convert it into text data.

[0486] Input: Audio data

[0487] Output: Text data: "Schedule a team meeting tomorrow at 10 AM"

[0488] Step 4:

[0489] The server analyzes the text data and identifies the task. The server uses a natural language processing engine to extract the intent from the text data. In this case, the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" are extracted.

[0490] Input: Text data: "Schedule a team meeting tomorrow at 10 AM"

[0491] Output: Time information "Tomorrow at 10 AM" and task information "Schedule a team meeting"

[0492] Step 5:

[0493] The server sends a request to the schedule management system. Based on the identified task information, the server sends an API request to add an event to the schedule management system.

[0494] Input: Time information "Tomorrow at 10 AM" and task information "Schedule a team meeting"

[0495] Output: A request to add an event to the schedule management system.

[0496] Step 6:

[0497] The server generates a notification message of the processing result, creates a notification of successful reservation, and sends it to the terminal.

[0498] Input: Response from the schedule management system (if successful)

[0499] Output: Notification message: "Your team meeting has been scheduled for tomorrow at 10 AM."

[0500] Step 7:

[0501] The device receives the notification message and displays it to the user, either in the notification center within the application or as a pop-up message.

[0502] Input: Notification message: "Team meeting booked for tomorrow at 10am"

[0503] Output: Display notification

[0504] Step 8:

[0505] The user checks the notification. The user checks the notification on the terminal and realizes that the reservation has been made successfully.

[0506] Input: The displayed notification message

[0507] Output: Recognition (no specific output)

[0508] (Application example 1)

[0509] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0510] In modern factories, many tasks are performed manually, resulting in issues with efficiency and accuracy. Misunderstandings and delays are particularly common when communicating work instructions and executing tasks. Frequent on-site operations by workers also increase the workload. There is a need for a system that can improve this situation and increase efficiency by automating factory operations.

[0511] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0512] In this invention, the server includes terminal means for capturing voice input, means for analyzing voice data received from the terminal means and converting it into text data, means for analyzing the text data and identifying the corresponding task, means for accessing the work machine control system based on the identified task and executing the required processing, means for notifying the terminal means of the processing results, and terminal means for displaying the notification. This makes it possible to efficiently and accurately transmit work instructions and automate work using voice instructions.

[0513] The "terminal means for capturing voice input" refers to a device equipped to receive a user's voice, such as a microphone or a smartphone.

[0514] "Means for analysis" refers to the technology or process for converting voice data into text data and analyzing the user's intent from that text data.

[0515] "Text data" refers to character string data converted from speech input, which can then be analyzed.

[0516] "Means for identification" refers to the technology or algorithm that finds and recognizes the corresponding task from the analyzed text data.

[0517] A "work machine control system" is a system that controls various machines and robots used in factories and production lines to carry out tasks.

[0518] "Means for performing processing" refers to hardware or software that performs actual actions or operations based on a specified task.

[0519] "Means for notifying" refers to the method or device for notifying the user of the processing results or status.

[0520] "Terminal means" refers to a device operated by a user, and is used for voice input, display of notifications, etc.

[0521] A "schedule management system" is a system that users use to manage tasks and events, including calendar apps and scheduling software.

[0522] A "voice recognition engine" is software or algorithms that analyze voice data and convert it into text data.

[0523] A "natural language processing engine" is a technology or algorithm that analyzes text data and understands user intent and commands.

[0524] The system of the present invention is designed to automate work instructions and operations in a factory using voice input. In this system, a user gives instructions by voice, and the robot automatically executes the work by analyzing the contents of the instructions. Specific embodiments are described in detail below.

[0525] First, the user uses a device with a dedicated application installed to input voice. This device is equipped with a microphone to capture voice input. The user presses the voice input button and inputs voice, such as "Move part B to the next line."

[0526] Next, the voice data is captured by the device and sent to a server via the Internet. The server uses a voice recognition engine to analyze the received voice data and convert it into text data. In this case, it is converted into text data such as "Move part B to the next line."

[0527] Next, the server uses a natural language processing engine to analyze the text data and identify the user's intent. In this case, the server extracts the movement instruction "to the next line" and the specific target "Part B."

[0528] Based on the identified task (i.e., moving a part using a factory robot), the server accesses the work machine control system and performs operations to execute the specified task. The server calls the APIs and protocols for robot control to execute the movement instructions.

[0529] When the operation is completed successfully, the server generates a message to notify the terminal of the result. For example, a message saying "Part B has moved to the next line" is generated. The terminal displays the received notification message to the user, who can confirm that the operation was performed correctly.

[0530] As a concrete example, if the user speaks "Move part B to the next line," the following prompt sentence is generated:

[0531] "Move part B to the next line."

[0532] This prompt allows the system to understand the user's instructions and the factory robot to automatically carry out the specified work. Through this series of processes, the system of the present invention realizes efficient and accurate transmission of work instructions and automation of work using voice input.

[0533] The hardware used includes microphones and smartphones to capture voice input and factory robots to perform the tasks, while the software used includes a speech recognition engine, a natural language processing engine, and an API for controlling the robots.

[0534] As described above, the system of the present invention efficiently transmits instructions for factory work through voice input, thereby realizing automation of work.

[0535] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0536] Step 1:

[0537] The user provides voice input.

[0538] Input: User speech (e.g., "Move part B to the next line").

[0539] Output: Audio data.

[0540] Specific operation: The user presses the voice input button and speaks, and the device's microphone captures the voice.

[0541] Step 2:

[0542] The terminal transmits the voice data to the server.

[0543] Input: Audio data.

[0544] Output: The audio data sent to the server.

[0545] Specific operation: The device sends the captured audio data to a server via the Internet.

[0546] Step 3:

[0547] The server converts the voice data into text data using a voice recognition engine.

[0548] Input: Audio data.

[0549] Output: Text data (e.g. "Move part B to the next line").

[0550] Specific operation: The server passes the received voice data to a voice recognition engine and converts the data into text format.

[0551] Step 4:

[0552] The server uses a natural language processing engine to analyze the text data and identify the user's intent.

[0553] Input: Text data.

[0554] Output: Data specifying the intent (e.g., "Task to move part B to the next line").

[0555] Specific operation: The server passes the text data to a natural language processing engine for analysis to identify the user's intent and task.

[0556] Step 5:

[0557] The server accesses the work machine control system and performs the necessary processing.

[0558] Input: Identified task data.

[0559] Output: Robot control command (e.g. "Move part B from line A to line B").

[0560] Specific operation: Based on the identified task, the server calls the API of the work machine control system to give instructions to the robot.

[0561] Step 6:

[0562] The processing result is notified to the terminal.

[0563] Input: Processing result (e.g., work completion notification).

[0564] Output: Notification message to the terminal (e.g. "Part B has moved to the next line").

[0565] Specific operation: The server receives the processing result from the work machine control system, generates a notification message, and sends it to the terminal.

[0566] Step 7:

[0567] The terminal displays the notification to the user.

[0568] Input: Notification message.

[0569] Output: A user-visible notification (e.g., "Part B has moved to the next line").

[0570] Specific operation: The device receives a notification message from the server and notifies the user by displaying the message on the screen or by voice notification.

[0571] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0572] The system of the present invention is equipped with a function for recognizing a user's emotions in addition to task management and business automation using voice input. MODES FOR CARRYING OUT THE INVENTION The following describes in detail embodiments of the present invention.

[0573] First, the user speaks into a device with a dedicated application installed. The device is equipped with a microphone for capturing voice input, and while pressing the voice input button, the user speaks, "Schedule a team meeting tomorrow at 10:00 AM."

[0574] The audio data is then captured by the terminal and transmitted over the Internet to a server.

[0575] The server uses a speech recognition engine to analyze the received voice data and convert it into text data, which in this case is "Schedule a team meeting tomorrow at 10 AM."

[0576] The server then passes the text and voice data to an emotion engine to identify the user's emotion. The emotion engine determines whether the user is stressed or relaxed based on the tone of the voice and the content of the text. For example, if the user's voice sounds urgent and tense, the emotion engine will recognize the user as being in a "tense" state.

[0577] The server then uses a natural language processing engine to analyze the text data and identify the user's intent, extracting the time information "tomorrow at 10:00 AM" and the task information "book a team meeting."

[0578] Based on the identified task (i.e., booking a team meeting), the server makes an API request to the scheduling system, including the date and time to book and details of the event.

[0579] The server sends an API request to the schedule management system and performs an operation to add an event to the specified time slot. For example, it calls the API of the schedule management system to execute a reservation.

[0580] Once the reservation is successfully completed, the server generates a message to notify the device of the result. This message is customized based on the user's emotions. For example, if the user is nervous, the server generates a message saying, "Your team meeting has been booked for tomorrow at 10:00 AM. Relax and move on to the next task!"

[0581] The terminal displays the received notification message to the user, and the user confirms that the reservation has been made correctly. The content of the notification message is designed to help the user relax.

[0582] For example, if a user says "Schedule a team meeting tomorrow at 10 AM" and the emotion engine recognizes the user's nervousness, the following happens:

[0583] 1. The user speaks into the device.

[0584] 2. The device sends the audio data to the server.

[0585] 3. The server uses a speech recognition engine to convert the speech into text data.

[0586] 4. The server uses the emotion engine to recognize the user's emotions.

[0587] 5. The server uses a natural language processing engine to analyze the text data and identify tasks.

[0588] 6. The server accesses the schedule management system and adds the event to the specified time slot.

[0589] 7. The server generates a notification message of the processing result that takes into consideration the user's feelings.

[0590] 8. The server notifies the terminal of the processing results.

[0591] 9. The device displays the notification to the user.

[0592] 10. The user acknowledges the notification, relaxes, and moves on to the next task.

[0593] As a result, the system of the present invention can provide a system that not only enables efficient task management and business automation through voice input, but also takes into consideration the psychological state of the user.

[0594] The processing flow will be explained below.

[0595] Step 1:

[0596] Users speak into a device with a dedicated application installed to perform a specific task, for example, by saying, "Schedule a team meeting tomorrow at 10 AM."

[0597] Step 2:

[0598] The device captures the user's voice input through a microphone and converts it into digital audio data, which is temporarily stored on the device.

[0599] Step 3:

[0600] The device transmits the captured audio data to a server over the Internet using a secure communication protocol (e.g., HTTPS).

[0601] Step 4:

[0602] The server sends the received voice data to a speech recognition engine, which converts the voice data into text data, such as "Schedule a team meeting tomorrow at 10:00 AM."

[0603] Step 5:

[0604] The server sends the converted text and voice data to the emotion engine, which analyzes the user's emotions. The emotion engine identifies the user's emotional state from the tone of the voice and the content of the text. For example, if the voice tone sounds urgent, it is determined to be "nervous."

[0605] Step 6:

[0606] The server sends the parsed text data to a natural language processing (NLP) engine to identify the user's intent, which extracts the time information "tomorrow at 10 AM" and the task information "book a team meeting."

[0607] Step 7:

[0608] The server creates an API request to access the schedule management system based on the extracted task information. This API request includes task details (date and time, event content, etc.).

[0609] Step 8:

[0610] The server sends the created API request to the schedule management system. Specifically, it sends an HTTP POST request to the API endpoint of the schedule management system.

[0611] Step 9:

[0612] The schedule management system processes the received API request, creates a new event at the specified date and time, and returns a success response to the server when the processing is complete.

[0613] Step 10:

[0614] The server receives a response from the scheduling management system, confirms that the reservation was successful, and then generates a customized notification message based on the user's sentiment. For example, it could create a message like, "Your team meeting has been booked for tomorrow at 10 AM. Relax and move on to your next task!"

[0615] Step 11:

[0616] The server then sends the generated notification message to the terminal, again using a secure communication protocol (e.g., HTTPS).

[0617] Step 12:

[0618] The device displays the notification message received from the server to the user, for example, as a push notification or a pop-up message within the app.

[0619] Step 13:

[0620] The user will see a notification message on their device and know that the reservation has been made successfully, allowing them to move on to the next task with peace of mind.

[0621] Through the above steps, the system of the present invention can provide a system that not only enables efficient task management and business automation through voice input, but also takes into consideration the psychological state of the user.

[0622] Example 2

[0623] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0624] Conventional task management systems simply convert users' voice input into text data and identify tasks, but lack the ability to recognize users' emotions. Furthermore, they lack consideration for users, as they are unable to take their emotional state into account when managing tasks or automating tasks. Furthermore, notification messages that do not take users' emotions into account can potentially impair the user experience.

[0625] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing voice data and converting it into text data, a means for analyzing the text data and voice data and recognizing the user's emotions, and a means for analyzing the text data and identifying the corresponding task. This enables task management and business automation that take the user's emotions into consideration.

[0626] "Voice input" refers to data in a format in which the user inputs spoken words into the terminal using a microphone.

[0627] "Terminal means" refers to a device or apparatus for capturing voice input and performing the necessary processing.

[0628] "Server means" refers to a computer system that performs analysis and data processing, and manages and provides necessary information.

[0629] "Voice data" refers to digital data generated from the voice input by a user into a terminal.

[0630] "Text data" refers to data that has been analyzed and converted into textual information from audio data.

[0631] "Emotion recognition" refers to the process of analyzing voice and text data to determine a user's emotions and mental state.

[0632] "Task identification" refers to the process of analyzing text data to clarify the specific actions and instructions intended by the user.

[0633] A "schedule management system" refers to software or services for managing schedules and tasks.

[0634] "Notification" refers to messages or alerts that inform users of processing results or information.

[0635] A "natural language processing engine" refers to algorithms and technologies for analyzing text data and understanding and processing human language.

[0636] "Speech recognition engine" refers to the algorithms and technologies used to analyze voice data and convert it into corresponding text data.

[0637] The system of the present invention is equipped with a function for recognizing a user's emotions in addition to task management and business automation using voice input. MODES FOR CARRYING OUT THE INVENTION The following describes in detail embodiments of the present invention.

[0638] The user speaks into a device with a dedicated application installed. The device is equipped with a microphone to capture voice input, and while pressing the voice input button, the user speaks, "Schedule a team meeting tomorrow at 10:00 AM."

[0639] The device captures audio using a built-in microphone, temporarily stores the audio data, and then transmits it to a server via the internet. The hardware used here could be a smartphone or tablet.

[0640] The server analyzes the received voice data using a voice recognition engine (for example, a general voice recognition engine) and converts the voice data into text data. For example, text data such as "Please schedule a team meeting tomorrow at 10:00 AM" is generated.

[0641] The server then passes the converted text and voice data to an emotion engine (e.g., a general emotion analysis engine) to analyze the user's emotions. This determines whether the user is stressed or relaxed. For example, if the user's voice sounds rushed and tense, the emotion engine will recognize the user as being in a "tense" state.

[0642] Next, the server uses a natural language processing engine (e.g., a general natural language processing model) to analyze the text data in detail and identify the user's intent. In this case, the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" are extracted.

[0643] Based on the identified task (i.e., booking a team meeting), the server makes an API request to access the scheduling system (e.g., a common calendar API). This request includes the date and time to book and details of the event. The server sends this request to the scheduling system, adding the event to the specified time slot.

[0644] Once the reservation is successfully completed, the server generates a message to notify the device of the result. This message is customized based on the user's emotions. For example, if the user is nervous, the server generates a message saying, "Your team meeting has been booked for tomorrow at 10:00 AM. Relax and move on to the next task!"

[0645] The terminal displays the notification message received from the server, letting the user know that the reservation has been made successfully. The content of the notification message is designed to help the user relax.

[0646] For example, if a user says "Schedule a team meeting tomorrow at 10 AM" and the emotion engine recognizes the user's nervousness, the following happens:

[0647] 1. The user speaks into the device.

[0648] 2. The device sends the audio data to the server.

[0649] 3. The server uses a speech recognition engine to convert the speech into text data.

[0650] 4. The server uses the emotion engine to recognize the user's emotions.

[0651] 5. The server uses a natural language processing engine to analyze the text data and identify tasks.

[0652] 6. The server accesses the schedule management system and adds the event to the specified time slot.

[0653] 7. The server generates a notification message of the processing result that takes into consideration the user's feelings.

[0654] 8. The server notifies the terminal of the processing results.

[0655] 9. The device displays the notification to the user.

[0656] 10. The user acknowledges the notification, relaxes, and moves on to the next task.

[0657] This system allows users to efficiently manage tasks through voice input and automate work processes while taking into consideration their psychological state.

[0658] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0659] Step 1:

[0660] The user provides voice input.

[0661] The user speaks into a device with a dedicated application installed, presses the voice input button, and says, "Schedule a team meeting for tomorrow at 10 a.m." This voice becomes the input data.

[0662] Step 2:

[0663] The device captures and transmits the audio data.

[0664] The device captures the user's voice using a built-in microphone, temporarily stores it as digital audio data, and then transmits the audio data to a server via the Internet. The output is the digitized audio data sent to the server.

[0665] Step 3:

[0666] The server converts the voice data into text data.

[0667] The server analyzes the received voice data using a speech recognition engine and converts it into text data. The collected voice data is converted into text information, and the text output is, for example, "Schedule a team meeting tomorrow at 10:00 AM."

[0668] Step 4:

[0669] The server recognizes the user's emotions from the text data and voice data.

[0670] The server then passes the converted text and voice data to an emotion recognition engine, which analyzes the user's emotions based on the tone and speed of the voice and the content of the text. For example, if the voice is spoken quickly and with a high pitch, it is determined to be "tension," and this information is output.

[0671] Step 5:

[0672] The server analyzes the text data and identifies the user's intent.

[0673] The server uses a natural language processing engine to perform detailed analysis of the text data and extract task and time information. For example, the analysis identifies instructions such as "tomorrow at 10:00 AM" and "book a team meeting." The output is specific task instructions.

[0674] Step 6:

[0675] The server accesses the schedule management system and reserves the event.

[0676] The server creates a request to the schedule management system's API based on the extracted task information. This request includes the reservation date and time and event details. The server sends this request to the schedule management system, which adds the event to the specified time slot. The output is the reservation execution result.

[0677] Step 7:

[0678] The server generates a reservation completion notification and sends it to the terminal.

[0679] The server confirms that the reservation has been successfully completed and generates a customized message based on the user's sentiment, for example, "Your team meeting has been booked for tomorrow at 10 AM. Relax and move on to the next task!", and sends this to the terminal. As an output, a customized notification message is generated.

[0680] Step 8:

[0681] The device displays a notification message.

[0682] The device displays the notification message received from the server as a pop-up notification or an in-app message, and the user can confirm that the reservation has been successfully made.

[0683] This means:

[0684] This enables efficient task management using voice input and business automation that takes into account the user's psychological state.

[0685] (Application example 2)

[0686] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0687] Conventional voice input systems simply convert voice data into text data and execute tasks based on that text. However, because they do not take into account the user's emotional state, they have the problem of not contributing to stress reduction or motivation improvement. In the food delivery field in particular, it is an issue that they are unable to provide appropriate responses when the user is in a hurry or feeling stressed.

[0688] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0689] In this invention, the server includes means for identifying the user's emotion from the voice data and text data, means for generating customized feedback according to the identified emotion, and means for notifying the terminal means of the processing result and the emotion-considered feedback, thereby making it possible to provide a response customized according to the user's emotional state.

[0690] "Voice input" refers to the act of generating voice data by the user speaking.

[0691] The "terminal means" is a device that allows a user to input voice, and includes a microphone.

[0692] The "server means" is a computer system that analyzes and processes data received from the terminal means via a network.

[0693] "Voice data" refers to the user's voice signals captured by the terminal means.

[0694] "Text data" is a collection of character information obtained by analyzing voice data.

[0695] A "task" refers to a specific action or work that a user requests by voice input.

[0696] A "schedule management system" is a software application for completing tasks and coordinating schedules.

[0697] "Emotion identification" is the process of analyzing audio and text data to determine the user's psychological state.

[0698] "Customized feedback" refers to response messages and guidance that are individually tailored to the user's emotional state.

[0699] The system of the present invention utilizes voice input for task management and business automation, and further recognizes the user's emotions to provide customized feedback. Detailed embodiments for implementing the present invention will be described below.

[0700] Hardware and software used

[0701] Hardware

[0702] Terminal means: A smartphone is used as a device for capturing voice input. This smartphone has a built-in microphone.

[0703] Server means: A computer system is used to analyze data and perform necessary processing.

[0704] software

[0705] Speech recognition engine: Uses the speech_recognition library to convert voice data into text data.

[0706] Natural Language Processing Engine: Uses the TextBlob library to analyze text data and identify tasks and sentiments.

[0707] HTTP Request Library: The requests library is used to communicate between the server means and the schedule management system and the feedback processing system.

[0708] System processing procedure

[0709] 1. Capture voice input

[0710] The user speaks into the microphone of the smartphone. Voice input is the act of generating voice data by speaking. For example, the user may say, "I'd like one pizza, please."

[0711] 2. Converting voice data to text

[0712] The voice data captured by the terminal means is transmitted to the server means, which converts the voice data into text data using a speech_recognition library.

[0713] 3. Emotion Identification

[0714] The server identifies the user's emotion from the converted text data and voice data. It analyzes the text data using the TextBlob library to determine the user's emotional state. For example, if the text is "One pizza please," it determines whether the user is "relaxed" or "in a hurry" based on the content and tone of the voice.

[0715] 4. Task execution

[0716] The server means sends an order to the food delivery API using the requests library based on the identified task, with the text data and emotion data included in the payload.

[0717] 5. Feedback Generation and Notification

[0718] The server generates customized feedback according to the user's emotional state and notifies the terminal, thereby allowing the user to move on to the next task without feeling stressed. For example, if the user is relaxed, a message such as "Your order has been completed. Please wait a moment" is displayed.

[0719] Specific examples

[0720] For example, if a user says "I'd like one pizza, please," the system converts the order into text, recognizes the user's emotion as "relaxed," sends the order to a food delivery API, and then displays feedback such as "Your order has been completed. Please wait a moment."

[0721] Prompt Sentence Examples

[0722] "Voice input: 'One pizza please'

[0723] Emotion Recognition: Relax

[0724] Action: User is relaxed, so show 'Your order is complete. Please wait a moment.'

[0725] In this way, the system of the present invention realizes efficient task management and business automation through voice input, and also provides services that take into consideration the psychological state of the user.

[0726] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0727] Step 1:

[0728] Capturing voice input

[0729] The user inputs voice by speaking into the terminal means. For example, the user may say, "I'd like one pizza, please." The terminal means uses a microphone to capture this voice data.

[0730] Input: User's voice

[0731] Output: Captured audio data

[0732] Step 2:

[0733] Converting audio data to text

[0734] The voice data captured by the terminal means is transmitted to the server means, which analyzes the voice data using the speech_recognition library and converts it into text data.

[0735] Input: Captured audio data

[0736] Output: Text data

[0737] Step 3:

[0738] Emotion Identification

[0739] The server identifies the user's emotion from the text data and voice data. It analyzes the text data using the TextBlob library to determine the user's emotional state. For example, if the text is "One pizza please" and the voice tone is calm, it determines that the user is "relaxed."

[0740] Input: Text data and tone information of audio data

[0741] Output: User's emotional state (e.g., relaxed, rushed, stressed)

[0742] Step 4:

[0743] Task execution

[0744] The server sends an order to the food delivery API based on the identified task, with the text data and emotion data included in the payload. It then uses the requests library to create an HTTP request and process the order.

[0745] Input: Text data, user's emotional state

[0746] Output: API request response (order confirmation)

[0747] Step 5:

[0748] Feedback Generation and Notification

[0749] The server means generates a feedback message customized according to the user's emotional state and notifies the terminal means of the message. For example, if the user is relaxed, the server means generates a message saying, "Your order is complete. Please wait a moment."

[0750] Input: API request response, user emotional state

[0751] Output: Customized feedback message

[0752] Step 6:

[0753] View Feedback

[0754] The terminal means displays the feedback message received from the server means to the user, thereby confirming that the order has been processed correctly.

[0755] Input: Customized feedback message

[0756] Output: Display of feedback message

[0757] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0758] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0759] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0760] [Third embodiment]

[0761] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0762] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0763] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0764] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0765] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0766] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0767] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0768] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0769] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0770] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0771] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0772] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0773] The system of the present invention is designed to realize task management and business automation using voice input. The following describes in detail the embodiments of the present invention.

[0774] First, a user uses a device with a dedicated application installed to input voice. This device is equipped with a microphone to capture voice input. The user presses the voice input button and inputs voice, such as "Schedule a team meeting tomorrow at 10:00 AM."

[0775] The audio data is then captured by the terminal and transmitted over the Internet to a server.

[0776] The server uses a speech recognition engine to analyze the received voice data and convert it into text data, in this case "Schedule a team meeting tomorrow at 10 AM."

[0777] The server then uses a natural language processing engine to analyze the text data and identify the user's intent, extracting the time information "tomorrow at 10:00 AM" and the task information "book a team meeting."

[0778] Based on the identified task (i.e., booking a team meeting), the server accesses a schedule management system (e.g., a calendar service) and performs an operation to add an event for the specified time slot. The server calls the appropriate API to complete the booking.

[0779] If the reservation is successful, the server generates a message to notify the terminal of the result, for example, a message saying "A team meeting has been reserved for tomorrow at 10:00 AM."

[0780] The terminal displays the received notification message to the user, and the user confirms that the reservation has been made correctly.

[0781] As a concrete example, if a user says "Schedule a team meeting tomorrow at 10 AM," the following occurs:

[0782] 1. The user speaks into the device.

[0783] 2. The device sends the audio data to the server.

[0784] 3. The server uses a speech recognition engine to convert the speech into text data.

[0785] 4. The server uses a natural language processing engine to analyze the text data and identify tasks.

[0786] 5. The server accesses the schedule management system and adds the event to the specified time slot.

[0787] 6. The server notifies the terminal of the processing results.

[0788] 7. The device displays the notification to the user.

[0789] 8. The user confirms the notification.

[0790] Through this series of processes, the system of the present invention realizes efficient task management and business automation using voice input.

[0791] The processing flow will be explained below.

[0792] Step 1:

[0793] The user speaks into a device with a dedicated application installed, pressing the voice input button and saying something like, "Schedule a team meeting for tomorrow at 10:00 AM."

[0794] Step 2:

[0795] The device captures voice input from the user and converts it into digital audio data, which is temporarily stored on the device.

[0796] Step 3:

[0797] The device transmits the captured audio data to a server over the internet using a secure communication protocol such as HTTPS or WebSocket.

[0798] Step 4:

[0799] The server passes the received voice data to a speech recognition engine, which converts the digital voice into text data, which then translates to "Schedule a team meeting tomorrow at 10 AM."

[0800] Step 5:

[0801] The server receives the text data and passes it to a natural language processing engine for analysis, which identifies the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" from the text.

[0802] Step 6:

[0803] The server uses the identified task information to create an API request to the schedule management system, including the date and time to be booked and details of the event.

[0804] Step 7:

[0805] The server sends an API request to the schedule management system, for example, an HTTP POST request to the API endpoint of a calendar service.

[0806] Step 8:

[0807] The schedule management system processes the received API request, creates a new event at the specified date and time, and returns a success response to the server if the process is completed successfully.

[0808] Step 9:

[0809] The server receives the response from the schedule management system, confirms that the reservation was successful, and then generates a notification message as a result of the processing: "A team meeting has been booked for tomorrow at 10:00 AM."

[0810] Step 10:

[0811] The server then sends the generated notification message to the device, again using a secure communication protocol such as HTTPS or WebSocket.

[0812] Step 11:

[0813] The device displays the notification message received from the server to the user, for example, as a push notification or a pop-up message within the app.

[0814] Step 12:

[0815] The user will see a notification message on the terminal and will know that the reservation has been made successfully. Based on the notification, the user can consider the next action to take.

[0816] The above steps enable efficient task management and business automation through voice input.

[0817] Example 1

[0818] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0819] While there are a wide variety of solutions for task management and work efficiency today, efficient and intuitive methods using voice input have not yet been fully established. In particular, there is room for improvement in systems that automatically schedule tasks using voice and easily realize complex natural language processing. There is a need to significantly reduce user effort by providing a consistent automated process from voice input, task identification, registration in a schedule management system, and notification of the results.

[0820] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0821] In this invention, the server includes means for analyzing received voice data using a voice recognition engine and converting it into text data, means for analyzing the text data using a natural language processing engine and identifying the corresponding task, and means for accessing the schedule management system based on the identified task and executing the necessary processing. This makes it possible to consistently automate everything from voice input to task identification, automatic schedule addition, and notification of processing results.

[0822] "Voice input" is the act of conveying information or instructions to a system using voice.

[0823] The "terminal means" is a device that allows a user to input voice data and capture and transmit the data.

[0824] The "server means" is a device or program for analyzing received data and executing various processes.

[0825] "Audio Data" means a digital representation of the sound captured by an audio input.

[0826] "Text data" is character information that has been converted by analyzing voice data.

[0827] A "voice recognition engine" is software or hardware that analyzes voice data and converts it into text data.

[0828] A "natural language processing engine" is software or hardware that analyzes text data and understands and interprets human language.

[0829] A "task" is a specific activity or work undertaken to achieve a particular goal.

[0830] A "schedule management system" is software or hardware for organizing and managing tasks and events in chronological order.

[0831] "Notification" refers to a message or information presentation that conveys processing results or information to the user.

[0832] "Analysis" is the process of extracting meaning and information from data and understanding it.

[0833] The system of the present invention is designed to realize task management and business automation using voice input. This system includes a terminal means for capturing voice input, a server means for analyzing the voice data and converting it into text data, a server means for analyzing the text data and identifying tasks, a server means for accessing a schedule management system based on the identified tasks and executing processing, and a server means for notifying the terminal means of the processing results.

[0834] First, a user uses a device with a dedicated application installed to input voice data. This device is equipped with a microphone to capture voice data. When the user inputs voice data such as "Schedule a team meeting tomorrow at 10:00 AM," the voice data is captured by the device and sent to the server via the Internet.

[0835] The server uses a speech recognition engine (e.g., speech recognition software) to analyze the received voice data and convert it into text data, in this case, "Schedule a team meeting tomorrow at 10:00 AM."

[0836] The server then uses a natural language processing engine (e.g., a natural language processing API) to analyze the text data and identify the user's intent. In this specific process, the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" are extracted.

[0837] Based on the identified task, i.e., booking a team meeting, the server accesses the schedule management system (e.g., a calendar service) and performs an operation to add an event for the specified time slot. The server performs this operation by calling the appropriate API.

[0838] When the reservation is successfully completed, the server generates a message to notify the terminal of the result. For example, a message saying "A team meeting has been reserved for tomorrow at 10:00 AM" is sent to the terminal, which displays it to the user. The user confirms the notification and knows that the reservation was made successfully.

[0839] As a specific example, the processing flow when a user voice-inputs "Book a team meeting for tomorrow at 10:00 AM" will be described.

[0840] In addition, by inputting the following prompt sentence into the generative AI model, an explanation based on the system's processing flow can be obtained.

[0841] Take a step-by-step guide to how the system works, given the voice input "Schedule a team meeting tomorrow at 10am."

[0842] The prompt sentences provide detailed explanations of the specific operations of each processing step.

[0843] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0844] Step 1:

[0845] The user provides voice input. Using a device with a dedicated application installed, the user says, "Schedule a team meeting for tomorrow at 10:00 AM."

[0846] Input: User's voice commands

[0847] Output: Audio data (digital format)

[0848] Step 2:

[0849] The device sends the audio data to the server using a secure communication protocol (e.g., HTTPS).

[0850] Input: Audio data

[0851] Output: Request sent to server

[0852] Step 3:

[0853] The server converts the voice data into text data. The server uses a voice recognition engine to analyze the voice data and convert it into text data.

[0854] Input: Audio data

[0855] Output: Text data: "Schedule a team meeting tomorrow at 10 AM"

[0856] Step 4:

[0857] The server analyzes the text data and identifies the task. The server uses a natural language processing engine to extract the intent from the text data. In this case, the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" are extracted.

[0858] Input: Text data: "Schedule a team meeting tomorrow at 10 AM"

[0859] Output: Time information "Tomorrow at 10 AM" and task information "Schedule a team meeting"

[0860] Step 5:

[0861] The server sends a request to the schedule management system. Based on the identified task information, the server sends an API request to add an event to the schedule management system.

[0862] Input: Time information "Tomorrow at 10 AM" and task information "Schedule a team meeting"

[0863] Output: A request to add an event to the schedule management system.

[0864] Step 6:

[0865] The server generates a notification message of the processing result, creates a notification of successful reservation, and sends it to the terminal.

[0866] Input: Response from the schedule management system (if successful)

[0867] Output: Notification message: "Your team meeting has been scheduled for tomorrow at 10 AM."

[0868] Step 7:

[0869] The device receives the notification message and displays it to the user, either in the notification center within the application or as a pop-up message.

[0870] Input: Notification message: "Team meeting booked for tomorrow at 10am"

[0871] Output: Display notification

[0872] Step 8:

[0873] The user checks the notification. The user checks the notification on the terminal and realizes that the reservation has been made successfully.

[0874] Input: The displayed notification message

[0875] Output: Recognition (no specific output)

[0876] (Application example 1)

[0877] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0878] In modern factories, many tasks are performed manually, resulting in issues with efficiency and accuracy. Misunderstandings and delays are particularly common when communicating work instructions and executing tasks. Frequent on-site operations by workers also increase the workload. There is a need for a system that can improve this situation and increase efficiency by automating factory operations.

[0879] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0880] In this invention, the server includes terminal means for capturing voice input, means for analyzing voice data received from the terminal means and converting it into text data, means for analyzing the text data and identifying the corresponding task, means for accessing the work machine control system based on the identified task and executing the required processing, means for notifying the terminal means of the processing results, and terminal means for displaying the notification. This makes it possible to efficiently and accurately transmit work instructions and automate work using voice instructions.

[0881] The "terminal means for capturing voice input" refers to a device equipped to receive a user's voice, such as a microphone or a smartphone.

[0882] "Means for analysis" refers to the technology or process for converting voice data into text data and analyzing the user's intent from that text data.

[0883] "Text data" refers to character string data converted from speech input, which can then be analyzed.

[0884] "Means for identification" refers to the technology or algorithm that finds and recognizes the corresponding task from the analyzed text data.

[0885] A "work machine control system" is a system that controls various machines and robots used in factories and production lines to carry out tasks.

[0886] "Means for performing processing" refers to hardware or software that performs actual actions or operations based on a specified task.

[0887] "Means for notifying" refers to the method or device for notifying the user of the processing results or status.

[0888] "Terminal means" refers to a device operated by a user, and is used for voice input, display of notifications, etc.

[0889] A "schedule management system" is a system that users use to manage tasks and events, including calendar apps and scheduling software.

[0890] A "voice recognition engine" is software or algorithms that analyze voice data and convert it into text data.

[0891] A "natural language processing engine" is a technology or algorithm that analyzes text data and understands user intent and commands.

[0892] The system of the present invention is designed to automate work instructions and operations in a factory using voice input. In this system, a user gives instructions by voice, and the robot automatically executes the work by analyzing the contents of the instructions. Specific embodiments are described in detail below.

[0893] First, the user uses a device with a dedicated application installed to input voice. This device is equipped with a microphone to capture voice input. The user presses the voice input button and inputs voice, such as "Move part B to the next line."

[0894] Next, the voice data is captured by the device and sent to a server via the Internet. The server uses a voice recognition engine to analyze the received voice data and convert it into text data. In this case, it is converted into text data such as "Move part B to the next line."

[0895] Next, the server uses a natural language processing engine to analyze the text data and identify the user's intent. In this case, the server extracts the movement instruction "to the next line" and the specific target "Part B."

[0896] Based on the identified task (i.e., moving a part using a factory robot), the server accesses the work machine control system and performs operations to execute the specified task. The server calls the APIs and protocols for robot control to execute the movement instructions.

[0897] When the operation is completed successfully, the server generates a message to notify the terminal of the result. For example, a message saying "Part B has moved to the next line" is generated. The terminal displays the received notification message to the user, who can confirm that the operation was performed correctly.

[0898] As a concrete example, if the user speaks "Move part B to the next line," the following prompt sentence is generated:

[0899] "Move part B to the next line."

[0900] This prompt allows the system to understand the user's instructions and the factory robot to automatically carry out the specified work. Through this series of processes, the system of the present invention realizes efficient and accurate transmission of work instructions and automation of work using voice input.

[0901] The hardware used includes microphones and smartphones to capture voice input and factory robots to perform the tasks, while the software used includes a speech recognition engine, a natural language processing engine, and an API for controlling the robots.

[0902] As described above, the system of the present invention efficiently transmits instructions for factory work through voice input, thereby realizing automation of work.

[0903] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0904] Step 1:

[0905] The user provides voice input.

[0906] Input: User speech (e.g., "Move part B to the next line").

[0907] Output: Audio data.

[0908] Specific operation: The user presses the voice input button and speaks, and the device's microphone captures the voice.

[0909] Step 2:

[0910] The terminal transmits the voice data to the server.

[0911] Input: Audio data.

[0912] Output: The audio data sent to the server.

[0913] Specific operation: The device sends the captured audio data to a server via the Internet.

[0914] Step 3:

[0915] The server converts the voice data into text data using a voice recognition engine.

[0916] Input: Audio data.

[0917] Output: Text data (e.g. "Move part B to the next line").

[0918] Specific operation: The server passes the received voice data to a voice recognition engine and converts the data into text format.

[0919] Step 4:

[0920] The server uses a natural language processing engine to analyze the text data and identify the user's intent.

[0921] Input: Text data.

[0922] Output: Data specifying the intent (e.g., "Task to move part B to the next line").

[0923] Specific operation: The server passes the text data to a natural language processing engine for analysis to identify the user's intent and task.

[0924] Step 5:

[0925] The server accesses the work machine control system and performs the necessary processing.

[0926] Input: Identified task data.

[0927] Output: Robot control command (e.g. "Move part B from line A to line B").

[0928] Specific operation: Based on the identified task, the server calls the API of the work machine control system to give instructions to the robot.

[0929] Step 6:

[0930] The processing result is notified to the terminal.

[0931] Input: Processing result (e.g., work completion notification).

[0932] Output: Notification message to the terminal (e.g. "Part B has moved to the next line").

[0933] Specific operation: The server receives the processing result from the work machine control system, generates a notification message, and sends it to the terminal.

[0934] Step 7:

[0935] The terminal displays the notification to the user.

[0936] Input: Notification message.

[0937] Output: A user-visible notification (e.g., "Part B has moved to the next line").

[0938] Specific operation: The device receives a notification message from the server and notifies the user by displaying the message on the screen or by voice notification.

[0939] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0940] The system of the present invention is equipped with a function for recognizing a user's emotions in addition to task management and business automation using voice input. MODES FOR CARRYING OUT THE INVENTION The following describes in detail embodiments of the present invention.

[0941] First, the user speaks into a device with a dedicated application installed. The device is equipped with a microphone for capturing voice input, and while pressing the voice input button, the user speaks, "Schedule a team meeting tomorrow at 10:00 AM."

[0942] The audio data is then captured by the terminal and transmitted over the Internet to a server.

[0943] The server uses a speech recognition engine to analyze the received voice data and convert it into text data, which in this case is "Schedule a team meeting tomorrow at 10 AM."

[0944] The server then passes the text and voice data to an emotion engine to identify the user's emotion. The emotion engine determines whether the user is stressed or relaxed based on the tone of the voice and the content of the text. For example, if the user's voice sounds urgent and tense, the emotion engine will recognize the user as being in a "tense" state.

[0945] The server then uses a natural language processing engine to analyze the text data and identify the user's intent, extracting the time information "tomorrow at 10:00 AM" and the task information "book a team meeting."

[0946] Based on the identified task (i.e., booking a team meeting), the server makes an API request to the scheduling system, including the date and time to book and details of the event.

[0947] The server sends an API request to the schedule management system and performs an operation to add an event to the specified time slot. For example, it calls the API of the schedule management system to execute a reservation.

[0948] Once the reservation is successfully completed, the server generates a message to notify the device of the result. This message is customized based on the user's emotions. For example, if the user is nervous, the server generates a message saying, "Your team meeting has been booked for tomorrow at 10:00 AM. Relax and move on to the next task!"

[0949] The terminal displays the received notification message to the user, and the user confirms that the reservation has been made correctly. The content of the notification message is designed to help the user relax.

[0950] For example, if a user says "Schedule a team meeting tomorrow at 10 AM" and the emotion engine recognizes the user's nervousness, the following happens:

[0951] 1. The user speaks into the device.

[0952] 2. The device sends the audio data to the server.

[0953] 3. The server uses a speech recognition engine to convert the speech into text data.

[0954] 4. The server uses the emotion engine to recognize the user's emotions.

[0955] 5. The server uses a natural language processing engine to analyze the text data and identify tasks.

[0956] 6. The server accesses the schedule management system and adds the event to the specified time slot.

[0957] 7. The server generates a notification message of the processing result that takes into consideration the user's feelings.

[0958] 8. The server notifies the terminal of the processing results.

[0959] 9. The device displays the notification to the user.

[0960] 10. The user acknowledges the notification, relaxes, and moves on to the next task.

[0961] As a result, the system of the present invention can provide a system that not only enables efficient task management and business automation through voice input, but also takes into consideration the psychological state of the user.

[0962] The processing flow will be explained below.

[0963] Step 1:

[0964] Users speak into a device with a dedicated application installed to perform a specific task, for example, by saying, "Schedule a team meeting tomorrow at 10 AM."

[0965] Step 2:

[0966] The device captures the user's voice input through a microphone and converts it into digital audio data, which is temporarily stored on the device.

[0967] Step 3:

[0968] The device transmits the captured audio data to a server over the Internet using a secure communication protocol (e.g., HTTPS).

[0969] Step 4:

[0970] The server sends the received voice data to a speech recognition engine, which converts the voice data into text data, such as "Schedule a team meeting tomorrow at 10:00 AM."

[0971] Step 5:

[0972] The server sends the converted text and voice data to the emotion engine, which analyzes the user's emotions. The emotion engine identifies the user's emotional state from the tone of the voice and the content of the text. For example, if the voice tone sounds urgent, it is determined to be "nervous."

[0973] Step 6:

[0974] The server sends the parsed text data to a natural language processing (NLP) engine to identify the user's intent, which extracts the time information "tomorrow at 10 AM" and the task information "book a team meeting."

[0975] Step 7:

[0976] The server creates an API request to access the schedule management system based on the extracted task information. This API request includes task details (date and time, event content, etc.).

[0977] Step 8:

[0978] The server sends the created API request to the schedule management system. Specifically, it sends an HTTP POST request to the API endpoint of the schedule management system.

[0979] Step 9:

[0980] The schedule management system processes the received API request, creates a new event at the specified date and time, and returns a success response to the server when the processing is complete.

[0981] Step 10:

[0982] The server receives a response from the scheduling management system, confirms that the reservation was successful, and then generates a customized notification message based on the user's sentiment. For example, it could create a message like, "Your team meeting has been booked for tomorrow at 10 AM. Relax and move on to your next task!"

[0983] Step 11:

[0984] The server then sends the generated notification message to the terminal, again using a secure communication protocol (e.g., HTTPS).

[0985] Step 12:

[0986] The device displays the notification message received from the server to the user, for example, as a push notification or a pop-up message within the app.

[0987] Step 13:

[0988] The user will see a notification message on their device and know that the reservation has been made successfully, allowing them to move on to the next task with peace of mind.

[0989] Through the above steps, the system of the present invention can provide a system that not only enables efficient task management and business automation through voice input, but also takes into consideration the psychological state of the user.

[0990] Example 2

[0991] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0992] Conventional task management systems simply convert users' voice input into text data and identify tasks, but lack the ability to recognize users' emotions. Furthermore, they lack consideration for users, as they are unable to take their emotional state into account when managing tasks or automating tasks. Furthermore, notification messages that do not take users' emotions into account can potentially impair the user experience.

[0993] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing voice data and converting it into text data, a means for analyzing the text data and voice data and recognizing the user's emotions, and a means for analyzing the text data and identifying the corresponding task. This enables task management and business automation that take the user's emotions into consideration.

[0994] "Voice input" refers to data in a format in which the user inputs spoken words into the terminal using a microphone.

[0995] "Terminal means" refers to a device or apparatus for capturing voice input and performing the necessary processing.

[0996] "Server means" refers to a computer system that performs analysis and data processing, and manages and provides necessary information.

[0997] "Voice data" refers to digital data generated from the voice input by a user into a terminal.

[0998] "Text data" refers to data that has been analyzed and converted into textual information from audio data.

[0999] "Emotion recognition" refers to the process of analyzing voice and text data to determine a user's emotions and mental state.

[1000] "Task identification" refers to the process of analyzing text data to clarify the specific actions and instructions intended by the user.

[1001] A "schedule management system" refers to software or services for managing schedules and tasks.

[1002] "Notification" refers to messages or alerts that inform users of processing results or information.

[1003] A "natural language processing engine" refers to algorithms and technologies for analyzing text data and understanding and processing human language.

[1004] "Speech recognition engine" refers to the algorithms and technologies used to analyze voice data and convert it into corresponding text data.

[1005] The system of the present invention is equipped with a function for recognizing a user's emotions in addition to task management and business automation using voice input. MODES FOR CARRYING OUT THE INVENTION The following describes in detail embodiments of the present invention.

[1006] The user speaks into a device with a dedicated application installed. The device is equipped with a microphone to capture voice input, and while pressing the voice input button, the user speaks, "Schedule a team meeting tomorrow at 10:00 AM."

[1007] The device captures audio using a built-in microphone, temporarily stores the audio data, and then transmits it to a server via the internet. The hardware used here could be a smartphone or tablet.

[1008] The server analyzes the received voice data using a voice recognition engine (for example, a general voice recognition engine) and converts the voice data into text data. For example, text data such as "Please schedule a team meeting tomorrow at 10:00 AM" is generated.

[1009] The server then passes the converted text and voice data to an emotion engine (e.g., a general emotion analysis engine) to analyze the user's emotions. This determines whether the user is stressed or relaxed. For example, if the user's voice sounds rushed and tense, the emotion engine will recognize the user as being in a "tense" state.

[1010] Next, the server uses a natural language processing engine (e.g., a general natural language processing model) to analyze the text data in detail and identify the user's intent. In this case, the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" are extracted.

[1011] Based on the identified task (i.e., booking a team meeting), the server makes an API request to access the scheduling system (e.g., a common calendar API). This request includes the date and time to book and details of the event. The server sends this request to the scheduling system, adding the event to the specified time slot.

[1012] Once the reservation is successfully completed, the server generates a message to notify the device of the result. This message is customized based on the user's emotions. For example, if the user is nervous, the server generates a message saying, "Your team meeting has been booked for tomorrow at 10:00 AM. Relax and move on to the next task!"

[1013] The terminal displays the notification message received from the server, letting the user know that the reservation has been made successfully. The content of the notification message is designed to help the user relax.

[1014] For example, if a user says "Schedule a team meeting tomorrow at 10 AM" and the emotion engine recognizes the user's nervousness, the following happens:

[1015] 1. The user speaks into the device.

[1016] 2. The device sends the audio data to the server.

[1017] 3. The server uses a speech recognition engine to convert the speech into text data.

[1018] 4. The server uses the emotion engine to recognize the user's emotions.

[1019] 5. The server uses a natural language processing engine to analyze the text data and identify tasks.

[1020] 6. The server accesses the schedule management system and adds the event to the specified time slot.

[1021] 7. The server generates a notification message of the processing result that takes into consideration the user's feelings.

[1022] 8. The server notifies the terminal of the processing results.

[1023] 9. The device displays the notification to the user.

[1024] 10. The user acknowledges the notification, relaxes, and moves on to the next task.

[1025] This system allows users to efficiently manage tasks through voice input and automate work processes while taking into consideration their psychological state.

[1026] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1027] Step 1:

[1028] The user provides voice input.

[1029] The user speaks into a device with a dedicated application installed, presses the voice input button, and says, "Schedule a team meeting for tomorrow at 10 a.m." This voice becomes the input data.

[1030] Step 2:

[1031] The device captures and transmits the audio data.

[1032] The device captures the user's voice using a built-in microphone, temporarily stores it as digital audio data, and then transmits the audio data to a server via the Internet. The output is the digitized audio data sent to the server.

[1033] Step 3:

[1034] The server converts the voice data into text data.

[1035] The server analyzes the received voice data using a speech recognition engine and converts it into text data. The collected voice data is converted into text information, and the text output is, for example, "Schedule a team meeting tomorrow at 10:00 AM."

[1036] Step 4:

[1037] The server recognizes the user's emotions from the text data and voice data.

[1038] The server then passes the converted text and voice data to an emotion recognition engine, which analyzes the user's emotions based on the tone and speed of the voice and the content of the text. For example, if the voice is spoken quickly and with a high pitch, it is determined to be "tension," and this information is output.

[1039] Step 5:

[1040] The server analyzes the text data and identifies the user's intent.

[1041] The server uses a natural language processing engine to perform detailed analysis of the text data and extract task and time information. For example, the analysis identifies instructions such as "tomorrow at 10:00 AM" and "book a team meeting." The output is specific task instructions.

[1042] Step 6:

[1043] The server accesses the schedule management system and reserves the event.

[1044] The server creates a request to the schedule management system's API based on the extracted task information. This request includes the reservation date and time and event details. The server sends this request to the schedule management system, which adds the event to the specified time slot. The output is the reservation execution result.

[1045] Step 7:

[1046] The server generates a reservation completion notification and sends it to the terminal.

[1047] The server confirms that the reservation has been successfully completed and generates a customized message based on the user's sentiment, for example, "Your team meeting has been booked for tomorrow at 10 AM. Relax and move on to the next task!", and sends this to the terminal. As an output, a customized notification message is generated.

[1048] Step 8:

[1049] The device displays a notification message.

[1050] The device displays the notification message received from the server as a pop-up notification or an in-app message, and the user can confirm that the reservation has been successfully made.

[1051] This means:

[1052] This enables efficient task management using voice input and business automation that takes into account the user's psychological state.

[1053] (Application example 2)

[1054] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1055] Conventional voice input systems simply convert voice data into text data and execute tasks based on that text. However, because they do not take into account the user's emotional state, they have the problem of not contributing to stress reduction or motivation improvement. In the food delivery field in particular, it is an issue that they are unable to provide appropriate responses when the user is in a hurry or feeling stressed.

[1056] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1057] In this invention, the server includes means for identifying the user's emotion from the voice data and text data, means for generating customized feedback according to the identified emotion, and means for notifying the terminal means of the processing result and the emotion-considered feedback, thereby making it possible to provide a response customized according to the user's emotional state.

[1058] "Voice input" refers to the act of generating voice data by the user speaking.

[1059] The "terminal means" is a device that allows a user to input voice, and includes a microphone.

[1060] The "server means" is a computer system that analyzes and processes data received from the terminal means via a network.

[1061] "Voice data" refers to the user's voice signals captured by the terminal means.

[1062] "Text data" is a collection of character information obtained by analyzing voice data.

[1063] A "task" refers to a specific action or work that a user requests by voice input.

[1064] A "schedule management system" is a software application for completing tasks and coordinating schedules.

[1065] "Emotion identification" is the process of analyzing audio and text data to determine the user's psychological state.

[1066] "Customized feedback" refers to response messages and guidance that are individually tailored to the user's emotional state.

[1067] The system of the present invention utilizes voice input for task management and business automation, and further recognizes the user's emotions to provide customized feedback. Detailed embodiments for implementing the present invention will be described below.

[1068] Hardware and software used

[1069] Hardware

[1070] Terminal means: A smartphone is used as a device for capturing voice input. This smartphone has a built-in microphone.

[1071] Server means: A computer system is used to analyze data and perform necessary processing.

[1072] software

[1073] Speech recognition engine: Uses the speech_recognition library to convert voice data into text data.

[1074] Natural Language Processing Engine: Uses the TextBlob library to analyze text data and identify tasks and sentiments.

[1075] HTTP Request Library: The requests library is used to communicate between the server means and the schedule management system and the feedback processing system.

[1076] System processing procedure

[1077] 1. Capture voice input

[1078] The user speaks into the microphone of the smartphone. Voice input is the act of generating voice data by speaking. For example, the user may say, "I'd like one pizza, please."

[1079] 2. Converting voice data to text

[1080] The voice data captured by the terminal means is transmitted to the server means, which converts the voice data into text data using a speech_recognition library.

[1081] 3. Emotion Identification

[1082] The server identifies the user's emotion from the converted text data and voice data. It analyzes the text data using the TextBlob library to determine the user's emotional state. For example, if the text is "One pizza please," it determines whether the user is "relaxed" or "in a hurry" based on the content and tone of the voice.

[1083] 4. Task execution

[1084] The server means sends an order to the food delivery API using the requests library based on the identified task, with the text data and emotion data included in the payload.

[1085] 5. Feedback Generation and Notification

[1086] The server generates customized feedback according to the user's emotional state and notifies the terminal, thereby allowing the user to move on to the next task without feeling stressed. For example, if the user is relaxed, a message such as "Your order has been completed. Please wait a moment" is displayed.

[1087] Specific examples

[1088] For example, if a user says "I'd like one pizza, please," the system converts the order into text, recognizes the user's emotion as "relaxed," sends the order to a food delivery API, and then displays feedback such as "Your order has been completed. Please wait a moment."

[1089] Prompt Sentence Examples

[1090] "Voice input: 'One pizza please'

[1091] Emotion Recognition: Relax

[1092] Action: User is relaxed, so show 'Your order is complete. Please wait a moment.'

[1093] In this way, the system of the present invention realizes efficient task management and business automation through voice input, and also provides services that take into consideration the psychological state of the user.

[1094] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1095] Step 1:

[1096] Capturing voice input

[1097] The user inputs voice by speaking into the terminal means. For example, the user may say, "I'd like one pizza, please." The terminal means uses a microphone to capture this voice data.

[1098] Input: User's voice

[1099] Output: Captured audio data

[1100] Step 2:

[1101] Converting audio data to text

[1102] The voice data captured by the terminal means is transmitted to the server means, which analyzes the voice data using the speech_recognition library and converts it into text data.

[1103] Input: Captured audio data

[1104] Output: Text data

[1105] Step 3:

[1106] Emotion Identification

[1107] The server identifies the user's emotion from the text data and voice data. It analyzes the text data using the TextBlob library to determine the user's emotional state. For example, if the text is "One pizza please" and the voice tone is calm, it determines that the user is "relaxed."

[1108] Input: Text data and tone information of audio data

[1109] Output: User's emotional state (e.g., relaxed, rushed, stressed)

[1110] Step 4:

[1111] Task execution

[1112] The server sends an order to the food delivery API based on the identified task, with the text data and emotion data included in the payload. It then uses the requests library to create an HTTP request and process the order.

[1113] Input: Text data, user's emotional state

[1114] Output: API request response (order confirmation)

[1115] Step 5:

[1116] Feedback Generation and Notification

[1117] The server means generates a feedback message customized according to the user's emotional state and notifies the terminal means of the message. For example, if the user is relaxed, the server means generates a message saying, "Your order is complete. Please wait a moment."

[1118] Input: API request response, user emotional state

[1119] Output: Customized feedback message

[1120] Step 6:

[1121] View Feedback

[1122] The terminal means displays the feedback message received from the server means to the user, thereby confirming that the order has been processed correctly.

[1123] Input: Customized feedback message

[1124] Output: Display of feedback message

[1125] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1126] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1127] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1128] [Fourth embodiment]

[1129] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1130] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1131] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1132] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1133] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1134] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1135] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1136] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1137] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1138] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1139] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1140] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1141] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1142] The system of the present invention is designed to realize task management and business automation using voice input. The following describes in detail the embodiments of the present invention.

[1143] First, a user uses a device with a dedicated application installed to input voice. This device is equipped with a microphone to capture voice input. The user presses the voice input button and inputs voice, such as "Schedule a team meeting tomorrow at 10:00 AM."

[1144] The audio data is then captured by the terminal and transmitted over the Internet to a server.

[1145] The server uses a speech recognition engine to analyze the received voice data and convert it into text data, in this case "Schedule a team meeting tomorrow at 10 AM."

[1146] The server then uses a natural language processing engine to analyze the text data and identify the user's intent, extracting the time information "tomorrow at 10:00 AM" and the task information "book a team meeting."

[1147] Based on the identified task (i.e., booking a team meeting), the server accesses a schedule management system (e.g., a calendar service) and performs an operation to add an event for the specified time slot. The server calls the appropriate API to complete the booking.

[1148] If the reservation is successful, the server generates a message to notify the terminal of the result, for example, a message saying "A team meeting has been reserved for tomorrow at 10:00 AM."

[1149] The terminal displays the received notification message to the user, and the user confirms that the reservation has been made correctly.

[1150] As a concrete example, if a user says "Schedule a team meeting tomorrow at 10 AM," the following occurs:

[1151] 1. The user speaks into the device.

[1152] 2. The device sends the audio data to the server.

[1153] 3. The server uses a speech recognition engine to convert the speech into text data.

[1154] 4. The server uses a natural language processing engine to analyze the text data and identify tasks.

[1155] 5. The server accesses the schedule management system and adds the event to the specified time slot.

[1156] 6. The server notifies the terminal of the processing results.

[1157] 7. The device displays the notification to the user.

[1158] 8. The user confirms the notification.

[1159] Through this series of processes, the system of the present invention realizes efficient task management and business automation using voice input.

[1160] The processing flow will be explained below.

[1161] Step 1:

[1162] The user speaks into a device with a dedicated application installed, pressing the voice input button and saying something like, "Schedule a team meeting for tomorrow at 10:00 AM."

[1163] Step 2:

[1164] The device captures voice input from the user and converts it into digital audio data, which is temporarily stored on the device.

[1165] Step 3:

[1166] The device transmits the captured audio data to a server over the internet using a secure communication protocol such as HTTPS or WebSocket.

[1167] Step 4:

[1168] The server passes the received voice data to a speech recognition engine, which converts the digital voice into text data, which then translates to "Schedule a team meeting tomorrow at 10 AM."

[1169] Step 5:

[1170] The server receives the text data and passes it to a natural language processing engine for analysis, which identifies the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" from the text.

[1171] Step 6:

[1172] The server uses the identified task information to create an API request to the schedule management system, including the date and time to be booked and details of the event.

[1173] Step 7:

[1174] The server sends an API request to the schedule management system, for example, an HTTP POST request to the API endpoint of a calendar service.

[1175] Step 8:

[1176] The schedule management system processes the received API request, creates a new event at the specified date and time, and returns a success response to the server if the process is completed successfully.

[1177] Step 9:

[1178] The server receives the response from the schedule management system, confirms that the reservation was successful, and then generates a notification message as a result of the processing: "A team meeting has been booked for tomorrow at 10:00 AM."

[1179] Step 10:

[1180] The server then sends the generated notification message to the device, again using a secure communication protocol such as HTTPS or WebSocket.

[1181] Step 11:

[1182] The device displays the notification message received from the server to the user, for example, as a push notification or a pop-up message within the app.

[1183] Step 12:

[1184] The user will see a notification message on the terminal and will know that the reservation has been made successfully. Based on the notification, the user can consider the next action to take.

[1185] The above steps enable efficient task management and business automation through voice input.

[1186] Example 1

[1187] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1188] While there are a wide variety of solutions for task management and work efficiency today, efficient and intuitive methods using voice input have not yet been fully established. In particular, there is room for improvement in systems that automatically schedule tasks using voice and easily realize complex natural language processing. There is a need to significantly reduce user effort by providing a consistent automated process from voice input, task identification, registration in a schedule management system, and notification of the results.

[1189] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1190] In this invention, the server includes means for analyzing received voice data using a voice recognition engine and converting it into text data, means for analyzing the text data using a natural language processing engine and identifying the corresponding task, and means for accessing the schedule management system based on the identified task and executing the necessary processing. This makes it possible to consistently automate everything from voice input to task identification, automatic schedule addition, and notification of processing results.

[1191] "Voice input" is the act of conveying information or instructions to a system using voice.

[1192] The "terminal means" is a device that allows a user to input voice data and capture and transmit the data.

[1193] The "server means" is a device or program for analyzing received data and executing various processes.

[1194] "Audio Data" means a digital representation of the sound captured by an audio input.

[1195] "Text data" is character information that has been converted by analyzing voice data.

[1196] A "voice recognition engine" is software or hardware that analyzes voice data and converts it into text data.

[1197] A "natural language processing engine" is software or hardware that analyzes text data and understands and interprets human language.

[1198] A "task" is a specific activity or work undertaken to achieve a particular goal.

[1199] A "schedule management system" is software or hardware for organizing and managing tasks and events in chronological order.

[1200] "Notification" refers to a message or information presentation that conveys processing results or information to the user.

[1201] "Analysis" is the process of extracting meaning and information from data and understanding it.

[1202] The system of the present invention is designed to realize task management and business automation using voice input. This system includes a terminal means for capturing voice input, a server means for analyzing the voice data and converting it into text data, a server means for analyzing the text data and identifying tasks, a server means for accessing a schedule management system based on the identified tasks and executing processing, and a server means for notifying the terminal means of the processing results.

[1203] First, a user uses a device with a dedicated application installed to input voice data. This device is equipped with a microphone to capture voice data. When the user inputs voice data such as "Schedule a team meeting tomorrow at 10:00 AM," the voice data is captured by the device and sent to the server via the Internet.

[1204] The server uses a speech recognition engine (e.g., speech recognition software) to analyze the received voice data and convert it into text data, in this case, "Schedule a team meeting tomorrow at 10:00 AM."

[1205] The server then uses a natural language processing engine (e.g., a natural language processing API) to analyze the text data and identify the user's intent. In this specific process, the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" are extracted.

[1206] Based on the identified task, i.e., booking a team meeting, the server accesses the schedule management system (e.g., a calendar service) and performs an operation to add an event for the specified time slot. The server performs this operation by calling the appropriate API.

[1207] When the reservation is successfully completed, the server generates a message to notify the terminal of the result. For example, a message saying "A team meeting has been reserved for tomorrow at 10:00 AM" is sent to the terminal, which displays it to the user. The user confirms the notification and knows that the reservation was made successfully.

[1208] As a specific example, the processing flow when a user voice-inputs "Book a team meeting for tomorrow at 10:00 AM" will be described.

[1209] In addition, by inputting the following prompt sentence into the generative AI model, an explanation based on the system's processing flow can be obtained.

[1210] Take a step-by-step guide to how the system works, given the voice input "Schedule a team meeting tomorrow at 10am."

[1211] The prompt sentences provide detailed explanations of the specific operations of each processing step.

[1212] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1213] Step 1:

[1214] The user provides voice input. Using a device with a dedicated application installed, the user says, "Schedule a team meeting for tomorrow at 10:00 AM."

[1215] Input: User's voice commands

[1216] Output: Audio data (digital format)

[1217] Step 2:

[1218] The device sends the audio data to the server using a secure communication protocol (e.g., HTTPS).

[1219] Input: Audio data

[1220] Output: Request sent to server

[1221] Step 3:

[1222] The server converts the voice data into text data. The server uses a voice recognition engine to analyze the voice data and convert it into text data.

[1223] Input: Audio data

[1224] Output: Text data: "Schedule a team meeting tomorrow at 10 AM"

[1225] Step 4:

[1226] The server analyzes the text data and identifies the task. The server uses a natural language processing engine to extract the intent from the text data. In this case, the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" are extracted.

[1227] Input: Text data: "Schedule a team meeting tomorrow at 10 AM"

[1228] Output: Time information "Tomorrow at 10 AM" and task information "Schedule a team meeting"

[1229] Step 5:

[1230] The server sends a request to the schedule management system. Based on the identified task information, the server sends an API request to add an event to the schedule management system.

[1231] Input: Time information "Tomorrow at 10 AM" and task information "Schedule a team meeting"

[1232] Output: A request to add an event to the schedule management system.

[1233] Step 6:

[1234] The server generates a notification message of the processing result, creates a notification of successful reservation, and sends it to the terminal.

[1235] Input: Response from the schedule management system (if successful)

[1236] Output: Notification message: "Your team meeting has been scheduled for tomorrow at 10 AM."

[1237] Step 7:

[1238] The device receives the notification message and displays it to the user, either in the notification center within the application or as a pop-up message.

[1239] Input: Notification message: "Team meeting booked for tomorrow at 10am"

[1240] Output: Display notification

[1241] Step 8:

[1242] The user checks the notification. The user checks the notification on the terminal and realizes that the reservation has been made successfully.

[1243] Input: The displayed notification message

[1244] Output: Recognition (no specific output)

[1245] (Application example 1)

[1246] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1247] In modern factories, many tasks are performed manually, resulting in issues with efficiency and accuracy. Misunderstandings and delays are particularly common when communicating work instructions and executing tasks. Frequent on-site operations by workers also increase the workload. There is a need for a system that can improve this situation and increase efficiency by automating factory operations.

[1248] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1249] In this invention, the server includes terminal means for capturing voice input, means for analyzing voice data received from the terminal means and converting it into text data, means for analyzing the text data and identifying the corresponding task, means for accessing the work machine control system based on the identified task and executing the required processing, means for notifying the terminal means of the processing results, and terminal means for displaying the notification. This makes it possible to efficiently and accurately transmit work instructions and automate work using voice instructions.

[1250] The "terminal means for capturing voice input" refers to a device equipped to receive a user's voice, such as a microphone or a smartphone.

[1251] "Means for analysis" refers to the technology or process for converting voice data into text data and analyzing the user's intent from that text data.

[1252] "Text data" refers to character string data converted from speech input, which can then be analyzed.

[1253] "Means for identification" refers to the technology or algorithm that finds and recognizes the corresponding task from the analyzed text data.

[1254] A "work machine control system" is a system that controls various machines and robots used in factories and production lines to carry out tasks.

[1255] "Means for performing processing" refers to hardware or software that performs actual actions or operations based on a specified task.

[1256] "Means for notifying" refers to the method or device for notifying the user of the processing results or status.

[1257] "Terminal means" refers to a device operated by a user, and is used for voice input, display of notifications, etc.

[1258] A "schedule management system" is a system that users use to manage tasks and events, including calendar apps and scheduling software.

[1259] A "voice recognition engine" is software or algorithms that analyze voice data and convert it into text data.

[1260] A "natural language processing engine" is a technology or algorithm that analyzes text data and understands user intent and commands.

[1261] The system of the present invention is designed to automate work instructions and operations in a factory using voice input. In this system, a user gives instructions by voice, and the robot automatically executes the work by analyzing the contents of the instructions. Specific embodiments are described in detail below.

[1262] First, the user uses a device with a dedicated application installed to input voice. This device is equipped with a microphone to capture voice input. The user presses the voice input button and inputs voice, such as "Move part B to the next line."

[1263] Next, the voice data is captured by the device and sent to a server via the Internet. The server uses a voice recognition engine to analyze the received voice data and convert it into text data. In this case, it is converted into text data such as "Move part B to the next line."

[1264] Next, the server uses a natural language processing engine to analyze the text data and identify the user's intent. In this case, the server extracts the movement instruction "to the next line" and the specific target "Part B."

[1265] Based on the identified task (i.e., moving a part using a factory robot), the server accesses the work machine control system and performs operations to execute the specified task. The server calls the APIs and protocols for robot control to execute the movement instructions.

[1266] When the operation is completed successfully, the server generates a message to notify the terminal of the result. For example, a message saying "Part B has moved to the next line" is generated. The terminal displays the received notification message to the user, who can confirm that the operation was performed correctly.

[1267] As a concrete example, if the user speaks "Move part B to the next line," the following prompt sentence is generated:

[1268] "Move part B to the next line."

[1269] This prompt allows the system to understand the user's instructions and the factory robot to automatically carry out the specified work. Through this series of processes, the system of the present invention realizes efficient and accurate transmission of work instructions and automation of work using voice input.

[1270] The hardware used includes microphones and smartphones to capture voice input and factory robots to perform the tasks, while the software used includes a speech recognition engine, a natural language processing engine, and an API for controlling the robots.

[1271] As described above, the system of the present invention efficiently transmits instructions for factory work through voice input, thereby realizing automation of work.

[1272] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1273] Step 1:

[1274] The user provides voice input.

[1275] Input: User speech (e.g., "Move part B to the next line").

[1276] Output: Audio data.

[1277] Specific operation: The user presses the voice input button and speaks, and the device's microphone captures the voice.

[1278] Step 2:

[1279] The terminal transmits the voice data to the server.

[1280] Input: Audio data.

[1281] Output: The audio data sent to the server.

[1282] Specific operation: The device sends the captured audio data to a server via the Internet.

[1283] Step 3:

[1284] The server converts the voice data into text data using a voice recognition engine.

[1285] Input: Audio data.

[1286] Output: Text data (e.g. "Move part B to the next line").

[1287] Specific operation: The server passes the received voice data to a voice recognition engine and converts the data into text format.

[1288] Step 4:

[1289] The server uses a natural language processing engine to analyze the text data and identify the user's intent.

[1290] Input: Text data.

[1291] Output: Data specifying the intent (e.g., "Task to move part B to the next line").

[1292] Specific operation: The server passes the text data to a natural language processing engine for analysis to identify the user's intent and task.

[1293] Step 5:

[1294] The server accesses the work machine control system and performs the necessary processing.

[1295] Input: Identified task data.

[1296] Output: Robot control command (e.g. "Move part B from line A to line B").

[1297] Specific operation: Based on the identified task, the server calls the API of the work machine control system to give instructions to the robot.

[1298] Step 6:

[1299] The processing result is notified to the terminal.

[1300] Input: Processing result (e.g., work completion notification).

[1301] Output: Notification message to the terminal (e.g. "Part B has moved to the next line").

[1302] Specific operation: The server receives the processing result from the work machine control system, generates a notification message, and sends it to the terminal.

[1303] Step 7:

[1304] The terminal displays the notification to the user.

[1305] Input: Notification message.

[1306] Output: A user-visible notification (e.g., "Part B has moved to the next line").

[1307] Specific operation: The device receives a notification message from the server and notifies the user by displaying the message on the screen or by voice notification.

[1308] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1309] The system of the present invention is equipped with a function for recognizing a user's emotions in addition to task management and business automation using voice input. MODES FOR CARRYING OUT THE INVENTION The following describes in detail embodiments of the present invention.

[1310] First, the user speaks into a device with a dedicated application installed. The device is equipped with a microphone for capturing voice input, and while pressing the voice input button, the user speaks, "Schedule a team meeting tomorrow at 10:00 AM."

[1311] The audio data is then captured by the terminal and transmitted over the Internet to a server.

[1312] The server uses a speech recognition engine to analyze the received voice data and convert it into text data, which in this case is "Schedule a team meeting tomorrow at 10 AM."

[1313] The server then passes the text and voice data to an emotion engine to identify the user's emotion. The emotion engine determines whether the user is stressed or relaxed based on the tone of the voice and the content of the text. For example, if the user's voice sounds urgent and tense, the emotion engine will recognize the user as being in a "tense" state.

[1314] The server then uses a natural language processing engine to analyze the text data and identify the user's intent, extracting the time information "tomorrow at 10:00 AM" and the task information "book a team meeting."

[1315] Based on the identified task (i.e., booking a team meeting), the server makes an API request to the scheduling system, including the date and time to book and details of the event.

[1316] The server sends an API request to the schedule management system and performs an operation to add an event to the specified time slot. For example, it calls the API of the schedule management system to execute a reservation.

[1317] Once the reservation is successfully completed, the server generates a message to notify the device of the result. This message is customized based on the user's emotions. For example, if the user is nervous, the server generates a message saying, "Your team meeting has been booked for tomorrow at 10:00 AM. Relax and move on to the next task!"

[1318] The terminal displays the received notification message to the user, and the user confirms that the reservation has been made correctly. The content of the notification message is designed to help the user relax.

[1319] For example, if a user says "Schedule a team meeting tomorrow at 10 AM" and the emotion engine recognizes the user's nervousness, the following happens:

[1320] 1. The user speaks into the device.

[1321] 2. The device sends the audio data to the server.

[1322] 3. The server uses a speech recognition engine to convert the speech into text data.

[1323] 4. The server uses the emotion engine to recognize the user's emotions.

[1324] 5. The server uses a natural language processing engine to analyze the text data and identify tasks.

[1325] 6. The server accesses the schedule management system and adds the event to the specified time slot.

[1326] 7. The server generates a notification message of the processing result that takes into consideration the user's feelings.

[1327] 8. The server notifies the terminal of the processing results.

[1328] 9. The device displays the notification to the user.

[1329] 10. The user acknowledges the notification, relaxes, and moves on to the next task.

[1330] As a result, the system of the present invention can provide a system that not only enables efficient task management and business automation through voice input, but also takes into consideration the psychological state of the user.

[1331] The processing flow will be explained below.

[1332] Step 1:

[1333] Users speak into a device with a dedicated application installed to perform a specific task, for example, by saying, "Schedule a team meeting tomorrow at 10 AM."

[1334] Step 2:

[1335] The device captures the user's voice input through a microphone and converts it into digital audio data, which is temporarily stored on the device.

[1336] Step 3:

[1337] The device transmits the captured audio data to a server over the Internet using a secure communication protocol (e.g., HTTPS).

[1338] Step 4:

[1339] The server sends the received voice data to a speech recognition engine, which converts the voice data into text data, such as "Schedule a team meeting tomorrow at 10:00 AM."

[1340] Step 5:

[1341] The server sends the converted text and voice data to the emotion engine, which analyzes the user's emotions. The emotion engine identifies the user's emotional state from the tone of the voice and the content of the text. For example, if the voice tone sounds urgent, it is determined to be "nervous."

[1342] Step 6:

[1343] The server sends the parsed text data to a natural language processing (NLP) engine to identify the user's intent, which extracts the time information "tomorrow at 10 AM" and the task information "book a team meeting."

[1344] Step 7:

[1345] The server creates an API request to access the schedule management system based on the extracted task information. This API request includes task details (date and time, event content, etc.).

[1346] Step 8:

[1347] The server sends the created API request to the schedule management system. Specifically, it sends an HTTP POST request to the API endpoint of the schedule management system.

[1348] Step 9:

[1349] The schedule management system processes the received API request, creates a new event at the specified date and time, and returns a success response to the server when the processing is complete.

[1350] Step 10:

[1351] The server receives a response from the scheduling management system, confirms that the reservation was successful, and then generates a customized notification message based on the user's sentiment. For example, it could create a message like, "Your team meeting has been booked for tomorrow at 10 AM. Relax and move on to your next task!"

[1352] Step 11:

[1353] The server then sends the generated notification message to the terminal, again using a secure communication protocol (e.g., HTTPS).

[1354] Step 12:

[1355] The device displays the notification message received from the server to the user, for example, as a push notification or a pop-up message within the app.

[1356] Step 13:

[1357] The user will see a notification message on their device and know that the reservation has been made successfully, allowing them to move on to the next task with peace of mind.

[1358] Through the above steps, the system of the present invention can provide a system that not only enables efficient task management and business automation through voice input, but also takes into consideration the psychological state of the user.

[1359] Example 2

[1360] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1361] Conventional task management systems simply convert users' voice input into text data and identify tasks, but lack the ability to recognize users' emotions. Furthermore, they lack consideration for users, as they are unable to take their emotional state into account when managing tasks or automating tasks. Furthermore, notification messages that do not take users' emotions into account can potentially impair the user experience.

[1362] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing voice data and converting it into text data, a means for analyzing the text data and voice data and recognizing the user's emotions, and a means for analyzing the text data and identifying the corresponding task. This enables task management and business automation that take the user's emotions into consideration.

[1363] "Voice input" refers to data in a format in which the user inputs spoken words into the terminal using a microphone.

[1364] "Terminal means" refers to a device or apparatus for capturing voice input and performing the necessary processing.

[1365] "Server means" refers to a computer system that performs analysis and data processing, and manages and provides necessary information.

[1366] "Voice data" refers to digital data generated from the voice input by a user into a terminal.

[1367] "Text data" refers to data that has been analyzed and converted into textual information from audio data.

[1368] "Emotion recognition" refers to the process of analyzing voice and text data to determine a user's emotions and mental state.

[1369] "Task identification" refers to the process of analyzing text data to clarify the specific actions and instructions intended by the user.

[1370] A "schedule management system" refers to software or services for managing schedules and tasks.

[1371] "Notification" refers to messages or alerts that inform users of processing results or information.

[1372] A "natural language processing engine" refers to algorithms and technologies for analyzing text data and understanding and processing human language.

[1373] "Speech recognition engine" refers to the algorithms and technologies used to analyze voice data and convert it into corresponding text data.

[1374] The system of the present invention is equipped with a function for recognizing a user's emotions in addition to task management and business automation using voice input. MODES FOR CARRYING OUT THE INVENTION The following describes in detail embodiments of the present invention.

[1375] The user speaks into a device with a dedicated application installed. The device is equipped with a microphone to capture voice input, and while pressing the voice input button, the user speaks, "Schedule a team meeting tomorrow at 10:00 AM."

[1376] The device captures audio using a built-in microphone, temporarily stores the audio data, and then transmits it to a server via the internet. The hardware used here could be a smartphone or tablet.

[1377] The server analyzes the received voice data using a voice recognition engine (for example, a general voice recognition engine) and converts the voice data into text data. For example, text data such as "Please schedule a team meeting tomorrow at 10:00 AM" is generated.

[1378] The server then passes the converted text and voice data to an emotion engine (e.g., a general emotion analysis engine) to analyze the user's emotions. This determines whether the user is stressed or relaxed. For example, if the user's voice sounds rushed and tense, the emotion engine will recognize the user as being in a "tense" state.

[1379] Next, the server uses a natural language processing engine (e.g., a general natural language processing model) to analyze the text data in detail and identify the user's intent. In this case, the time information "tomorrow at 10:00 AM" and the task information "book a team meeting" are extracted.

[1380] Based on the identified task (i.e., booking a team meeting), the server makes an API request to access the scheduling system (e.g., a common calendar API). This request includes the date and time to book and details of the event. The server sends this request to the scheduling system, adding the event to the specified time slot.

[1381] Once the reservation is successfully completed, the server generates a message to notify the device of the result. This message is customized based on the user's emotions. For example, if the user is nervous, the server generates a message saying, "Your team meeting has been booked for tomorrow at 10:00 AM. Relax and move on to the next task!"

[1382] The terminal displays the notification message received from the server, letting the user know that the reservation has been made successfully. The content of the notification message is designed to help the user relax.

[1383] For example, if a user says "Schedule a team meeting tomorrow at 10 AM" and the emotion engine recognizes the user's nervousness, the following happens:

[1384] 1. The user speaks into the device.

[1385] 2. The device sends the audio data to the server.

[1386] 3. The server uses a speech recognition engine to convert the speech into text data.

[1387] 4. The server uses the emotion engine to recognize the user's emotions.

[1388] 5. The server uses a natural language processing engine to analyze the text data and identify tasks.

[1389] 6. The server accesses the schedule management system and adds the event to the specified time slot.

[1390] 7. The server generates a notification message of the processing result that takes into consideration the user's feelings.

[1391] 8. The server notifies the terminal of the processing results.

[1392] 9. The device displays the notification to the user.

[1393] 10. The user acknowledges the notification, relaxes, and moves on to the next task.

[1394] This system allows users to efficiently manage tasks through voice input and automate work processes while taking into consideration their psychological state.

[1395] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1396] Step 1:

[1397] The user provides voice input.

[1398] The user speaks into a device with a dedicated application installed, presses the voice input button, and says, "Schedule a team meeting for tomorrow at 10 a.m." This voice becomes the input data.

[1399] Step 2:

[1400] The device captures and transmits the audio data.

[1401] The device captures the user's voice using a built-in microphone, temporarily stores it as digital audio data, and then transmits the audio data to a server via the Internet. The output is the digitized audio data sent to the server.

[1402] Step 3:

[1403] The server converts the voice data into text data.

[1404] The server analyzes the received voice data using a speech recognition engine and converts it into text data. The collected voice data is converted into text information, and the text output is, for example, "Schedule a team meeting tomorrow at 10:00 AM."

[1405] Step 4:

[1406] The server recognizes the user's emotions from the text data and voice data.

[1407] The server then passes the converted text and voice data to an emotion recognition engine, which analyzes the user's emotions based on the tone and speed of the voice and the content of the text. For example, if the voice is spoken quickly and with a high pitch, it is determined to be "tension," and this information is output.

[1408] Step 5:

[1409] The server analyzes the text data and identifies the user's intent.

[1410] The server uses a natural language processing engine to perform detailed analysis of the text data and extract task and time information. For example, the analysis identifies instructions such as "tomorrow at 10:00 AM" and "book a team meeting." The output is specific task instructions.

[1411] Step 6:

[1412] The server accesses the schedule management system and reserves the event.

[1413] The server creates a request to the schedule management system's API based on the extracted task information. This request includes the reservation date and time and event details. The server sends this request to the schedule management system, which adds the event to the specified time slot. The output is the reservation execution result.

[1414] Step 7:

[1415] The server generates a reservation completion notification and sends it to the terminal.

[1416] The server confirms that the reservation has been successfully completed and generates a customized message based on the user's sentiment, for example, "Your team meeting has been booked for tomorrow at 10 AM. Relax and move on to the next task!", and sends this to the terminal. As an output, a customized notification message is generated.

[1417] Step 8:

[1418] The device displays a notification message.

[1419] The device displays the notification message received from the server as a pop-up notification or an in-app message, and the user can confirm that the reservation has been successfully made.

[1420] This means:

[1421] This enables efficient task management using voice input and business automation that takes into account the user's psychological state.

[1422] (Application example 2)

[1423] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1424] Conventional voice input systems simply convert voice data into text data and execute tasks based on that text. However, because they do not take into account the user's emotional state, they have the problem of not contributing to stress reduction or motivation improvement. In the food delivery field in particular, it is an issue that they are unable to provide appropriate responses when the user is in a hurry or feeling stressed.

[1425] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1426] In this invention, the server includes means for identifying the user's emotion from the voice data and text data, means for generating customized feedback according to the identified emotion, and means for notifying the terminal means of the processing result and the emotion-considered feedback, thereby making it possible to provide a response customized according to the user's emotional state.

[1427] "Voice input" refers to the act of generating voice data by the user speaking.

[1428] The "terminal means" is a device that allows a user to input voice, and includes a microphone.

[1429] The "server means" is a computer system that analyzes and processes data received from the terminal means via a network.

[1430] "Voice data" refers to the user's voice signals captured by the terminal means.

[1431] "Text data" is a collection of character information obtained by analyzing voice data.

[1432] A "task" refers to a specific action or work that a user requests by voice input.

[1433] A "schedule management system" is a software application for completing tasks and coordinating schedules.

[1434] "Emotion identification" is the process of analyzing audio and text data to determine the user's psychological state.

[1435] "Customized feedback" refers to response messages and guidance that are individually tailored to the user's emotional state.

[1436] The system of the present invention utilizes voice input for task management and business automation, and further recognizes the user's emotions to provide customized feedback. Detailed embodiments for implementing the present invention will be described below.

[1437] Hardware and software used

[1438] Hardware

[1439] Terminal means: A smartphone is used as a device for capturing voice input. This smartphone has a built-in microphone.

[1440] Server means: A computer system is used to analyze data and perform necessary processing.

[1441] software

[1442] Speech recognition engine: Uses the speech_recognition library to convert voice data into text data.

[1443] Natural Language Processing Engine: Uses the TextBlob library to analyze text data and identify tasks and sentiments.

[1444] HTTP Request Library: The requests library is used to communicate between the server means and the schedule management system and the feedback processing system.

[1445] System processing procedure

[1446] 1. Capture voice input

[1447] The user speaks into the microphone of the smartphone. Voice input is the act of generating voice data by speaking. For example, the user may say, "I'd like one pizza, please."

[1448] 2. Converting voice data to text

[1449] The voice data captured by the terminal means is transmitted to the server means, which converts the voice data into text data using a speech_recognition library.

[1450] 3. Emotion Identification

[1451] The server identifies the user's emotion from the converted text data and voice data. It analyzes the text data using the TextBlob library to determine the user's emotional state. For example, if the text is "One pizza please," it determines whether the user is "relaxed" or "in a hurry" based on the content and tone of the voice.

[1452] 4. Task execution

[1453] The server means sends an order to the food delivery API using the requests library based on the identified task, with the text data and emotion data included in the payload.

[1454] 5. Feedback Generation and Notification

[1455] The server generates customized feedback according to the user's emotional state and notifies the terminal, thereby allowing the user to move on to the next task without feeling stressed. For example, if the user is relaxed, a message such as "Your order has been completed. Please wait a moment" is displayed.

[1456] Specific examples

[1457] For example, if a user says "I'd like one pizza, please," the system converts the order into text, recognizes the user's emotion as "relaxed," sends the order to a food delivery API, and then displays feedback such as "Your order has been completed. Please wait a moment."

[1458] Prompt Sentence Examples

[1459] "Voice input: 'One pizza please'

[1460] Emotion Recognition: Relax

[1461] Action: User is relaxed, so show 'Your order is complete. Please wait a moment.'

[1462] In this way, the system of the present invention realizes efficient task management and business automation through voice input, and also provides services that take into consideration the psychological state of the user.

[1463] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1464] Step 1:

[1465] Capturing voice input

[1466] The user inputs voice by speaking into the terminal means. For example, the user may say, "I'd like one pizza, please." The terminal means uses a microphone to capture this voice data.

[1467] Input: User's voice

[1468] Output: Captured audio data

[1469] Step 2:

[1470] Converting audio data to text

[1471] The voice data captured by the terminal means is transmitted to the server means, which analyzes the voice data using the speech_recognition library and converts it into text data.

[1472] Input: Captured audio data

[1473] Output: Text data

[1474] Step 3:

[1475] Emotion Identification

[1476] The server identifies the user's emotion from the text data and voice data. It analyzes the text data using the TextBlob library to determine the user's emotional state. For example, if the text is "One pizza please" and the voice tone is calm, it determines that the user is "relaxed."

[1477] Input: Text data and tone information of audio data

[1478] Output: User's emotional state (e.g., relaxed, rushed, stressed)

[1479] Step 4:

[1480] Task execution

[1481] The server sends an order to the food delivery API based on the identified task, with the text data and emotion data included in the payload. It then uses the requests library to create an HTTP request and process the order.

[1482] Input: Text data, user's emotional state

[1483] Output: API request response (order confirmation)

[1484] Step 5:

[1485] Feedback Generation and Notification

[1486] The server means generates a feedback message customized according to the user's emotional state and notifies the terminal means of the message. For example, if the user is relaxed, the server means generates a message saying, "Your order is complete. Please wait a moment."

[1487] Input: API request response, user emotional state

[1488] Output: Customized feedback message

[1489] Step 6:

[1490] View Feedback

[1491] The terminal means displays the feedback message received from the server means to the user, thereby confirming that the order has been processed correctly.

[1492] Input: Customized feedback message

[1493] Output: Display of feedback message

[1494] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1495] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1496] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1497] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1498] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1499] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1500] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1501] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1502] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1503] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1504] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1505] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1506] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1507] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1508] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1509] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1510] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1511] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1512] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1513] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1514] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1515] The following is further disclosed regarding the above embodiment.

[1516] (Claim 1)

[1517] terminal means for capturing voice input;

[1518] a server means for analyzing the voice data received from the terminal means and converting it into text data;

[1519] a server means for analyzing the text data and identifying a corresponding task;

[1520] a server means for accessing the schedule management system based on the identified task and executing necessary processing;

[1521] a server means for notifying the terminal means of a processing result;

[1522] a terminal means for displaying the notification;

[1523] A system including:

[1524] (Claim 2)

[1525] 2. The system of claim 1, wherein the means for analyzing the voice input uses a voice recognition engine.

[1526] (Claim 3)

[1527] 2. The system according to claim 1, wherein the task analysis means analyzes the text data using a natural language processing engine.

[1528] "Example 1"

[1529] (Claim 1)

[1530] terminal means for capturing voice input;

[1531] a server means for analyzing the voice data received from the terminal means and converting it into text data;

[1532] a server means for analyzing the text data and identifying a corresponding task;

[1533] a server means for accessing the schedule management system based on the identified task and executing necessary processing;

[1534] a server means for notifying the terminal means of a processing result;

[1535] a terminal means for displaying the notification;

[1536] A system including:

[1537] (Claim 2)

[1538] 2. The system of claim 1, wherein the means for analyzing the voice input uses a voice recognition engine.

[1539] (Claim 3)

[1540] 2. The system according to claim 1, wherein the task analysis means analyzes the text data using a natural language processing engine.

[1541] "Application Example 1"

[1542] (Claim 1)

[1543] terminal means for capturing voice input;

[1544] a server means for analyzing the voice data received from the terminal means and converting it into text data;

[1545] a server means for analyzing the text data and identifying a corresponding task;

[1546] a server means for accessing the work machine control system based on the identified task and executing necessary processing;

[1547] a server means for notifying the terminal means of a processing result;

[1548] a terminal means for displaying the notification;

[1549] A system including:

[1550] (Claim 2)

[1551] 2. The system of claim 1, wherein the means for analyzing the voice input uses a voice recognition engine.

[1552] (Claim 3)

[1553] 2. The system according to claim 1, wherein the task analysis means analyzes the text data using a natural language processing engine.

[1554] "Example 2: Combining Emotion Engines"

[1555] (Claim 1)

[1556] terminal means for capturing voice input;

[1557] a server means for analyzing the voice data received from the terminal means and converting it into text data;

[1558] a server means for analyzing the text data and the voice data and recognizing the user's emotion;

[1559] a server means for analyzing the text data and identifying a corresponding task;

[1560] a server means for accessing the schedule management system based on the identified task and executing necessary processing;

[1561] a server means for notifying the terminal means of a processing result;

[1562] a terminal means for displaying the notification;

[1563] A system including:

[1564] (Claim 2)

[1565] 2. The system of claim 1, wherein the means for analyzing the voice input uses a voice recognition engine.

[1566] (Claim 3)

[1567] 2. The system according to claim 1, wherein the task analysis means analyzes the text data using a natural language processing engine.

[1568] "Application example 2 when combining emotion engines"

[1569] (Claim 1)

[1570] terminal means for capturing voice input;

[1571] a server means for analyzing the voice data received from the terminal means and converting it into text data;

[1572] a server means for analyzing the text data and identifying a corresponding task;

[1573] a server means for accessing the schedule management system based on the identified task and executing necessary processing;

[1574] a server means for identifying a user's emotion from the voice data and text data;

[1575] server means for generating customized feedback in response to the identified emotion;

[1576] a server means for notifying the terminal means of a processing result and feedback that takes emotion into consideration;

[1577] a terminal means for displaying the notification;

[1578] A system including:

[1579] (Claim 2)

[1580] 2. The system of claim 1, wherein the means for analyzing the voice input uses a voice recognition engine.

[1581] (Claim 3)

[1582] 2. The system according to claim 1, wherein the task analysis means analyzes the text data using a natural language processing engine. [Explanation of symbols]

[1583] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. terminal means for capturing voice input; a server means for analyzing the voice data received from the terminal means and converting it into text data; a server means for analyzing the text data and identifying a corresponding task; a server means for accessing the schedule management system based on the identified task and executing necessary processing; a server means for notifying the terminal means of a processing result; a terminal means for displaying the notification; A system including:

2. 2. The system of claim 1, wherein the means for analyzing the speech input utilizes a speech recognition engine.

3. 2. The system according to claim 1, wherein the task analysis means analyzes the text data using a natural language processing engine.

Images