system

Abstract

We provide the system. [Solution] A means for receiving audio data and converting it into text data using speech recognition technology, A means for receiving image data and extracting task-related information using image recognition technology, A method for generating and prioritizing tasks from text data using natural language processing techniques, A means of sending the generated task to the terminal and setting a reminder, A means to display a task list to the user and allow them to edit tasks, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In many companies and organizations, it is an issue to efficiently manage tasks that occur during meetings and daily operations. Conventionally, it has been necessary to manually input and manage tasks, which has been difficult work especially for people with attention disorders. Also, there is a risk of missing important tasks or misjudging priorities, which may lead to a decrease in work efficiency.

Means for Solving the Problems

[0005] This invention provides a system that automatically generates and manages tasks by receiving and analyzing audio and image data. It converts audio data into text using speech recognition technology and extracts task-related information using image recognition technology. Furthermore, natural language processing technology instantly forms tasks from the generated text data and sets priorities. The generated tasks are notified to the user via a terminal, allowing the user to easily edit and manage them. In particular, to support users with attention deficits, enhanced reminder functions can prevent important tasks from being overlooked and improve work efficiency.

[0006] "Audio data" refers to sound information collected in a format that can be analyzed by speech recognition technology.

[0007] "Speech recognition technology" is a technology that converts speech data into text data and is used to automatically analyze the content of speech.

[0008] "Text data" refers to data expressed in string format, and includes digital information such as linguistic information.

[0009] "Image data" refers to visual information collected in a format that can be analyzed by image recognition technology.

[0010] "Image recognition technology" refers to the technology that analyzes image data and automatically extracts specific patterns or information.

[0011] "Task-related information" refers to information extracted from images and audio that should be managed as a task.

[0012] "Natural language processing technology" refers to the technology that analyzes text data, understands the meaning and structure of language, and processes it.

[0013] A "task" refers to a task or job that a user must perform, and is managed as information including its schedule, deadline, and priority.

[0014] "Priority" refers to an indicator for determining the order of execution based on the importance and urgency of tasks.

[0015] "Terminal" refers to an electronic device that provides an interface directly operated by a user, and receives and displays information from a server.

[0016] "Reminder" refers to a function that notifies a user at the deadline or important time point of a task, and has the role of preventing task omissions.

Brief Explanation of Drawings

[0017] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. ​​​​​​​​​​​​​​​​​​​​​​​​​It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the language used in the following description will be explained.

[0020] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0021] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] This invention is a system that automatically generates and manages tasks from voice and images, aiming to improve the user's work efficiency. This system is implemented through interaction between a server, a terminal, and the user.

[0039] Server Embodiment

[0040] The server converts received audio data into text data using speech recognition technology. This makes it possible to transcribe speech and instructions during meetings into text. It also uses image recognition technology to extract task-related information from whiteboards and handwritten notes. Furthermore, it uses natural language processing technology to analyze the information obtained from audio and images and automatically generate tasks. The generated tasks are assigned priorities, and the server registers them in a database and sends them to the terminal.

[0041] Terminal embodiment

[0042] The terminal receives task data sent from the server and displays it to the user. The specific task details, deadlines, and priorities are visually displayed, allowing the user to plan their daily work based on this information. Furthermore, the terminal receives reminder notifications from the server and informs the user at the appropriate time.

[0043] User interaction

[0044] Users perform tasks based on those displayed on their device. The task list is intuitive and easy to understand, allowing users to quickly grasp what needs to be done next. Users can also edit and add tasks via their device, and adjust task priorities and deadlines as needed.

[0045] Specific example

[0046] If, during a meeting, the task "Submit the project plan by next Monday" is spoken aloud, the device sends the audio to the server. The server uses speech recognition to transcribe it into text, recognizing it as the task "Submit the project plan." Further, natural language processing technology analyzes the deadline as "next Monday," generates a task based on this, and sends it to the device. The user can then check the task displayed on the device and add it to their schedule. A reminder function automatically notifies the user as the deadline approaches, preventing tasks from being missed.

[0047] Thus, the present invention provides a system that can efficiently manage user tasks by utilizing voice and image data. In particular, for users with attention disorders, the automatic task generation and reminder functions contribute to supporting their work.

[0048] The following describes the processing flow.

[0049] Step 1:

[0050] The device uses a microphone and camera to capture audio and image data. It acquires audio of what the user says during a meeting and content written on a whiteboard, and prepares to send that data to a server.

[0051] Step 2:

[0052] The server receives audio data transmitted from the terminal. Using a speech recognition engine, this audio data is converted into text data, and the content of the meeting is obtained as written information.

[0053] Step 3:

[0054] The server also receives image data simultaneously. An image recognition algorithm is applied to extract text information from the image. This converts the content written on the whiteboard or in handwritten notes into text.

[0055] Step 4:

[0056] The server analyzes text information obtained from audio and image data using natural language processing techniques. It identifies information that should be managed as tasks and registers them in the database as structured tasks.

[0057] Step 5:

[0058] The server prioritizes the generated tasks. Priorities are automatically set based on the importance and deadline of each task.

[0059] Step 6:

[0060] The server sends prioritized tasks to the terminal. In addition, it sets appropriate reminders and prepares to send notifications at the appropriate time.

[0061] Step 7:

[0062] The terminal receives task information sent from the server and displays it to the user as a task list. The task details, priority, and deadline are visually displayed.

[0063] Step 8:

[0064] Users review tasks displayed on their devices and edit or add tasks as needed. They adjust task priorities and deadlines and send updated information to the server via their devices.

[0065] Step 9:

[0066] The device receives pre-set reminders and notifies the user at the appropriate time, ensuring that the user does not miss important tasks.

[0067] (Example 1)

[0068] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0069] In today's workplaces and work environments, information is diverse and its management is complex. In particular, audio and image information, if not processed quickly and accurately, can lead to decreased work efficiency and information leaks. Furthermore, for users with attention deficits, prioritizing tasks and optimizing important notifications are crucial. To effectively address these challenges, there is a need for systems that utilize audio and image to automate work management and improve work efficiency.

[0070] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0071] In this invention, the server includes means for receiving audio information and converting it into text information using speech recognition technology, means for receiving image information and extracting business-related information using image recognition technology, and means for generating tasks from the text information and prioritizing them using natural language processing technology. This enables rapid and accurate processing of audio and image information, automatic generation and prioritization of tasks, prevention of leakage of important information, and improvement of user work efficiency.

[0072] "Audio information" refers to data that represents human speech as digital signals, and is the subject of speech recognition technology.

[0073] "Speech recognition technology" is a technology that receives audio information as input and converts it into text information.

[0074] "Textual information" refers to data in text format converted from audio information using speech recognition technology.

[0075] "Image information" refers to providing visual information, such as whiteboards or handwritten notes, as digital data.

[0076] "Image recognition technology" is a technology that extracts specific information or characters from image information, enabling machines to understand visual information.

[0077] "Business-related information" refers to information necessary for carrying out business operations, obtained through image recognition technology.

[0078] "Natural language processing technology" is a technology that analyzes textual information, understands its meaning, and uses it to generate and prioritize tasks.

[0079] "Work" refers to specific tasks or operations generated based on audio and image information.

[0080] "Priority" is an indicator that shows the urgency and importance of executing a generated task, and contributes to the efficiency of task management.

[0081] An "information processing terminal" is a device that receives business information transmitted from a server and presents it visually to the user.

[0082] A "notification function" is a system that informs users of important tasks and deadlines via their information processing terminals.

[0083] "Users" refer to the individuals who use this system and are the entities responsible for managing and carrying out their tasks.

[0084] A "task list" is a list-format display shown on an information processing terminal that shows the specific details of the tasks that have been generated.

[0085] "Generative AI technology" is a technology that uses artificial intelligence to automatically generate and adjust tasks, thereby improving efficiency.

[0086] This invention is a system that automatically generates and manages tasks using voice and image information, with the aim of improving the user's work efficiency. This system is realized through the interaction of a server, terminals, and users.

[0087] Server Embodiment

[0088] The server receives voice information sent from the user and converts it into text using speech recognition technology. Specifically, tools such as Google® Cloud Speech-to-Text API could be used. For image information, image recognition technologies such as OpenCV and Google Cloud Vision API are used to extract business-related information. Furthermore, natural language processing technologies such as spaCy and Google Cloud Natural Language API are used to analyze and generate business information from the text information and assign priorities. This generated business information is stored in a database and sent to the terminal.

[0089] Terminal embodiment

[0090] The terminal receives task information sent from the server. The received tasks are displayed visually to the user, clearly showing their content, deadline, and priority. The terminal also supports the server's notification function, providing users with timely reminders and preventing tasks from being overlooked.

[0091] User interaction

[0092] Users perform their daily tasks by referring to work information displayed on their terminals. The tasks are displayed visually in a list format, allowing users to operate them intuitively. Furthermore, users can edit, add, and adjust the priority of tasks via the terminal. This enables users to flexibly manage their own schedules.

[0093] Specific example

[0094] For example, if a voice message is made during a meeting stating, "Submit the report by next Monday," the user's device sends this message to the server. The server uses speech recognition technology to transcribe the message into text and then uses natural language processing to generate the task "Submit Report." This task is prioritized with a deadline of "next Monday" and sent to the device. The user can then review the task displayed on their device and manage or adjust tasks as needed.

[0095] Example of a prompt

[0096] "Please describe a system that automatically generates and manages tasks based on comments made during meetings."

[0097] As described above, by combining these technologies, it is possible to reduce the burden on users in managing their work and improve operational efficiency.

[0098] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0099] Step 1:

[0100] The server receives audio information from the user. This audio information is converted into text using speech recognition technology. The audio data, as input, is processed using the Google Cloud Speech-to-Text API. This extracts the spoken content as text. As a concrete example, during a meeting, the user sends the audio message "The project deadline is next Monday" to the server.

[0101] Step 2:

[0102] The server receives image information from the user. Image recognition technology is used to extract business-related information from the image data. The input image data is processed using OpenCV or the Google Cloud Vision API, and the output is the content of a whiteboard or handwritten note converted into text. As a specific example, the user sends a photo of a whiteboard with the text "Marketing Strategy Meeting" written on it to the server.

[0103] Step 3:

[0104] The server analyzes textual information using natural language processing (NLP) technology to generate task content. The textual information used as input is processed by NLP tools such as spaCy. Through analysis, the task content and deadline are identified, and the task is generated. For example, the textual information "The project deadline is next Monday" becomes the task "Project Deadline," and a deadline is set.

[0105] Step 4:

[0106] The server prioritizes the generated tasks. Using a generation AI model, it evaluates the importance and urgency of the tasks and determines their priority. The output is a list of tasks with assigned priorities. Specifically, tasks with "project deadlines" are compared with other tasks, and their priority is determined.

[0107] Step 5:

[0108] The server sends prioritized task information to the terminal. The information processing terminal receives the task details and presents them visually to the user. The task data, as input, is sent from the server and displayed as a task list on the terminal's display. Specifically, the smartphone displays "Project Deadline (High Priority)".

[0109] Step 6:

[0110] Users check the work information displayed on their terminal and perform their tasks. Users can edit, add, and adjust the priority of tasks. Through these operations, they can adjust deadlines and add new tasks as needed. Specifically, users can change the deadline for a "project deadline" and add related tasks, among other adjustments.

[0111] (Application Example 1)

[0112] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0113] In modern factories, improving work efficiency is crucial, but manually transcribing voice instructions and whiteboard information into text and managing work instructions is time-consuming and laborious. Furthermore, delays and errors can occur due to slow real-time updates of work instructions and inefficient transmission of instructions to automated equipment. Additionally, it is essential to ensure that important work instructions are reliably communicated to workers with attention deficits. Improving these conditions and streamlining work management is necessary.

[0114] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0115] In this invention, the server includes means for receiving acoustic information and converting it into document information through language processing, means for receiving visual information and extracting work-related information using object recognition technology, and means for generating work instructions from the document information and assigning priorities using natural language processing technology. This makes it possible to quickly generate work instructions from voice instructions and visual information and transmit and notify automated devices in real time. This improves work efficiency and enables appropriate instruction notification to workers with attention impairments.

[0116] "Acoustic information" refers to all data related to voice and sound, including the content of voice instructions.

[0117] "Language processing" refers to the technology of converting audio data into natural language, generating document information from acoustic information.

[0118] "Document information" refers to converted text data, which is information that has been transcribed from spoken instructions into text.

[0119] "Visual information" refers to all data related to images and videos, including information written on whiteboards or paper.

[0120] "Object recognition technology" refers to technology that automatically identifies and extracts specific information from visual information, and is a means of obtaining work-related information.

[0121] "Work-related information" refers to information about the content and methods of performing a task, and is extracted from visual information.

[0122] "Natural language processing technology" refers to the technology that generates work instructions from document information and sets priorities for those instructions.

[0123] A "work instruction" refers to an instruction that clearly specifies the tasks to be performed, and is sent to a terminal or automated device as a generated task.

[0124] "Priority" refers to an indicator that determines the urgency and importance of carrying out a work instruction.

[0125] "Automated equipment" refers to machines and robots that operate in factories and production lines based on work instructions.

[0126] "Notification" refers to a means of informing users or automated equipment of generated work instructions, and includes providing information in real time.

[0127] The system implementing this invention is designed to improve the efficiency of work instructions in factories and production lines. This invention utilizes acoustic and visual information to instantly generate work instructions and notify automated equipment and users.

[0128] The server uses smart devices (e.g., smartphones or smart glasses) equipped with high-sensitivity microphones to receive acoustic information. The server converts this acoustic information into document information using speech recognition software such as the Google Speech-to-Text API. Visual information is similarly acquired by devices equipped with cameras, and work-related information is extracted using object recognition technology such as OpenCV or TENSORFLOW®.

[0129] Furthermore, the server uses natural language processing technology to generate work instructions from document information and assigns priorities using generative AI models such as BERT. These generated work instructions are sent to automated equipment in real time, improving the efficiency of factory operations.

[0130] For example, if a worker says, "Prepare for shipment of product A by 2 PM," the system receives this voice command and generates a work instruction called "Prepare for shipment of product A." At the same time, if "Ship 10 units of product B" is written on a whiteboard, the system acquires this as visual information, generates an instruction called "Ship product B," and executes them sequentially based on priority.

[0131] For example, a prompt could be: "Prepare product A for shipment by 2 PM. Then ship 10 units of product B." Based on this prompt, the system efficiently generates work instructions and sends them to the automated equipment.

[0132] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0133] Step 1:

[0134] The server receives acoustic information through the microphone of a smart device. This acoustic information is converted into document information using the Google Speech-to-Text API. The acoustic information is input as audio data, and the converted document information is output as text data.

[0135] Step 2:

[0136] The server receives visual information using the camera of a smart device. The received visual information is analyzed using object recognition technology with OpenCV or TensorFlow, and task-related information is extracted. The visual information is input as image data, and the task-related information is output as text data.

[0137] Step 3:

[0138] The server analyzes document information and work-related information using natural language processing technology, specifically generative AI models such as BERT, to generate work instructions and assign priorities. Document information and work-related information are input, and the generated work instructions are output as text data.

[0139] Step 4:

[0140] The server sends the generated work instructions to the automated equipment. Information is transmitted in real time, and the automated equipment is configured to operate based on the instructions. The work instructions are provided as input and output as executable commands.

[0141] Step 5:

[0142] Users can review the work instructions displayed on the terminal and make corrections or approvals as needed. The terminal displays the entered instructions on the screen, accepts user input, and sends the updated instructions to the server.

[0143] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0144] This invention combines an emotion engine with a system that automatically generates and manages tasks from voice and image data. This enables task management that takes into account the user's emotional state, thereby improving work efficiency. The system's configuration and operation are described below in detail in the following specific embodiments.

[0145] Server Embodiment

[0146] The server converts the audio data received from the terminal into text using speech recognition technology, and also analyzes the user's emotions from the audio using an emotion engine. The resulting text data is then analyzed using natural language processing technology, and tasks are automatically generated. The generated tasks are prioritized based on the emotion data. For example, if the user is feeling stressed, the priority of the task is temporarily set lower to reduce the burden.

[0147] The server also receives image data and extracts task-related information using image recognition technology. Furthermore, it utilizes the results of sentiment analysis to send users not only appropriate reminders but also encouraging messages when necessary.

[0148] Terminal embodiment

[0149] The device receives task and emotion-based notifications sent from the server and displays them to the user. Tasks are organized by priority, making them easy for the user to review and manage. Depending on the emotional state, the device displays timely reminders, work-related suggestions, and encouraging messages.

[0150] User interaction

[0151] Users perform tasks based on a task list displayed on their device. Additional tasks or changes can be easily updated via the device. The device provides notifications that take sentiment data into account, allowing users to efficiently complete tasks while receiving emotion-based feedback.

[0152] Specific example

[0153] If a user verbally states, "I'll finish the presentation materials for next week," while preparing for a presentation, the device sends this audio data to the server. The server uses speech recognition to transcribe the data into text and an emotion engine to analyze the user's emotions regarding the presentation. If the emotion "nervousness" is detected, the server generates a task, adjusts the priority of related tasks to a lower level, and sends it to the device. In addition to the task list, the device displays a message to the user suggesting ways to relax.

[0154] In this way, the present invention provides a system that reduces the psychological burden on users and improves work efficiency by combining emotion analysis with task management. In particular, for users with attention disorders, work support can be enhanced by sending notifications that are sensitive to their emotions.

[0155] The following describes the processing flow.

[0156] Step 1:

[0157] The device captures the user's speech using a microphone and records it as audio data. Furthermore, it also records additional data such as tone and speed of voice to take into account the user's emotions.

[0158] Step 2:

[0159] The device sends audio data to the server. During this process, metadata, including the time and location of the capture, is added to the audio data.

[0160] Step 3:

[0161] The server converts the received audio data into text data using speech recognition technology. This allows the user's spoken content to be obtained as text information.

[0162] Step 4:

[0163] The server analyzes the voice data using an emotion engine to estimate the user's emotional state. For example, it analyzes the tone and speed of the voice to detect emotions such as tension and stress.

[0164] Step 5:

[0165] The server analyzes the transcribed data using natural language processing technology and automatically generates related tasks. It identifies the content and deadlines of tasks spoken aloud and registers them in the system.

[0166] Step 6:

[0167] The server adjusts the priority of generated tasks, taking emotional data into consideration. If a user is experiencing stress, it will lower the priority of non-urgent tasks, for example.

[0168] Step 7:

[0169] The server sends task information to the terminal. It also generates and transmits emotionally-based messages of encouragement and suggestions to reduce the burden.

[0170] Step 8:

[0171] The device displays a task list and notifications based on sentiment analysis to the user. The task list is organized by priority, and encouraging messages are displayed alongside it.

[0172] Step 9:

[0173] Users can view tasks on their devices and edit or add actions as needed. Emotion-responsive feedback allows for more effective task management.

[0174] Step 10:

[0175] The device feeds back task progress and new updates to the server. This improves the accuracy of task management and the quality of services based on user sentiment data.

[0176] (Example 2)

[0177] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0178] Traditional task management systems process and manage tasks without considering the user's emotional state, which can lead to psychological burden and decreased work efficiency. Furthermore, users with attention deficits require particularly considerate notification methods. Therefore, there is a need for a system that can adjust task priorities according to the user's emotional state, enabling efficient and less burdensome task management.

[0179] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0180] In this invention, the server includes means for receiving audio data and converting it into text information using acoustic analysis technology, means for receiving image information and extracting work-related information using image analysis technology, and means for evaluating the user's emotional state using emotion analysis technology and adjusting the priority of tasks based on that state. This enables flexible task management based on the user's emotional state.

[0181] "Acoustic analysis technology" is a technology that analyzes audio data as a digital signal and converts it into textual information.

[0182] "Image analysis technology" is a technique that processes image information as digital data and extracts meaningful information and patterns.

[0183] "Natural language processing technology" is a technology that enables computers to understand, interpret, and generate human language.

[0184] "Emotional analysis technology" is a technology that evaluates and analyzes a user's emotional state from data such as voice and text.

[0185] "User terminal" refers to an electronic device used by a user to view and edit various notifications and information.

[0186] A "task list" refers to a collection of tasks generated by the system, organized by priority and content.

[0187] "Attention deficit" refers to a condition characterized by difficulty in sustaining attention, and conventional methods of treatment are often complex or insufficient.

[0188] This invention is a system that analyzes voice and image information along with the user's emotional state to effectively manage tasks. Specific embodiments of this system are described below.

[0189] Server Embodiment

[0190] The server receives audio data from the terminal and converts it into text data using acoustic analysis technology. Specifically, it utilizes general-purpose audio analysis software. It also receives image information and extracts task-related information using image analysis technology. General-purpose image analysis software is used for this process. Next, the server uses sentiment analysis technology to evaluate the user's emotions from the audio and text data and adjusts the priority of the tasks generated based on that evaluation. Sentiment analysis software is used for this process. The generated tasks and associated messages are sent to the user's terminal, and the user is notified.

[0191] Terminal embodiment

[0192] The terminal receives task information and messages sent from the server and displays them to the user. This allows the user to manage tasks according to their own status. The terminal can also edit tasks based on user actions, allowing the user to easily add new tasks or modify existing ones.

[0193] User interaction

[0194] Users proceed with their tasks based on a list of tasks displayed on their device. Furthermore, if improvements are needed, they can receive suggestions for task modifications or emotionally-driven suggestions via their device. This allows users to efficiently perform tasks in a way that suits their emotional state.

[0195] Specific example

[0196] For example, if a user says, "I'll finish preparing for next week's meeting," the server converts this audio into text and, through sentiment analysis, determines that the user is feeling "stressed." The server uses this information to adjust the priority of related tasks and provides the user with tasks that include messages suggesting relaxation. This allows the user to work while taking their emotions into consideration.

[0197] An example of a prompt message is: "Convert this audio data to text, analyze the user's emotions, and generate optimal task management suggestions."

[0198] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0199] Step 1:

[0200] The server receives audio data from the terminal. This audio data contains information in which the user has given linguistic instructions to the system. The server converts the audio data into text data using acoustic analysis technology. In this process, speech analysis software is used to analyze the audio waveform and generate a string of characters. The output is text data converted by speech recognition.

[0201] Step 2:

[0202] The server uses text obtained from audio data as input and evaluates the user's emotional state using sentiment analysis technology. In this step, sentiment analysis software is used to analyze emotions such as "tension" or "joy" from factors such as sentence tone and language choice. The output is tag information for the emotions the user is presumed to be feeling.

[0203] Step 3:

[0204] The server uses natural language processing techniques to generate tasks based on the text data obtained in the previous step. Furthermore, it prioritizes the tasks, taking into account the sentiment information obtained in step 2. Specifically, it utilizes a generative AI model to determine tasks based on user requests and their associated priorities. The output is task information prioritized based on sentiment.

[0205] Step 4:

[0206] The server also processes image information received from terminals. Here, image analysis techniques are used to extract task-related information from the image data. For example, key points from meeting slides or documents are identified and incorporated into the task. The output is supplementary information that can be used for task generation.

[0207] Step 5:

[0208] The server sends the generated task information and appropriate messages to the user's terminal. These messages include reminders and suggestions based on the user's emotions. As output, the terminal displays a task summary and an emotionally sensitive message to the user.

[0209] Step 6:

[0210] The terminal displays received task information and messages to the user. The user can check the task list on the terminal and perform tasks based on it. Tasks can also be edited as needed. Input is data from the server, and output is the content displayed on the user interface.

[0211] (Application Example 2)

[0212] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0213] To address the diverse tasks and problems citizens face in their daily lives, efficient task management that takes emotional states into account is required. However, conventional task management systems struggle to reflect users' emotional states in order to appropriately prioritize tasks and make suggestions that contribute to stress reduction, potentially increasing citizens' stress levels. Addressing these challenges is essential.

[0214] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0215] In this invention, the server includes means for receiving voice data and converting it into text data using voice recognition technology, means for analyzing emotions from the voice using an emotion engine and adjusting priorities based on the emotional state, and means for sending the generated tasks to a terminal and setting emotionally sensitive reminders and suggestion messages. This enables optimal task management and stress reduction suggestions according to the emotional state of citizens.

[0216] "Audio data" refers to audio information recorded in digital format and processed by a device.

[0217] "Speech recognition technology" is a technology that converts input speech into text and then understands and analyzes its content.

[0218] "Text data" refers to character information converted using technologies such as speech recognition, and is digital data that can be analyzed by a system.

[0219] "Image data" refers to visual information recorded in a digital format that can be processed mechanically.

[0220] "Image recognition technology" is a technique that extracts and analyzes specific information or patterns from image data.

[0221] "Information" refers to data and knowledge necessary to achieve a specific purpose.

[0222] An "emotion engine" is an algorithm or technology that analyzes, quantifies, and evaluates a user's emotional state from voice and text data.

[0223] "Priority" refers to the criteria used to determine the order in which tasks or processes should be performed, and is a sequence set based on importance and urgency.

[0224] "Natural language processing technology" is a general term encompassing technologies for handling human language on computers, including semantic analysis and information extraction.

[0225] A "task" refers to a series of tasks or activities performed in order to achieve a specific goal.

[0226] A "reminder" is a feature that notifies you to ensure you don't forget important tasks or events.

[0227] A "suggestion message" is a message that includes advice or guidance provided to encourage users to take a specific action or think a particular way.

[0228] A "terminal" refers to a device used by a user to access and operate a system.

[0229] A "task list" refers to a list of tasks organized for the user to manage and review.

[0230] "Editing" refers to the process of changing or updating existing data or information.

[0231] In this system, the server uses speech recognition technology to convert citizens' voice data into text data, and then analyzes that text data using natural language processing technology to generate appropriate tasks. It also uses an emotion engine to perform sentiment analysis and adjust task priorities based on the user's emotional state. Based on the sentiment analysis results, it generates suggestion messages and reminders to reduce stress. Tasks and messages sent from the server are displayed on the terminal, allowing the user to review them and edit tasks as needed.

[0232] The specific processing uses technologies such as the Google Cloud Speech-to-Text API for speech recognition and spaCy and the BERT model for natural language processing. Sentiment analysis is performed using APIs such as AWS® Comprehend and Microsoft® Azure® Text Analytics. By combining these, the server can manage tasks in real time, taking into account the user's voice and emotions, providing personalized support to citizens of the smart city.

[0233] For example, if a citizen uses their smartphone to voice-input "I want to solve the park litter problem," this voice data is converted to text and analyzed on a server. If the emotional state is determined to be "dissatisfied," the system will prioritize the task and suggest contacting the relevant authorities immediately. The user will also be notified with a message such as "We will address this promptly," providing reassurance. An example of a prompt in this case would be: "A citizen's request to solve the park litter problem. We will consider the emotional state and suggest the most appropriate response."

[0234] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0235] Step 1:

[0236] The user provides voice input to the device. Speech recognition technology (Google Cloud Speech-to-Text API) is used to convert the voice data into text data. The input is voice data, and the output is text data. During this process, appropriate formatting is performed as the voice signal is converted into text.

[0237] Step 2:

[0238] The server analyzes the generated text data using natural language processing techniques (such as spaCy and the BERT model) and extracts information. The input is the text data obtained in step 1, and the output is the analyzed information such as meaning and keywords. Various natural language processing techniques, including text structuring and semantic analysis, are used for this data processing.

[0239] Step 3:

[0240] The server uses an emotion engine (such as AWS Comprehend or Microsoft Azure Text Analytics) to analyze user emotions from text data. The input is the text data obtained in step 1, and the output is data that quantifies the user's emotional state. In this data calculation, emotion indicators are extracted, and the user's psychological state is evaluated.

[0241] Step 4:

[0242] The server generates tasks and sets priorities based on the sentiment analysis results. The input is the information obtained in steps 2 and 3, and the output is task data with sentiment-based priorities. This process uses a generative AI model to provide personalized task suggestions to the user.

[0243] Step 5:

[0244] The server sends generated tasks and suggestion messages to the terminal, including emotionally sensitive reminders. The input is the task data obtained in step 4, and the output is information to the user in notification format.

[0245] Step 6:

[0246] The user reviews tasks and messages received on their device and edits the task list as needed. The input is information from step 5, and the output is the updated task list. In this step, tasks are fine-tuned through user interaction.

[0247] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0248] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0249] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0250] [Second Embodiment]

[0251] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0252] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0253] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0254] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0255] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0256] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0257] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0258] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0259] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0260] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0261] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0262] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0263] This invention is a system that automatically generates and manages tasks from voice and images, aiming to improve the user's work efficiency. This system is implemented through interaction between a server, a terminal, and the user.

[0264] Server Embodiment

[0265] The server converts received audio data into text data using speech recognition technology. This makes it possible to transcribe speech and instructions during meetings into text. It also uses image recognition technology to extract task-related information from whiteboards and handwritten notes. Furthermore, it uses natural language processing technology to analyze the information obtained from audio and images and automatically generate tasks. The generated tasks are assigned priorities, and the server registers them in a database and sends them to the terminal.

[0266] Terminal embodiment

[0267] The terminal receives task data sent from the server and displays it to the user. The specific task details, deadlines, and priorities are visually displayed, allowing the user to plan their daily work based on this information. Furthermore, the terminal receives reminder notifications from the server and informs the user at the appropriate time.

[0268] User interaction

[0269] Users perform tasks based on those displayed on their device. The task list is intuitive and easy to understand, allowing users to quickly grasp what needs to be done next. Users can also edit and add tasks via their device, and adjust task priorities and deadlines as needed.

[0270] Specific example

[0271] If, during a meeting, the task "Submit the project plan by next Monday" is spoken aloud, the device sends the audio to the server. The server uses speech recognition to transcribe it into text, recognizing it as the task "Submit the project plan." Further, natural language processing technology analyzes the deadline as "next Monday," generates a task based on this, and sends it to the device. The user can then check the task displayed on the device and add it to their schedule. A reminder function automatically notifies the user as the deadline approaches, preventing tasks from being missed.

[0272] Thus, the present invention provides a system that can efficiently manage user tasks by utilizing voice and image data. In particular, for users with attention disorders, the automatic task generation and reminder functions contribute to supporting their work.

[0273] The following describes the processing flow.

[0274] Step 1:

[0275] The device uses a microphone and camera to capture audio and image data. It acquires audio of what the user says during a meeting and content written on a whiteboard, and prepares to send that data to a server.

[0276] Step 2:

[0277] The server receives audio data transmitted from the terminal. Using a speech recognition engine, this audio data is converted into text data, and the content of the meeting is obtained as written information.

[0278] Step 3:

[0279] The server also receives image data simultaneously. An image recognition algorithm is applied to extract text information from the image. This converts the content written on the whiteboard or in handwritten notes into text.

[0280] Step 4:

[0281] The server analyzes text information obtained from audio and image data using natural language processing techniques. It identifies information that should be managed as tasks and registers them in the database as structured tasks.

[0282] Step 5:

[0283] The server assigns priorities to the generated tasks. Based on the importance and deadline of the tasks, priorities are automatically set.

[0284] Step 6:

[0285] The server sends the tasks with assigned priorities to the terminal. In addition, appropriate reminders are set to prepare for notification at the corresponding timing.

[0286] Step 7:

[0287] The terminal receives the task information sent from the server and displays it as a task list to the user. The content, priority, deadline, etc. of the tasks are visually shown.

[0288] Step 8:

[0289] The user checks the tasks displayed on the terminal and edits or adds them as needed. The user adjusts the priority and deadline of the tasks and sends the updated information to the server via the terminal.

[0290] Step 9:

[0291] The terminal receives the pre-set reminder and notifies the user at the appropriate timing. This ensures that the user does not miss important tasks.

[0292] (Example 1)

[0293] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0294] In today's workplaces and work environments, information is diverse and its management is complex. In particular, audio and image information, if not processed quickly and accurately, can lead to decreased work efficiency and information leaks. Furthermore, for users with attention deficits, prioritizing tasks and optimizing important notifications are crucial. To effectively address these challenges, there is a need for systems that utilize audio and image to automate work management and improve work efficiency.

[0295] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0296] In this invention, the server includes means for receiving audio information and converting it into text information using speech recognition technology, means for receiving image information and extracting business-related information using image recognition technology, and means for generating tasks from the text information and prioritizing them using natural language processing technology. This enables rapid and accurate processing of audio and image information, automatic generation and prioritization of tasks, prevention of leakage of important information, and improvement of user work efficiency.

[0297] "Audio information" refers to data that represents human speech as digital signals, and is the subject of speech recognition technology.

[0298] "Speech recognition technology" is a technology that receives audio information as input and converts it into text information.

[0299] "Textual information" refers to data in text format converted from audio information using speech recognition technology.

[0300] "Image information" refers to providing visual information, such as whiteboards or handwritten notes, as digital data.

[0301] "Image recognition technology" is a technology that extracts specific information or characters from image information, enabling machines to understand visual information.

[0302] "Business-related information" refers to information obtained through image recognition technology and necessary for the performance of business operations.

[0303] "Natural language processing technology" is a technology that analyzes character information, understands its meaning, and performs business generation and prioritization.

[0304] "Business" refers to specific operations or tasks generated based on voice information and image information.

[0305] "Priority" is an indicator showing the urgency and importance of execution for the generated business, and it contributes to the efficiency improvement of business management.

[0306] "Information processing terminal" is a device that receives business information sent from a server and visually presents it to a user.

[0307] "Notification function" is a mechanism that notifies a user of important business and deadlines via an information processing terminal.

[0308] "User" refers to a person who uses this system and is the entity that manages and performs business operations.

[0309] "Business list" is a list-form display on an information processing terminal that shows the specific content of the generated business.

[0310] <000^0977>"Generative AI technology" is a technology that automatically generates and adjusts business using artificial intelligence to improve efficiency.

[0311] The present invention is a system that utilizes voice information and image information to automatically generate and manage business, aiming to improve the business efficiency of users. This system is realized through the interaction of a server, a terminal, and a user.

[0312] Embodiment of the server

[0313] The server receives voice information sent from the user and converts it into text using speech recognition technology. Specifically, tools such as the Google Cloud Speech-to-Text API could be used. For image information, image recognition technologies such as OpenCV and the Google Cloud Vision API are used to extract business-related information. Furthermore, natural language processing technologies such as spaCy and the Google Cloud Natural Language API are used to analyze and generate business information from the text information and assign priorities. This generated business information is stored in a database and sent to the terminal.

[0314] Terminal embodiment

[0315] The terminal receives task information sent from the server. The received tasks are displayed visually to the user, clearly showing their content, deadline, and priority. The terminal also supports the server's notification function, providing users with timely reminders and preventing tasks from being overlooked.

[0316] User interaction

[0317] Users perform their daily tasks by referring to work information displayed on their terminals. The tasks are displayed visually in a list format, allowing users to operate them intuitively. Furthermore, users can edit, add, and adjust the priority of tasks via the terminal. This enables users to flexibly manage their own schedules.

[0318] Specific example

[0319] For example, if a voice message is made during a meeting stating, "Submit the report by next Monday," the user's device sends this message to the server. The server uses speech recognition technology to transcribe the message into text and then uses natural language processing to generate the task "Submit Report." This task is prioritized with a deadline of "next Monday" and sent to the device. The user can then review the task displayed on their device and manage or adjust tasks as needed.

[0320] Example of a prompt

[0321] "Please describe a system that automatically generates and manages tasks based on comments made during meetings."

[0322] As described above, by combining these technologies, it is possible to reduce the burden on users in managing their work and improve operational efficiency.

[0323] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0324] Step 1:

[0325] The server receives audio information from the user. This audio information is converted into text using speech recognition technology. The audio data, as input, is processed using the Google Cloud Speech-to-Text API. This extracts the spoken content as text. As a concrete example, during a meeting, the user sends the audio message "The project deadline is next Monday" to the server.

[0326] Step 2:

[0327] The server receives image information from the user. Image recognition technology is used to extract business-related information from the image data. The input image data is processed using OpenCV or the Google Cloud Vision API, and the output is the content of a whiteboard or handwritten note converted into text. As a specific example, the user sends a photo of a whiteboard with the text "Marketing Strategy Meeting" written on it to the server.

[0328] Step 3:

[0329] The server analyzes textual information using natural language processing (NLP) technology to generate task content. The textual information used as input is processed by NLP tools such as spaCy. Through analysis, the task content and deadline are identified, and the task is generated. For example, the textual information "The project deadline is next Monday" becomes the task "Project Deadline," and a deadline is set.

[0330] Step 4:

[0331] The server prioritizes the generated tasks. Using a generation AI model, it evaluates the importance and urgency of the tasks and determines their priority. The output is a list of tasks with assigned priorities. Specifically, tasks with "project deadlines" are compared with other tasks, and their priority is determined.

[0332] Step 5:

[0333] The server sends prioritized task information to the terminal. The information processing terminal receives the task details and presents them visually to the user. The task data, as input, is sent from the server and displayed as a task list on the terminal's display. Specifically, the smartphone displays "Project Deadline (High Priority)".

[0334] Step 6:

[0335] Users check the work information displayed on their terminal and perform their tasks. Users can edit, add, and adjust the priority of tasks. Through these operations, they can adjust deadlines and add new tasks as needed. Specifically, users can change the deadline for a "project deadline" and add related tasks, among other adjustments.

[0336] (Application Example 1)

[0337] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0338] In modern factories, improving work efficiency is crucial, but manually transcribing voice instructions and whiteboard information into text and managing work instructions is time-consuming and laborious. Furthermore, delays and errors can occur due to slow real-time updates of work instructions and inefficient transmission of instructions to automated equipment. Additionally, it is essential to ensure that important work instructions are reliably communicated to workers with attention deficits. Improving these conditions and streamlining work management is necessary.

[0339] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0340] In this invention, the server includes means for receiving acoustic information and converting it into document information through language processing, means for receiving visual information and extracting work-related information using object recognition technology, and means for generating work instructions from the document information and assigning priorities using natural language processing technology. This makes it possible to quickly generate work instructions from voice instructions and visual information and transmit and notify automated devices in real time. This improves work efficiency and enables appropriate instruction notification to workers with attention impairments.

[0341] "Acoustic information" refers to all data related to voice and sound, including the content of voice instructions.

[0342] "Language processing" refers to the technology of converting audio data into natural language, generating document information from acoustic information.

[0343] "Document information" refers to converted text data, which is information that has been transcribed from spoken instructions into text.

[0344] "Visual information" refers to all data related to images and videos, including information written on whiteboards or paper.

[0345] "Object recognition technology" refers to technology that automatically identifies and extracts specific information from visual information, and is a means of obtaining work-related information.

[0346] "Work-related information" refers to information about the content and methods of performing a task, and is extracted from visual information.

[0347] "Natural language processing technology" refers to the technology that generates work instructions from document information and sets priorities for those instructions.

[0348] A "work instruction" refers to an instruction that clearly specifies the tasks to be performed, and is sent to a terminal or automated device as a generated task.

[0349] "Priority" refers to an indicator that determines the urgency and importance of carrying out a work instruction.

[0350] "Automated equipment" refers to machines and robots that operate in factories and production lines based on work instructions.

[0351] "Notification" refers to a means of informing users or automated equipment of generated work instructions, and includes providing information in real time.

[0352] The system implementing this invention is designed to improve the efficiency of work instructions in factories and production lines. This invention utilizes acoustic and visual information to instantly generate work instructions and notify automated equipment and users.

[0353] The server uses smart devices (e.g., smartphones or smart glasses) equipped with high-sensitivity microphones to receive acoustic information. The server then converts this acoustic information into document information using speech recognition software such as the Google Speech-to-Text API. Visual information is similarly acquired by devices equipped with cameras, and task-related information is extracted using object recognition techniques such as OpenCV or TensorFlow.

[0354] Furthermore, the server uses natural language processing technology to generate work instructions from document information and assigns priorities using generative AI models such as BERT. These generated work instructions are sent to automated equipment in real time, improving the efficiency of factory operations.

[0355] For example, if a worker says, "Prepare for shipment of product A by 2 PM," the system receives this voice command and generates a work instruction called "Prepare for shipment of product A." At the same time, if "Ship 10 units of product B" is written on a whiteboard, the system acquires this as visual information, generates an instruction called "Ship product B," and executes them sequentially based on priority.

[0356] For example, a prompt could be: "Prepare product A for shipment by 2 PM. Then ship 10 units of product B." Based on this prompt, the system efficiently generates work instructions and sends them to the automated equipment.

[0357] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0358] Step 1:

[0359] The server receives acoustic information through the microphone of a smart device. This acoustic information is converted into document information using the Google Speech-to-Text API. The acoustic information is input as audio data, and the converted document information is output as text data.

[0360] Step 2:

[0361] The server receives visual information using the camera of a smart device. The received visual information is analyzed using object recognition technology with OpenCV or TensorFlow, and task-related information is extracted. The visual information is input as image data, and the task-related information is output as text data.

[0362] Step 3:

[0363] The server analyzes document information and work-related information using natural language processing technology, specifically generative AI models such as BERT, to generate work instructions and assign priorities. Document information and work-related information are input, and the generated work instructions are output as text data.

[0364] Step 4:

[0365] The server sends the generated work instructions to the automated equipment. Information is transmitted in real time, and the automated equipment is configured to operate based on the instructions. The work instructions are provided as input and output as executable commands.

[0366] Step 5:

[0367] Users can review the work instructions displayed on the terminal and make corrections or approvals as needed. The terminal displays the entered instructions on the screen, accepts user input, and sends the updated instructions to the server.

[0368] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0369] This invention combines an emotion engine with a system that automatically generates and manages tasks from voice and image data. This enables task management that takes into account the user's emotional state, thereby improving work efficiency. The system's configuration and operation are described below in detail in the following specific embodiments.

[0370] Server Embodiment

[0371] The server converts the audio data received from the terminal into text using speech recognition technology, and also analyzes the user's emotions from the audio using an emotion engine. The resulting text data is then analyzed using natural language processing technology, and tasks are automatically generated. The generated tasks are prioritized based on the emotion data. For example, if the user is feeling stressed, the priority of the task is temporarily set lower to reduce the burden.

[0372] The server also receives image data and extracts task-related information using image recognition technology. Furthermore, it utilizes the results of sentiment analysis to send users not only appropriate reminders but also encouraging messages when necessary.

[0373] Terminal embodiment

[0374] The device receives task and emotion-based notifications sent from the server and displays them to the user. Tasks are organized by priority, making them easy for the user to review and manage. Depending on the emotional state, the device displays timely reminders, work-related suggestions, and encouraging messages.

[0375] User interaction

[0376] Users perform tasks based on a task list displayed on their device. Additional tasks or changes can be easily updated via the device. The device provides notifications that take sentiment data into account, allowing users to efficiently complete tasks while receiving emotion-based feedback.

[0377] Specific example

[0378] If a user verbally states, "I'll finish the presentation materials for next week," while preparing for a presentation, the device sends this audio data to the server. The server uses speech recognition to transcribe the data into text and an emotion engine to analyze the user's emotions regarding the presentation. If the emotion "nervousness" is detected, the server generates a task, adjusts the priority of related tasks to a lower level, and sends it to the device. In addition to the task list, the device displays a message to the user suggesting ways to relax.

[0379] In this way, the present invention provides a system that reduces the psychological burden on users and improves work efficiency by combining emotion analysis with task management. In particular, for users with attention disorders, work support can be enhanced by sending notifications that are sensitive to their emotions.

[0380] The following describes the processing flow.

[0381] Step 1:

[0382] The device captures the user's speech using a microphone and records it as audio data. Furthermore, it also records additional data such as tone and speed of voice to take into account the user's emotions.

[0383] Step 2:

[0384] The device sends audio data to the server. During this process, metadata, including the time and location of the capture, is added to the audio data.

[0385] Step 3:

[0386] The server converts the received audio data into text data using speech recognition technology. This allows the user's spoken content to be obtained as text information.

[0387] Step 4:

[0388] The server analyzes the voice data using an emotion engine to estimate the user's emotional state. For example, it analyzes the tone and speed of the voice to detect emotions such as tension and stress.

[0389] Step 5:

[0390] The server analyzes the transcribed data using natural language processing technology and automatically generates related tasks. It identifies the content and deadlines of tasks spoken aloud and registers them in the system.

[0391] Step 6:

[0392] The server adjusts the priority of generated tasks, taking emotional data into consideration. If a user is experiencing stress, it will lower the priority of non-urgent tasks, for example.

[0393] Step 7:

[0394] The server sends task information to the terminal. It also generates and transmits emotionally-based messages of encouragement and suggestions to reduce the burden.

[0395] Step 8:

[0396] The device displays a task list and notifications based on sentiment analysis to the user. The task list is organized by priority, and encouraging messages are displayed alongside it.

[0397] Step 9:

[0398] Users can view tasks on their devices and edit or add actions as needed. Emotion-responsive feedback allows for more effective task management.

[0399] Step 10:

[0400] The device feeds back task progress and new updates to the server. This improves the accuracy of task management and the quality of services based on user sentiment data.

[0401] (Example 2)

[0402] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0403] Traditional task management systems process and manage tasks without considering the user's emotional state, which can lead to psychological burden and decreased work efficiency. Furthermore, users with attention deficits require particularly considerate notification methods. Therefore, there is a need for a system that can adjust task priorities according to the user's emotional state, enabling efficient and less burdensome task management.

[0404] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0405] In this invention, the server includes means for receiving audio data and converting it into text information using acoustic analysis technology, means for receiving image information and extracting work-related information using image analysis technology, and means for evaluating the user's emotional state using emotion analysis technology and adjusting the priority of tasks based on that state. This enables flexible task management based on the user's emotional state.

[0406] "Acoustic analysis technology" is a technology that analyzes audio data as a digital signal and converts it into textual information.

[0407] "Image analysis technology" is a technique that processes image information as digital data and extracts meaningful information and patterns.

[0408] "Natural language processing technology" is a technology that enables computers to understand, interpret, and generate human language.

[0409] "Emotional analysis technology" is a technology that evaluates and analyzes a user's emotional state from data such as voice and text.

[0410] "User terminal" refers to an electronic device used by a user to view and edit various notifications and information.

[0411] A "task list" refers to a collection of tasks generated by the system, organized by priority and content.

[0412] "Attention deficit" refers to a condition characterized by difficulty in sustaining attention, and conventional methods of treatment are often complex or insufficient.

[0413] This invention is a system that analyzes voice and image information along with the user's emotional state to effectively manage tasks. Specific embodiments of this system are described below.

[0414] Server Embodiment

[0415] The server receives audio data from the terminal and converts it into text data using acoustic analysis technology. Specifically, it utilizes general-purpose audio analysis software. It also receives image information and extracts task-related information using image analysis technology. General-purpose image analysis software is used for this process. Next, the server uses sentiment analysis technology to evaluate the user's emotions from the audio and text data and adjusts the priority of the tasks generated based on that evaluation. Sentiment analysis software is used for this process. The generated tasks and associated messages are sent to the user's terminal, and the user is notified.

[0416] Terminal embodiment

[0417] The terminal receives task information and messages sent from the server and displays them to the user. This allows the user to manage tasks according to their own status. The terminal can also edit tasks based on user actions, allowing the user to easily add new tasks or modify existing ones.

[0418] User interaction

[0419] Users proceed with their tasks based on a list of tasks displayed on their device. Furthermore, if improvements are needed, they can receive suggestions for task modifications or emotionally-driven suggestions via their device. This allows users to efficiently perform tasks in a way that suits their emotional state.

[0420] Specific example

[0421] For example, if a user says, "I'll finish preparing for next week's meeting," the server converts this audio into text and, through sentiment analysis, determines that the user is feeling "stressed." The server uses this information to adjust the priority of related tasks and provides the user with tasks that include messages suggesting relaxation. This allows the user to work while taking their emotions into consideration.

[0422] An example of a prompt message is: "Convert this audio data to text, analyze the user's emotions, and generate optimal task management suggestions."

[0423] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0424] Step 1:

[0425] The server receives audio data from the terminal. This audio data contains information in which the user has given linguistic instructions to the system. The server converts the audio data into text data using acoustic analysis technology. In this process, speech analysis software is used to analyze the audio waveform and generate a string of characters. The output is text data converted by speech recognition.

[0426] Step 2:

[0427] The server uses text obtained from audio data as input and evaluates the user's emotional state using sentiment analysis technology. In this step, sentiment analysis software is used to analyze emotions such as "tension" or "joy" from factors such as sentence tone and language choice. The output is tag information for the emotions the user is presumed to be feeling.

[0428] Step 3:

[0429] The server uses natural language processing techniques to generate tasks based on the text data obtained in the previous step. Furthermore, it prioritizes the tasks, taking into account the sentiment information obtained in step 2. Specifically, it utilizes a generative AI model to determine tasks based on user requests and their associated priorities. The output is task information prioritized based on sentiment.

[0430] Step 4:

[0431] The server also processes image information received from terminals. Here, image analysis techniques are used to extract task-related information from the image data. For example, key points from meeting slides or documents are identified and incorporated into the task. The output is supplementary information that can be used for task generation.

[0432] Step 5:

[0433] The server sends the generated task information and appropriate messages to the user's terminal. These messages include reminders and suggestions based on the user's emotions. As output, the terminal displays a task summary and an emotionally sensitive message to the user.

[0434] Step 6:

[0435] The terminal displays received task information and messages to the user. The user can check the task list on the terminal and perform tasks based on it. Tasks can also be edited as needed. Input is data from the server, and output is the content displayed on the user interface.

[0436] (Application Example 2)

[0437] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0438] To address the diverse tasks and problems citizens face in their daily lives, efficient task management that takes emotional states into account is required. However, conventional task management systems struggle to reflect users' emotional states in order to appropriately prioritize tasks and make suggestions that contribute to stress reduction, potentially increasing citizens' stress levels. Addressing these challenges is essential.

[0439] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0440] In this invention, the server includes means for receiving voice data and converting it into text data using voice recognition technology, means for analyzing emotions from the voice using an emotion engine and adjusting priorities based on the emotional state, and means for sending the generated tasks to a terminal and setting emotionally sensitive reminders and suggestion messages. This enables optimal task management and stress reduction suggestions according to the emotional state of citizens.

[0441] "Audio data" refers to audio information recorded in digital format and processed by a device.

[0442] "Speech recognition technology" is a technology that converts input speech into text and then understands and analyzes its content.

[0443] "Text data" refers to character information converted using technologies such as speech recognition, and is digital data that can be analyzed by a system.

[0444] "Image data" refers to visual information recorded in a digital format that can be processed mechanically.

[0445] "Image recognition technology" is a technique that extracts and analyzes specific information or patterns from image data.

[0446] "Information" refers to data and knowledge necessary to achieve a specific purpose.

[0447] An "emotion engine" is an algorithm or technology that analyzes, quantifies, and evaluates a user's emotional state from voice and text data.

[0448] "Priority" refers to the criteria used to determine the order in which tasks or processes should be performed, and is a sequence set based on importance and urgency.

[0449] "Natural language processing technology" is a general term encompassing technologies for handling human language on computers, including semantic analysis and information extraction.

[0450] A "task" refers to a series of tasks or activities performed in order to achieve a specific goal.

[0451] A "reminder" is a feature that notifies you to ensure you don't forget important tasks or events.

[0452] A "suggestion message" is a message that includes advice or guidance provided to encourage users to take a specific action or think a particular way.

[0453] A "terminal" refers to a device used by a user to access and operate a system.

[0454] A "task list" refers to a list of tasks organized for the user to manage and review.

[0455] "Editing" refers to the process of changing or updating existing data or information.

[0456] In this system, the server uses speech recognition technology to convert citizens' voice data into text data, and then analyzes that text data using natural language processing technology to generate appropriate tasks. It also uses an emotion engine to perform sentiment analysis and adjust task priorities based on the user's emotional state. Based on the sentiment analysis results, it generates suggestion messages and reminders to reduce stress. Tasks and messages sent from the server are displayed on the terminal, allowing the user to review them and edit tasks as needed.

[0457] The specific processing involves using technologies such as the Google Cloud Speech-to-Text API for speech recognition and spaCy and BERT models for natural language processing. Sentiment analysis is performed using APIs such as AWS Comprehend and Microsoft Azure Text Analytics. By combining these, the server can manage tasks in real time, taking into account the user's voice and emotions, providing personalized support to citizens of the smart city.

[0458] For example, if a citizen uses their smartphone to voice-input "I want to solve the park litter problem," this voice data is converted to text and analyzed on a server. If the emotional state is determined to be "dissatisfied," the system will prioritize the task and suggest contacting the relevant authorities immediately. The user will also be notified with a message such as "We will address this promptly," providing reassurance. An example of a prompt in this case would be: "A citizen's request to solve the park litter problem. We will consider the emotional state and suggest the most appropriate response."

[0459] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0460] Step 1:

[0461] The user provides voice input to the device. Speech recognition technology (Google Cloud Speech-to-Text API) is used to convert the voice data into text data. The input is voice data, and the output is text data. During this process, appropriate formatting is performed as the voice signal is converted into text.

[0462] Step 2:

[0463] The server analyzes the generated text data using natural language processing techniques (such as spaCy and the BERT model) and extracts information. The input is the text data obtained in step 1, and the output is the analyzed information such as meaning and keywords. Various natural language processing techniques, including text structuring and semantic analysis, are used for this data processing.

[0464] Step 3:

[0465] The server uses an emotion engine (such as AWS Comprehend or Microsoft Azure Text Analytics) to analyze user emotions from text data. The input is the text data obtained in step 1, and the output is data that quantifies the user's emotional state. In this data calculation, emotion indicators are extracted, and the user's psychological state is evaluated.

[0466] Step 4:

[0467] The server generates tasks and sets priorities based on the sentiment analysis results. The input is the information obtained in steps 2 and 3, and the output is task data with sentiment-based priorities. This process uses a generative AI model to provide personalized task suggestions to the user.

[0468] Step 5:

[0469] The server sends generated tasks and suggestion messages to the terminal, including emotionally sensitive reminders. The input is the task data obtained in step 4, and the output is information to the user in notification format.

[0470] Step 6:

[0471] The user reviews tasks and messages received on their device and edits the task list as needed. The input is information from step 5, and the output is the updated task list. In this step, tasks are fine-tuned through user interaction.

[0472] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0473] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0474] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0475] [Third Embodiment]

[0476] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0477] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0478] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0479] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0480] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0481] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0482] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0483] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0484] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0485] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0486] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0487] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0488] This invention is a system that automatically generates and manages tasks from voice and images, aiming to improve the user's work efficiency. This system is implemented through interaction between a server, a terminal, and the user.

[0489] Server Embodiment

[0490] The server converts received audio data into text data using speech recognition technology. This makes it possible to transcribe speech and instructions during meetings into text. It also uses image recognition technology to extract task-related information from whiteboards and handwritten notes. Furthermore, it uses natural language processing technology to analyze the information obtained from audio and images and automatically generate tasks. The generated tasks are assigned priorities, and the server registers them in a database and sends them to the terminal.

[0491] Terminal embodiment

[0492] The terminal receives task data sent from the server and displays it to the user. The specific task details, deadlines, and priorities are visually displayed, allowing the user to plan their daily work based on this information. Furthermore, the terminal receives reminder notifications from the server and informs the user at the appropriate time.

[0493] User interaction

[0494] Users perform tasks based on those displayed on their device. The task list is intuitive and easy to understand, allowing users to quickly grasp what needs to be done next. Users can also edit and add tasks via their device, and adjust task priorities and deadlines as needed.

[0495] Specific example

[0496] If, during a meeting, the task "Submit the project plan by next Monday" is spoken aloud, the device sends the audio to the server. The server uses speech recognition to transcribe it into text, recognizing it as the task "Submit the project plan." Further, natural language processing technology analyzes the deadline as "next Monday," generates a task based on this, and sends it to the device. The user can then check the task displayed on the device and add it to their schedule. A reminder function automatically notifies the user as the deadline approaches, preventing tasks from being missed.

[0497] Thus, the present invention provides a system that can efficiently manage user tasks by utilizing voice and image data. In particular, for users with attention disorders, the automatic task generation and reminder functions contribute to supporting their work.

[0498] The following describes the processing flow.

[0499] Step 1:

[0500] The device uses a microphone and camera to capture audio and image data. It acquires audio of what the user says during a meeting and content written on a whiteboard, and prepares to send that data to a server.

[0501] Step 2:

[0502] The server receives audio data transmitted from the terminal. Using a speech recognition engine, this audio data is converted into text data, and the content of the meeting is obtained as written information.

[0503] Step 3:

[0504] The server also receives image data simultaneously. An image recognition algorithm is applied to extract text information from the image. This converts the content written on the whiteboard or in handwritten notes into text.

[0505] Step 4:

[0506] The server analyzes text information obtained from audio and image data using natural language processing techniques. It identifies information that should be managed as tasks and registers them in the database as structured tasks.

[0507] Step 5:

[0508] The server prioritizes the generated tasks. Priorities are automatically set based on the importance and deadline of each task.

[0509] Step 6:

[0510] The server sends prioritized tasks to the terminal. In addition, it sets appropriate reminders and prepares to send notifications at the appropriate time.

[0511] Step 7:

[0512] The terminal receives task information sent from the server and displays it to the user as a task list. The task details, priority, and deadline are visually displayed.

[0513] Step 8:

[0514] Users review tasks displayed on their devices and edit or add tasks as needed. They adjust task priorities and deadlines and send updated information to the server via their devices.

[0515] Step 9:

[0516] The device receives pre-set reminders and notifies the user at the appropriate time, ensuring that the user does not miss important tasks.

[0517] (Example 1)

[0518] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0519] In today's workplaces and work environments, information is diverse and its management is complex. In particular, audio and image information, if not processed quickly and accurately, can lead to decreased work efficiency and information leaks. Furthermore, for users with attention deficits, prioritizing tasks and optimizing important notifications are crucial. To effectively address these challenges, there is a need for systems that utilize audio and image to automate work management and improve work efficiency.

[0520] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0521] In this invention, the server includes means for receiving audio information and converting it into text information using speech recognition technology, means for receiving image information and extracting business-related information using image recognition technology, and means for generating tasks from the text information and prioritizing them using natural language processing technology. This enables rapid and accurate processing of audio and image information, automatic generation and prioritization of tasks, prevention of leakage of important information, and improvement of user work efficiency.

[0522] "Audio information" refers to data that represents human speech as digital signals, and is the subject of speech recognition technology.

[0523] "Speech recognition technology" is a technology that receives audio information as input and converts it into text information.

[0524] "Textual information" refers to data in text format converted from audio information using speech recognition technology.

[0525] "Image information" refers to providing visual information, such as whiteboards or handwritten notes, as digital data.

[0526] "Image recognition technology" is a technology that extracts specific information or characters from image information, enabling machines to understand visual information.

[0527] "Business-related information" refers to information necessary for carrying out business operations, obtained through image recognition technology.

[0528] "Natural language processing technology" is a technology that analyzes textual information, understands its meaning, and uses it to generate and prioritize tasks.

[0529] "Work" refers to specific tasks or operations generated based on audio and image information.

[0530] "Priority" is an indicator that shows the urgency and importance of executing a generated task, and contributes to the efficiency of task management.

[0531] An "information processing terminal" is a device that receives business information transmitted from a server and presents it visually to the user.

[0532] A "notification function" is a system that informs users of important tasks and deadlines via their information processing terminals.

[0533] "Users" refer to the individuals who use this system and are the entities responsible for managing and carrying out their tasks.

[0534] A "task list" is a list-format display shown on an information processing terminal that shows the specific details of the tasks that have been generated.

[0535] "Generative AI technology" is a technology that uses artificial intelligence to automatically generate and adjust tasks, thereby improving efficiency.

[0536] This invention is a system that automatically generates and manages tasks using voice and image information, with the aim of improving the user's work efficiency. This system is realized through the interaction of a server, terminals, and users.

[0537] Server Embodiment

[0538] The server receives voice information sent from the user and converts it into text using speech recognition technology. Specifically, tools such as the Google Cloud Speech-to-Text API could be used. For image information, image recognition technologies such as OpenCV and the Google Cloud Vision API are used to extract business-related information. Furthermore, natural language processing technologies such as spaCy and the Google Cloud Natural Language API are used to analyze and generate business information from the text information and assign priorities. This generated business information is stored in a database and sent to the terminal.

[0539] Terminal embodiment

[0540] The terminal receives task information sent from the server. The received tasks are displayed visually to the user, clearly showing their content, deadline, and priority. The terminal also supports the server's notification function, providing users with timely reminders and preventing tasks from being overlooked.

[0541] User interaction

[0542] Users perform their daily tasks by referring to work information displayed on their terminals. The tasks are displayed visually in a list format, allowing users to operate them intuitively. Furthermore, users can edit, add, and adjust the priority of tasks via the terminal. This enables users to flexibly manage their own schedules.

[0543] Specific example

[0544] For example, if a voice message is made during a meeting stating, "Submit the report by next Monday," the user's device sends this message to the server. The server uses speech recognition technology to transcribe the message into text and then uses natural language processing to generate the task "Submit Report." This task is prioritized with a deadline of "next Monday" and sent to the device. The user can then review the task displayed on their device and manage or adjust tasks as needed.

[0545] Example of a prompt

[0546] "Please describe a system that automatically generates and manages tasks based on comments made during meetings."

[0547] As described above, by combining these technologies, it is possible to reduce the burden on users in managing their work and improve operational efficiency.

[0548] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0549] Step 1:

[0550] The server receives audio information from the user. This audio information is converted into text using speech recognition technology. The audio data, as input, is processed using the Google Cloud Speech-to-Text API. This extracts the spoken content as text. As a concrete example, during a meeting, the user sends the audio message "The project deadline is next Monday" to the server.

[0551] Step 2:

[0552] The server receives image information from the user. Image recognition technology is used to extract business-related information from the image data. The input image data is processed using OpenCV or the Google Cloud Vision API, and the output is the content of a whiteboard or handwritten note converted into text. As a specific example, the user sends a photo of a whiteboard with the text "Marketing Strategy Meeting" written on it to the server.

[0553] Step 3:

[0554] The server analyzes textual information using natural language processing (NLP) technology to generate task content. The textual information used as input is processed by NLP tools such as spaCy. Through analysis, the task content and deadline are identified, and the task is generated. For example, the textual information "The project deadline is next Monday" becomes the task "Project Deadline," and a deadline is set.

[0555] Step 4:

[0556] The server prioritizes the generated tasks. Using a generation AI model, it evaluates the importance and urgency of the tasks and determines their priority. The output is a list of tasks with assigned priorities. Specifically, tasks with "project deadlines" are compared with other tasks, and their priority is determined.

[0557] Step 5:

[0558] The server sends prioritized task information to the terminal. The information processing terminal receives the task details and presents them visually to the user. The task data, as input, is sent from the server and displayed as a task list on the terminal's display. Specifically, the smartphone displays "Project Deadline (High Priority)".

[0559] Step 6:

[0560] Users check the work information displayed on their terminal and perform their tasks. Users can edit, add, and adjust the priority of tasks. Through these operations, they can adjust deadlines and add new tasks as needed. Specifically, users can change the deadline for a "project deadline" and add related tasks, among other adjustments.

[0561] (Application Example 1)

[0562] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0563] In modern factories, improving work efficiency is crucial, but manually transcribing voice instructions and whiteboard information into text and managing work instructions is time-consuming and laborious. Furthermore, delays and errors can occur due to slow real-time updates of work instructions and inefficient transmission of instructions to automated equipment. Additionally, it is essential to ensure that important work instructions are reliably communicated to workers with attention deficits. Improving these conditions and streamlining work management is necessary.

[0564] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0565] In this invention, the server includes means for receiving acoustic information and converting it into document information through language processing, means for receiving visual information and extracting work-related information using object recognition technology, and means for generating work instructions from the document information and assigning priorities using natural language processing technology. This makes it possible to quickly generate work instructions from voice instructions and visual information and transmit and notify automated devices in real time. This improves work efficiency and enables appropriate instruction notification to workers with attention impairments.

[0566] "Acoustic information" refers to all data related to voice and sound, including the content of voice instructions.

[0567] "Language processing" refers to the technology of converting audio data into natural language, generating document information from acoustic information.

[0568] "Document information" refers to converted text data, which is information that has been transcribed from spoken instructions into text.

[0569] "Visual information" refers to all data related to images and videos, including information written on whiteboards or paper.

[0570] "Object recognition technology" refers to technology that automatically identifies and extracts specific information from visual information, and is a means of obtaining work-related information.

[0571] "Work-related information" refers to information about the content and methods of performing a task, and is extracted from visual information.

[0572] "Natural language processing technology" refers to the technology that generates work instructions from document information and sets priorities for those instructions.

[0573] A "work instruction" refers to an instruction that clearly specifies the tasks to be performed, and is sent to a terminal or automated device as a generated task.

[0574] "Priority" refers to an indicator that determines the urgency and importance of carrying out a work instruction.

[0575] "Automated equipment" refers to machines and robots that operate in factories and production lines based on work instructions.

[0576] "Notification" refers to a means of informing users or automated equipment of generated work instructions, and includes providing information in real time.

[0577] The system implementing this invention is designed to improve the efficiency of work instructions in factories and production lines. This invention utilizes acoustic and visual information to instantly generate work instructions and notify automated equipment and users.

[0578] The server uses smart devices (e.g., smartphones or smart glasses) equipped with high-sensitivity microphones to receive acoustic information. The server then converts this acoustic information into document information using speech recognition software such as the Google Speech-to-Text API. Visual information is similarly acquired by devices equipped with cameras, and task-related information is extracted using object recognition techniques such as OpenCV or TensorFlow.

[0579] Furthermore, the server uses natural language processing technology to generate work instructions from document information and assigns priorities using generative AI models such as BERT. These generated work instructions are sent to automated equipment in real time, improving the efficiency of factory operations.

[0580] For example, if a worker says, "Prepare for shipment of product A by 2 PM," the system receives this voice command and generates a work instruction called "Prepare for shipment of product A." At the same time, if "Ship 10 units of product B" is written on a whiteboard, the system acquires this as visual information, generates an instruction called "Ship product B," and executes them sequentially based on priority.

[0581] For example, a prompt could be: "Prepare product A for shipment by 2 PM. Then ship 10 units of product B." Based on this prompt, the system efficiently generates work instructions and sends them to the automated equipment.

[0582] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0583] Step 1:

[0584] The server receives acoustic information through the microphone of a smart device. This acoustic information is converted into document information using the Google Speech-to-Text API. The acoustic information is input as audio data, and the converted document information is output as text data.

[0585] Step 2:

[0586] The server receives visual information using the camera of a smart device. The received visual information is analyzed using object recognition technology with OpenCV or TensorFlow, and task-related information is extracted. The visual information is input as image data, and the task-related information is output as text data.

[0587] Step 3:

[0588] The server analyzes document information and work-related information using natural language processing technology, specifically generative AI models such as BERT, to generate work instructions and assign priorities. Document information and work-related information are input, and the generated work instructions are output as text data.

[0589] Step 4:

[0590] The server sends the generated work instructions to the automated equipment. Information is transmitted in real time, and the automated equipment is configured to operate based on the instructions. The work instructions are provided as input and output as executable commands.

[0591] Step 5:

[0592] Users can review the work instructions displayed on the terminal and make corrections or approvals as needed. The terminal displays the entered instructions on the screen, accepts user input, and sends the updated instructions to the server.

[0593] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0594] This invention combines an emotion engine with a system that automatically generates and manages tasks from voice and image data. This enables task management that takes into account the user's emotional state, thereby improving work efficiency. The system's configuration and operation are described below in detail in the following specific embodiments.

[0595] Server Embodiment

[0596] The server converts the audio data received from the terminal into text using speech recognition technology, and also analyzes the user's emotions from the audio using an emotion engine. The resulting text data is then analyzed using natural language processing technology, and tasks are automatically generated. The generated tasks are prioritized based on the emotion data. For example, if the user is feeling stressed, the priority of the task is temporarily set lower to reduce the burden.

[0597] The server also receives image data and extracts task-related information using image recognition technology. Furthermore, it utilizes the results of sentiment analysis to send users not only appropriate reminders but also encouraging messages when necessary.

[0598] Terminal embodiment

[0599] The device receives task and emotion-based notifications sent from the server and displays them to the user. Tasks are organized by priority, making them easy for the user to review and manage. Depending on the emotional state, the device displays timely reminders, work-related suggestions, and encouraging messages.

[0600] User interaction

[0601] Users perform tasks based on a task list displayed on their device. Additional tasks or changes can be easily updated via the device. The device provides notifications that take sentiment data into account, allowing users to efficiently complete tasks while receiving emotion-based feedback.

[0602] Specific example

[0603] If a user verbally states, "I'll finish the presentation materials for next week," while preparing for a presentation, the device sends this audio data to the server. The server uses speech recognition to transcribe the data into text and an emotion engine to analyze the user's emotions regarding the presentation. If the emotion "nervousness" is detected, the server generates a task, adjusts the priority of related tasks to a lower level, and sends it to the device. In addition to the task list, the device displays a message to the user suggesting ways to relax.

[0604] In this way, the present invention provides a system that reduces the psychological burden on users and improves work efficiency by combining emotion analysis with task management. In particular, for users with attention disorders, work support can be enhanced by sending notifications that are sensitive to their emotions.

[0605] The following describes the processing flow.

[0606] Step 1:

[0607] The device captures the user's speech using a microphone and records it as audio data. Furthermore, it also records additional data such as tone and speed of voice to take into account the user's emotions.

[0608] Step 2:

[0609] The device sends audio data to the server. During this process, metadata, including the time and location of the capture, is added to the audio data.

[0610] Step 3:

[0611] The server converts the received audio data into text data using speech recognition technology. This allows the user's spoken content to be obtained as text information.

[0612] Step 4:

[0613] The server analyzes the voice data using an emotion engine to estimate the user's emotional state. For example, it analyzes the tone and speed of the voice to detect emotions such as tension and stress.

[0614] Step 5:

[0615] The server analyzes the transcribed data using natural language processing technology and automatically generates related tasks. It identifies the content and deadlines of tasks spoken aloud and registers them in the system.

[0616] Step 6:

[0617] The server adjusts the priority of generated tasks, taking emotional data into consideration. If a user is experiencing stress, it will lower the priority of non-urgent tasks, for example.

[0618] Step 7:

[0619] The server sends task information to the terminal. It also generates and transmits emotionally-based messages of encouragement and suggestions to reduce the burden.

[0620] Step 8:

[0621] The device displays a task list and notifications based on sentiment analysis to the user. The task list is organized by priority, and encouraging messages are displayed alongside it.

[0622] Step 9:

[0623] Users can view tasks on their devices and edit or add actions as needed. Emotion-responsive feedback allows for more effective task management.

[0624] Step 10:

[0625] The device feeds back task progress and new updates to the server. This improves the accuracy of task management and the quality of services based on user sentiment data.

[0626] (Example 2)

[0627] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0628] Traditional task management systems process and manage tasks without considering the user's emotional state, which can lead to psychological burden and decreased work efficiency. Furthermore, users with attention deficits require particularly considerate notification methods. Therefore, there is a need for a system that can adjust task priorities according to the user's emotional state, enabling efficient and less burdensome task management.

[0629] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0630] In this invention, the server includes means for receiving audio data and converting it into text information using acoustic analysis technology, means for receiving image information and extracting work-related information using image analysis technology, and means for evaluating the user's emotional state using emotion analysis technology and adjusting the priority of tasks based on that state. This enables flexible task management based on the user's emotional state.

[0631] "Acoustic analysis technology" is a technology that analyzes audio data as a digital signal and converts it into textual information.

[0632] "Image analysis technology" is a technique that processes image information as digital data and extracts meaningful information and patterns.

[0633] "Natural language processing technology" is a technology that enables computers to understand, interpret, and generate human language.

[0634] "Emotional analysis technology" is a technology that evaluates and analyzes a user's emotional state from data such as voice and text.

[0635] "User terminal" refers to an electronic device used by a user to view and edit various notifications and information.

[0636] A "task list" refers to a collection of tasks generated by the system, organized by priority and content.

[0637] "Attention deficit" refers to a condition characterized by difficulty in sustaining attention, and conventional methods of treatment are often complex or insufficient.

[0638] This invention is a system that analyzes voice and image information along with the user's emotional state to effectively manage tasks. Specific embodiments of this system are described below.

[0639] Server Embodiment

[0640] The server receives audio data from the terminal and converts it into text data using acoustic analysis technology. Specifically, it utilizes general-purpose audio analysis software. It also receives image information and extracts task-related information using image analysis technology. General-purpose image analysis software is used for this process. Next, the server uses sentiment analysis technology to evaluate the user's emotions from the audio and text data and adjusts the priority of the tasks generated based on that evaluation. Sentiment analysis software is used for this process. The generated tasks and associated messages are sent to the user's terminal, and the user is notified.

[0641] Terminal embodiment

[0642] The terminal receives task information and messages sent from the server and displays them to the user. This allows the user to manage tasks according to their own status. The terminal can also edit tasks based on user actions, allowing the user to easily add new tasks or modify existing ones.

[0643] User interaction

[0644] Users proceed with their tasks based on a list of tasks displayed on their device. Furthermore, if improvements are needed, they can receive suggestions for task modifications or emotionally-driven suggestions via their device. This allows users to efficiently perform tasks in a way that suits their emotional state.

[0645] Specific example

[0646] For example, if a user says, "I'll finish preparing for next week's meeting," the server converts this audio into text and, through sentiment analysis, determines that the user is feeling "stressed." The server uses this information to adjust the priority of related tasks and provides the user with tasks that include messages suggesting relaxation. This allows the user to work while taking their emotions into consideration.

[0647] An example of a prompt message is: "Convert this audio data to text, analyze the user's emotions, and generate optimal task management suggestions."

[0648] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0649] Step 1:

[0650] The server receives audio data from the terminal. This audio data contains information in which the user has given linguistic instructions to the system. The server converts the audio data into text data using acoustic analysis technology. In this process, speech analysis software is used to analyze the audio waveform and generate a string of characters. The output is text data converted by speech recognition.

[0651] Step 2:

[0652] The server uses text obtained from audio data as input and evaluates the user's emotional state using sentiment analysis technology. In this step, sentiment analysis software is used to analyze emotions such as "tension" or "joy" from factors such as sentence tone and language choice. The output is tag information for the emotions the user is presumed to be feeling.

[0653] Step 3:

[0654] The server uses natural language processing techniques to generate tasks based on the text data obtained in the previous step. Furthermore, it prioritizes the tasks, taking into account the sentiment information obtained in step 2. Specifically, it utilizes a generative AI model to determine tasks based on user requests and their associated priorities. The output is task information prioritized based on sentiment.

[0655] Step 4:

[0656] The server also processes image information received from terminals. Here, image analysis techniques are used to extract task-related information from the image data. For example, key points from meeting slides or documents are identified and incorporated into the task. The output is supplementary information that can be used for task generation.

[0657] Step 5:

[0658] The server sends the generated task information and appropriate messages to the user's terminal. These messages include reminders and suggestions based on the user's emotions. As output, the terminal displays a task summary and an emotionally sensitive message to the user.

[0659] Step 6:

[0660] The terminal displays received task information and messages to the user. The user can check the task list on the terminal and perform tasks based on it. Tasks can also be edited as needed. Input is data from the server, and output is the content displayed on the user interface.

[0661] (Application Example 2)

[0662] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0663] To address the diverse tasks and problems citizens face in their daily lives, efficient task management that takes emotional states into account is required. However, conventional task management systems struggle to reflect users' emotional states in order to appropriately prioritize tasks and make suggestions that contribute to stress reduction, potentially increasing citizens' stress levels. Addressing these challenges is essential.

[0664] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0665] In this invention, the server includes means for receiving voice data and converting it into text data using voice recognition technology, means for analyzing emotions from the voice using an emotion engine and adjusting priorities based on the emotional state, and means for sending the generated tasks to a terminal and setting emotionally sensitive reminders and suggestion messages. This enables optimal task management and stress reduction suggestions according to the emotional state of citizens.

[0666] "Audio data" refers to audio information recorded in digital format and processed by a device.

[0667] "Speech recognition technology" is a technology that converts input speech into text and then understands and analyzes its content.

[0668] "Text data" refers to character information converted using technologies such as speech recognition, and is digital data that can be analyzed by a system.

[0669] "Image data" refers to visual information recorded in a digital format that can be processed mechanically.

[0670] "Image recognition technology" is a technique that extracts and analyzes specific information or patterns from image data.

[0671] "Information" refers to data and knowledge necessary to achieve a specific purpose.

[0672] An "emotion engine" is an algorithm or technology that analyzes, quantifies, and evaluates a user's emotional state from voice and text data.

[0673] "Priority" refers to the criteria used to determine the order in which tasks or processes should be performed, and is a sequence set based on importance and urgency.

[0674] "Natural language processing technology" is a general term encompassing technologies for handling human language on computers, including semantic analysis and information extraction.

[0675] A "task" refers to a series of tasks or activities performed in order to achieve a specific goal.

[0676] A "reminder" is a feature that notifies you to ensure you don't forget important tasks or events.

[0677] A "suggestion message" is a message that includes advice or guidance provided to encourage users to take a specific action or think a particular way.

[0678] A "terminal" refers to a device used by a user to access and operate a system.

[0679] A "task list" refers to a list of tasks organized for the user to manage and review.

[0680] "Editing" refers to the process of changing or updating existing data or information.

[0681] In this system, the server uses speech recognition technology to convert citizens' voice data into text data, and then analyzes that text data using natural language processing technology to generate appropriate tasks. It also uses an emotion engine to perform sentiment analysis and adjust task priorities based on the user's emotional state. Based on the sentiment analysis results, it generates suggestion messages and reminders to reduce stress. Tasks and messages sent from the server are displayed on the terminal, allowing the user to review them and edit tasks as needed.

[0682] The specific processing involves using technologies such as the Google Cloud Speech-to-Text API for speech recognition and spaCy and BERT models for natural language processing. Sentiment analysis is performed using APIs such as AWS Comprehend and Microsoft Azure Text Analytics. By combining these, the server can manage tasks in real time, taking into account the user's voice and emotions, providing personalized support to citizens of the smart city.

[0683] For example, if a citizen uses their smartphone to voice-input "I want to solve the park litter problem," this voice data is converted to text and analyzed on a server. If the emotional state is determined to be "dissatisfied," the system will prioritize the task and suggest contacting the relevant authorities immediately. The user will also be notified with a message such as "We will address this promptly," providing reassurance. An example of a prompt in this case would be: "A citizen's request to solve the park litter problem. We will consider the emotional state and suggest the most appropriate response."

[0684] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0685] Step 1:

[0686] The user provides voice input to the device. Speech recognition technology (Google Cloud Speech-to-Text API) is used to convert the voice data into text data. The input is voice data, and the output is text data. During this process, appropriate formatting is performed as the voice signal is converted into text.

[0687] Step 2:

[0688] The server analyzes the generated text data using natural language processing techniques (such as spaCy and the BERT model) and extracts information. The input is the text data obtained in step 1, and the output is the analyzed information such as meaning and keywords. Various natural language processing techniques, including text structuring and semantic analysis, are used for this data processing.

[0689] Step 3:

[0690] The server uses an emotion engine (such as AWS Comprehend or Microsoft Azure Text Analytics) to analyze user emotions from text data. The input is the text data obtained in step 1, and the output is data that quantifies the user's emotional state. In this data calculation, emotion indicators are extracted, and the user's psychological state is evaluated.

[0691] Step 4:

[0692] The server generates tasks and sets priorities based on the sentiment analysis results. The input is the information obtained in steps 2 and 3, and the output is task data with sentiment-based priorities. This process uses a generative AI model to provide personalized task suggestions to the user.

[0693] Step 5:

[0694] The server sends generated tasks and suggestion messages to the terminal, including emotionally sensitive reminders. The input is the task data obtained in step 4, and the output is information to the user in notification format.

[0695] Step 6:

[0696] The user reviews tasks and messages received on their device and edits the task list as needed. The input is information from step 5, and the output is the updated task list. In this step, tasks are fine-tuned through user interaction.

[0697] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0698] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0699] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0700] [Fourth Embodiment]

[0701] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0702] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0703] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0704] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0705] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0706] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0707] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0708] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0709] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0710] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0711] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0712] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0713] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0714] This invention is a system that automatically generates and manages tasks from voice and images, aiming to improve the user's work efficiency. This system is implemented through interaction between a server, a terminal, and the user.

[0715] Server Embodiment

[0716] The server converts received audio data into text data using speech recognition technology. This makes it possible to transcribe speech and instructions during meetings into text. It also uses image recognition technology to extract task-related information from whiteboards and handwritten notes. Furthermore, it uses natural language processing technology to analyze the information obtained from audio and images and automatically generate tasks. The generated tasks are assigned priorities, and the server registers them in a database and sends them to the terminal.

[0717] Terminal embodiment

[0718] The terminal receives task data sent from the server and displays it to the user. The specific task details, deadlines, and priorities are visually displayed, allowing the user to plan their daily work based on this information. Furthermore, the terminal receives reminder notifications from the server and informs the user at the appropriate time.

[0719] User interaction

[0720] Users perform tasks based on those displayed on their device. The task list is intuitive and easy to understand, allowing users to quickly grasp what needs to be done next. Users can also edit and add tasks via their device, and adjust task priorities and deadlines as needed.

[0721] Specific example

[0722] If, during a meeting, the task "Submit the project plan by next Monday" is spoken aloud, the device sends the audio to the server. The server uses speech recognition to transcribe it into text, recognizing it as the task "Submit the project plan." Further, natural language processing technology analyzes the deadline as "next Monday," generates a task based on this, and sends it to the device. The user can then check the task displayed on the device and add it to their schedule. A reminder function automatically notifies the user as the deadline approaches, preventing tasks from being missed.

[0723] Thus, the present invention provides a system that can efficiently manage user tasks by utilizing voice and image data. In particular, for users with attention disorders, the automatic task generation and reminder functions contribute to supporting their work.

[0724] The following describes the processing flow.

[0725] Step 1:

[0726] The device uses a microphone and camera to capture audio and image data. It acquires audio of what the user says during a meeting and content written on a whiteboard, and prepares to send that data to a server.

[0727] Step 2:

[0728] The server receives audio data transmitted from the terminal. Using a speech recognition engine, this audio data is converted into text data, and the content of the meeting is obtained as written information.

[0729] Step 3:

[0730] The server also receives image data simultaneously. An image recognition algorithm is applied to extract text information from the image. This converts the content written on the whiteboard or in handwritten notes into text.

[0731] Step 4:

[0732] The server analyzes text information obtained from audio and image data using natural language processing techniques. It identifies information that should be managed as tasks and registers them in the database as structured tasks.

[0733] Step 5:

[0734] The server prioritizes the generated tasks. Priorities are automatically set based on the importance and deadline of each task.

[0735] Step 6:

[0736] The server sends prioritized tasks to the terminal. In addition, it sets appropriate reminders and prepares to send notifications at the appropriate time.

[0737] Step 7:

[0738] The terminal receives task information sent from the server and displays it to the user as a task list. The task details, priority, and deadline are visually displayed.

[0739] Step 8:

[0740] Users review tasks displayed on their devices and edit or add tasks as needed. They adjust task priorities and deadlines and send updated information to the server via their devices.

[0741] Step 9:

[0742] The device receives pre-set reminders and notifies the user at the appropriate time, ensuring that the user does not miss important tasks.

[0743] (Example 1)

[0744] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0745] In today's workplaces and work environments, information is diverse and its management is complex. In particular, audio and image information, if not processed quickly and accurately, can lead to decreased work efficiency and information leaks. Furthermore, for users with attention deficits, prioritizing tasks and optimizing important notifications are crucial. To effectively address these challenges, there is a need for systems that utilize audio and image to automate work management and improve work efficiency.

[0746] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0747] In this invention, the server includes means for receiving audio information and converting it into text information using speech recognition technology, means for receiving image information and extracting business-related information using image recognition technology, and means for generating tasks from the text information and prioritizing them using natural language processing technology. This enables rapid and accurate processing of audio and image information, automatic generation and prioritization of tasks, prevention of leakage of important information, and improvement of user work efficiency.

[0748] "Audio information" refers to data that represents human speech as digital signals, and is the subject of speech recognition technology.

[0749] "Speech recognition technology" is a technology that receives audio information as input and converts it into text information.

[0750] "Textual information" refers to data in text format converted from audio information using speech recognition technology.

[0751] "Image information" refers to providing visual information, such as whiteboards or handwritten notes, as digital data.

[0752] "Image recognition technology" is a technology that extracts specific information or characters from image information, enabling machines to understand visual information.

[0753] "Business-related information" refers to information necessary for carrying out business operations, obtained through image recognition technology.

[0754] "Natural language processing technology" is a technology that analyzes textual information, understands its meaning, and uses it to generate and prioritize tasks.

[0755] "Work" refers to specific tasks or operations generated based on audio and image information.

[0756] "Priority" is an indicator that shows the urgency and importance of executing a generated task, and contributes to the efficiency of task management.

[0757] An "information processing terminal" is a device that receives business information transmitted from a server and presents it visually to the user.

[0758] A "notification function" is a system that informs users of important tasks and deadlines via their information processing terminals.

[0759] "Users" refer to the individuals who use this system and are the entities responsible for managing and carrying out their tasks.

[0760] A "task list" is a list-format display shown on an information processing terminal that shows the specific details of the tasks that have been generated.

[0761] "Generative AI technology" is a technology that uses artificial intelligence to automatically generate and adjust tasks, thereby improving efficiency.

[0762] This invention is a system that automatically generates and manages tasks using voice and image information, with the aim of improving the user's work efficiency. This system is realized through the interaction of a server, terminals, and users.

[0763] Server Embodiment

[0764] The server receives voice information sent from the user and converts it into text using speech recognition technology. Specifically, tools such as the Google Cloud Speech-to-Text API could be used. For image information, image recognition technologies such as OpenCV and the Google Cloud Vision API are used to extract business-related information. Furthermore, natural language processing technologies such as spaCy and the Google Cloud Natural Language API are used to analyze and generate business information from the text information and assign priorities. This generated business information is stored in a database and sent to the terminal.

[0765] Terminal embodiment

[0766] The terminal receives task information sent from the server. The received tasks are displayed visually to the user, clearly showing their content, deadline, and priority. The terminal also supports the server's notification function, providing users with timely reminders and preventing tasks from being overlooked.

[0767] User interaction

[0768] Users perform their daily tasks by referring to work information displayed on their terminals. The tasks are displayed visually in a list format, allowing users to operate them intuitively. Furthermore, users can edit, add, and adjust the priority of tasks via the terminal. This enables users to flexibly manage their own schedules.

[0769] Specific example

[0770] For example, if a voice message is made during a meeting stating, "Submit the report by next Monday," the user's device sends this message to the server. The server uses speech recognition technology to transcribe the message into text and then uses natural language processing to generate the task "Submit Report." This task is prioritized with a deadline of "next Monday" and sent to the device. The user can then review the task displayed on their device and manage or adjust tasks as needed.

[0771] Example of a prompt

[0772] "Please describe a system that automatically generates and manages tasks based on comments made during meetings."

[0773] As described above, by combining these technologies, it is possible to reduce the burden on users in managing their work and improve operational efficiency.

[0774] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0775] Step 1:

[0776] The server receives audio information from the user. This audio information is converted into text using speech recognition technology. The audio data, as input, is processed using the Google Cloud Speech-to-Text API. This extracts the spoken content as text. As a concrete example, during a meeting, the user sends the audio message "The project deadline is next Monday" to the server.

[0777] Step 2:

[0778] The server receives image information from the user. Image recognition technology is used to extract business-related information from the image data. The input image data is processed using OpenCV or the Google Cloud Vision API, and the output is the content of a whiteboard or handwritten note converted into text. As a specific example, the user sends a photo of a whiteboard with the text "Marketing Strategy Meeting" written on it to the server.

[0779] Step 3:

[0780] The server analyzes textual information using natural language processing (NLP) technology to generate task content. The textual information used as input is processed by NLP tools such as spaCy. Through analysis, the task content and deadline are identified, and the task is generated. For example, the textual information "The project deadline is next Monday" becomes the task "Project Deadline," and a deadline is set.

[0781] Step 4:

[0782] The server prioritizes the generated tasks. Using a generation AI model, it evaluates the importance and urgency of the tasks and determines their priority. The output is a list of tasks with assigned priorities. Specifically, tasks with "project deadlines" are compared with other tasks, and their priority is determined.

[0783] Step 5:

[0784] The server sends prioritized task information to the terminal. The information processing terminal receives the task details and presents them visually to the user. The task data, as input, is sent from the server and displayed as a task list on the terminal's display. Specifically, the smartphone displays "Project Deadline (High Priority)".

[0785] Step 6:

[0786] Users check the work information displayed on their terminal and perform their tasks. Users can edit, add, and adjust the priority of tasks. Through these operations, they can adjust deadlines and add new tasks as needed. Specifically, users can change the deadline for a "project deadline" and add related tasks, among other adjustments.

[0787] (Application Example 1)

[0788] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0789] In modern factories, improving work efficiency is crucial, but manually transcribing voice instructions and whiteboard information into text and managing work instructions is time-consuming and laborious. Furthermore, delays and errors can occur due to slow real-time updates of work instructions and inefficient transmission of instructions to automated equipment. Additionally, it is essential to ensure that important work instructions are reliably communicated to workers with attention deficits. Improving these conditions and streamlining work management is necessary.

[0790] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0791] In this invention, the server includes means for receiving acoustic information and converting it into document information through language processing, means for receiving visual information and extracting work-related information using object recognition technology, and means for generating work instructions from the document information and assigning priorities using natural language processing technology. This makes it possible to quickly generate work instructions from voice instructions and visual information and transmit and notify automated devices in real time. This improves work efficiency and enables appropriate instruction notification to workers with attention impairments.

[0792] "Acoustic information" refers to all data related to voice and sound, including the content of voice instructions.

[0793] "Language processing" refers to the technology of converting audio data into natural language, generating document information from acoustic information.

[0794] "Document information" refers to converted text data, which is information that has been transcribed from spoken instructions into text.

[0795] "Visual information" refers to all data related to images and videos, including information written on whiteboards or paper.

[0796] "Object recognition technology" refers to technology that automatically identifies and extracts specific information from visual information, and is a means of obtaining work-related information.

[0797] "Work-related information" refers to information about the content and methods of performing a task, and is extracted from visual information.

[0798] "Natural language processing technology" refers to the technology that generates work instructions from document information and sets priorities for those instructions.

[0799] A "work instruction" refers to an instruction that clearly specifies the tasks to be performed, and is sent to a terminal or automated device as a generated task.

[0800] "Priority" refers to an indicator that determines the urgency and importance of carrying out a work instruction.

[0801] "Automated equipment" refers to machines and robots that operate in factories and production lines based on work instructions.

[0802] "Notification" refers to a means of informing users or automated equipment of generated work instructions, and includes providing information in real time.

[0803] The system implementing this invention is designed to improve the efficiency of work instructions in factories and production lines. This invention utilizes acoustic and visual information to instantly generate work instructions and notify automated equipment and users.

[0804] The server uses smart devices (e.g., smartphones or smart glasses) equipped with high-sensitivity microphones to receive acoustic information. The server then converts this acoustic information into document information using speech recognition software such as the Google Speech-to-Text API. Visual information is similarly acquired by devices equipped with cameras, and task-related information is extracted using object recognition techniques such as OpenCV or TensorFlow.

[0805] Furthermore, the server uses natural language processing technology to generate work instructions from document information and assigns priorities using generative AI models such as BERT. These generated work instructions are sent to automated equipment in real time, improving the efficiency of factory operations.

[0806] For example, if a worker says, "Prepare for shipment of product A by 2 PM," the system receives this voice command and generates a work instruction called "Prepare for shipment of product A." At the same time, if "Ship 10 units of product B" is written on a whiteboard, the system acquires this as visual information, generates an instruction called "Ship product B," and executes them sequentially based on priority.

[0807] For example, a prompt could be: "Prepare product A for shipment by 2 PM. Then ship 10 units of product B." Based on this prompt, the system efficiently generates work instructions and sends them to the automated equipment.

[0808] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0809] Step 1:

[0810] The server receives acoustic information through the microphone of a smart device. This acoustic information is converted into document information using the Google Speech-to-Text API. The acoustic information is input as audio data, and the converted document information is output as text data.

[0811] Step 2:

[0812] The server receives visual information using the camera of a smart device. The received visual information is analyzed using object recognition technology with OpenCV or TensorFlow, and task-related information is extracted. The visual information is input as image data, and the task-related information is output as text data.

[0813] Step 3:

[0814] The server analyzes document information and work-related information using natural language processing technology, specifically generative AI models such as BERT, to generate work instructions and assign priorities. Document information and work-related information are input, and the generated work instructions are output as text data.

[0815] Step 4:

[0816] The server sends the generated work instructions to the automated equipment. Information is transmitted in real time, and the automated equipment is configured to operate based on the instructions. The work instructions are provided as input and output as executable commands.

[0817] Step 5:

[0818] Users can review the work instructions displayed on the terminal and make corrections or approvals as needed. The terminal displays the entered instructions on the screen, accepts user input, and sends the updated instructions to the server.

[0819] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0820] This invention combines an emotion engine with a system that automatically generates and manages tasks from voice and image data. This enables task management that takes into account the user's emotional state, thereby improving work efficiency. The system's configuration and operation are described below in detail in the following specific embodiments.

[0821] Server Embodiment

[0822] The server converts the audio data received from the terminal into text using speech recognition technology, and also analyzes the user's emotions from the audio using an emotion engine. The resulting text data is then analyzed using natural language processing technology, and tasks are automatically generated. The generated tasks are prioritized based on the emotion data. For example, if the user is feeling stressed, the priority of the task is temporarily set lower to reduce the burden.

[0823] The server also receives image data and extracts task-related information using image recognition technology. Furthermore, it utilizes the results of sentiment analysis to send users not only appropriate reminders but also encouraging messages when necessary.

[0824] Terminal embodiment

[0825] The device receives task and emotion-based notifications sent from the server and displays them to the user. Tasks are organized by priority, making them easy for the user to review and manage. Depending on the emotional state, the device displays timely reminders, work-related suggestions, and encouraging messages.

[0826] User interaction

[0827] Users perform tasks based on a task list displayed on their device. Additional tasks or changes can be easily updated via the device. The device provides notifications that take sentiment data into account, allowing users to efficiently complete tasks while receiving emotion-based feedback.

[0828] Specific example

[0829] If a user verbally states, "I'll finish the presentation materials for next week," while preparing for a presentation, the device sends this audio data to the server. The server uses speech recognition to transcribe the data into text and an emotion engine to analyze the user's emotions regarding the presentation. If the emotion "nervousness" is detected, the server generates a task, adjusts the priority of related tasks to a lower level, and sends it to the device. In addition to the task list, the device displays a message to the user suggesting ways to relax.

[0830] In this way, the present invention provides a system that reduces the psychological burden on users and improves work efficiency by combining emotion analysis with task management. In particular, for users with attention disorders, work support can be enhanced by sending notifications that are sensitive to their emotions.

[0831] The following describes the processing flow.

[0832] Step 1:

[0833] The device captures the user's speech using a microphone and records it as audio data. Furthermore, it also records additional data such as tone and speed of voice to take into account the user's emotions.

[0834] Step 2:

[0835] The device sends audio data to the server. During this process, metadata, including the time and location of the capture, is added to the audio data.

[0836] Step 3:

[0837] The server converts the received audio data into text data using speech recognition technology. This allows the user's spoken content to be obtained as text information.

[0838] Step 4:

[0839] The server analyzes the voice data using an emotion engine to estimate the user's emotional state. For example, it analyzes the tone and speed of the voice to detect emotions such as tension and stress.

[0840] Step 5:

[0841] The server analyzes the transcribed data using natural language processing technology and automatically generates related tasks. It identifies the content and deadlines of tasks spoken aloud and registers them in the system.

[0842] Step 6:

[0843] The server adjusts the priority of generated tasks, taking emotional data into consideration. If a user is experiencing stress, it will lower the priority of non-urgent tasks, for example.

[0844] Step 7:

[0845] The server sends task information to the terminal. It also generates and transmits emotionally-based messages of encouragement and suggestions to reduce the burden.

[0846] Step 8:

[0847] The device displays a task list and notifications based on sentiment analysis to the user. The task list is organized by priority, and encouraging messages are displayed alongside it.

[0848] Step 9:

[0849] Users can view tasks on their devices and edit or add actions as needed. Emotion-responsive feedback allows for more effective task management.

[0850] Step 10:

[0851] The device feeds back task progress and new updates to the server. This improves the accuracy of task management and the quality of services based on user sentiment data.

[0852] (Example 2)

[0853] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0854] Traditional task management systems process and manage tasks without considering the user's emotional state, which can lead to psychological burden and decreased work efficiency. Furthermore, users with attention deficits require particularly considerate notification methods. Therefore, there is a need for a system that can adjust task priorities according to the user's emotional state, enabling efficient and less burdensome task management.

[0855] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0856] In this invention, the server includes means for receiving audio data and converting it into text information using acoustic analysis technology, means for receiving image information and extracting work-related information using image analysis technology, and means for evaluating the user's emotional state using emotion analysis technology and adjusting the priority of tasks based on that state. This enables flexible task management based on the user's emotional state.

[0857] "Acoustic analysis technology" is a technology that analyzes audio data as a digital signal and converts it into textual information.

[0858] "Image analysis technology" is a technique that processes image information as digital data and extracts meaningful information and patterns.

[0859] "Natural language processing technology" is a technology that enables computers to understand, interpret, and generate human language.

[0860] "Emotional analysis technology" is a technology that evaluates and analyzes a user's emotional state from data such as voice and text.

[0861] "User terminal" refers to an electronic device used by a user to view and edit various notifications and information.

[0862] A "task list" refers to a collection of tasks generated by the system, organized by priority and content.

[0863] "Attention deficit" refers to a condition characterized by difficulty in sustaining attention, and conventional methods of treatment are often complex or insufficient.

[0864] This invention is a system that analyzes voice and image information along with the user's emotional state to effectively manage tasks. Specific embodiments of this system are described below.

[0865] Server Embodiment

[0866] The server receives audio data from the terminal and converts it into text data using acoustic analysis technology. Specifically, it utilizes general-purpose audio analysis software. It also receives image information and extracts task-related information using image analysis technology. General-purpose image analysis software is used for this process. Next, the server uses sentiment analysis technology to evaluate the user's emotions from the audio and text data and adjusts the priority of the tasks generated based on that evaluation. Sentiment analysis software is used for this process. The generated tasks and associated messages are sent to the user's terminal, and the user is notified.

[0867] Terminal embodiment

[0868] The terminal receives task information and messages sent from the server and displays them to the user. This allows the user to manage tasks according to their own status. The terminal can also edit tasks based on user actions, allowing the user to easily add new tasks or modify existing ones.

[0869] User interaction

[0870] Users proceed with their tasks based on a list of tasks displayed on their device. Furthermore, if improvements are needed, they can receive suggestions for task modifications or emotionally-driven suggestions via their device. This allows users to efficiently perform tasks in a way that suits their emotional state.

[0871] Specific example

[0872] For example, if a user says, "I'll finish preparing for next week's meeting," the server converts this audio into text and, through sentiment analysis, determines that the user is feeling "stressed." The server uses this information to adjust the priority of related tasks and provides the user with tasks that include messages suggesting relaxation. This allows the user to work while taking their emotions into consideration.

[0873] An example of a prompt message is: "Convert this audio data to text, analyze the user's emotions, and generate optimal task management suggestions."

[0874] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0875] Step 1:

[0876] The server receives audio data from the terminal. This audio data contains information in which the user has given linguistic instructions to the system. The server converts the audio data into text data using acoustic analysis technology. In this process, speech analysis software is used to analyze the audio waveform and generate a string of characters. The output is text data converted by speech recognition.

[0877] Step 2:

[0878] The server uses text obtained from audio data as input and evaluates the user's emotional state using sentiment analysis technology. In this step, sentiment analysis software is used to analyze emotions such as "tension" or "joy" from factors such as sentence tone and language choice. The output is tag information for the emotions the user is presumed to be feeling.

[0879] Step 3:

[0880] The server uses natural language processing techniques to generate tasks based on the text data obtained in the previous step. Furthermore, it prioritizes the tasks, taking into account the sentiment information obtained in step 2. Specifically, it utilizes a generative AI model to determine tasks based on user requests and their associated priorities. The output is task information prioritized based on sentiment.

[0881] Step 4:

[0882] The server also processes image information received from terminals. Here, image analysis techniques are used to extract task-related information from the image data. For example, key points from meeting slides or documents are identified and incorporated into the task. The output is supplementary information that can be used for task generation.

[0883] Step 5:

[0884] The server sends the generated task information and appropriate messages to the user's terminal. These messages include reminders and suggestions based on the user's emotions. As output, the terminal displays a task summary and an emotionally sensitive message to the user.

[0885] Step 6:

[0886] The terminal displays received task information and messages to the user. The user can check the task list on the terminal and perform tasks based on it. Tasks can also be edited as needed. Input is data from the server, and output is the content displayed on the user interface.

[0887] (Application Example 2)

[0888] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0889] To address the diverse tasks and problems citizens face in their daily lives, efficient task management that takes emotional states into account is required. However, conventional task management systems struggle to reflect users' emotional states in order to appropriately prioritize tasks and make suggestions that contribute to stress reduction, potentially increasing citizens' stress levels. Addressing these challenges is essential.

[0890] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0891] In this invention, the server includes means for receiving voice data and converting it into text data using voice recognition technology, means for analyzing emotions from the voice using an emotion engine and adjusting priorities based on the emotional state, and means for sending the generated tasks to a terminal and setting emotionally sensitive reminders and suggestion messages. This enables optimal task management and stress reduction suggestions according to the emotional state of citizens.

[0892] "Audio data" refers to audio information recorded in digital format and processed by a device.

[0893] "Speech recognition technology" is a technology that converts input speech into text and then understands and analyzes its content.

[0894] "Text data" refers to character information converted using technologies such as speech recognition, and is digital data that can be analyzed by a system.

[0895] "Image data" refers to visual information recorded in a digital format that can be processed mechanically.

[0896] "Image recognition technology" is a technique that extracts and analyzes specific information or patterns from image data.

[0897] "Information" refers to data and knowledge necessary to achieve a specific purpose.

[0898] An "emotion engine" is an algorithm or technology that analyzes, quantifies, and evaluates a user's emotional state from voice and text data.

[0899] "Priority" refers to the criteria used to determine the order in which tasks or processes should be performed, and is a sequence set based on importance and urgency.

[0900] "Natural language processing technology" is a general term encompassing technologies for handling human language on computers, including semantic analysis and information extraction.

[0901] A "task" refers to a series of tasks or activities performed in order to achieve a specific goal.

[0902] A "reminder" is a feature that notifies you to ensure you don't forget important tasks or events.

[0903] A "suggestion message" is a message that includes advice or guidance provided to encourage users to take a specific action or think a particular way.

[0904] A "terminal" refers to a device used by a user to access and operate a system.

[0905] A "task list" refers to a list of tasks organized for the user to manage and review.

[0906] "Editing" refers to the process of changing or updating existing data or information.

[0907] In this system, the server uses speech recognition technology to convert citizens' voice data into text data, and then analyzes that text data using natural language processing technology to generate appropriate tasks. It also uses an emotion engine to perform sentiment analysis and adjust task priorities based on the user's emotional state. Based on the sentiment analysis results, it generates suggestion messages and reminders to reduce stress. Tasks and messages sent from the server are displayed on the terminal, allowing the user to review them and edit tasks as needed.

[0908] The specific processing involves using technologies such as the Google Cloud Speech-to-Text API for speech recognition and spaCy and BERT models for natural language processing. Sentiment analysis is performed using APIs such as AWS Comprehend and Microsoft Azure Text Analytics. By combining these, the server can manage tasks in real time, taking into account the user's voice and emotions, providing personalized support to citizens of the smart city.

[0909] For example, if a citizen uses their smartphone to voice-input "I want to solve the park litter problem," this voice data is converted to text and analyzed on a server. If the emotional state is determined to be "dissatisfied," the system will prioritize the task and suggest contacting the relevant authorities immediately. The user will also be notified with a message such as "We will address this promptly," providing reassurance. An example of a prompt in this case would be: "A citizen's request to solve the park litter problem. We will consider the emotional state and suggest the most appropriate response."

[0910] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0911] Step 1:

[0912] The user provides voice input to the device. Speech recognition technology (Google Cloud Speech-to-Text API) is used to convert the voice data into text data. The input is voice data, and the output is text data. During this process, appropriate formatting is performed as the voice signal is converted into text.

[0913] Step 2:

[0914] The server analyzes the generated text data using natural language processing techniques (such as spaCy and the BERT model) and extracts information. The input is the text data obtained in step 1, and the output is the analyzed information such as meaning and keywords. Various natural language processing techniques, including text structuring and semantic analysis, are used for this data processing.

[0915] Step 3:

[0916] The server uses an emotion engine (such as AWS Comprehend or Microsoft Azure Text Analytics) to analyze user emotions from text data. The input is the text data obtained in step 1, and the output is data that quantifies the user's emotional state. In this data calculation, emotion indicators are extracted, and the user's psychological state is evaluated.

[0917] Step 4:

[0918] The server generates tasks and sets priorities based on the sentiment analysis results. The input is the information obtained in steps 2 and 3, and the output is task data with sentiment-based priorities. This process uses a generative AI model to provide personalized task suggestions to the user.

[0919] Step 5:

[0920] The server sends generated tasks and suggestion messages to the terminal, including emotionally sensitive reminders. The input is the task data obtained in step 4, and the output is information to the user in notification format.

[0921] Step 6:

[0922] The user reviews tasks and messages received on their device and edits the task list as needed. The input is information from step 5, and the output is the updated task list. In this step, tasks are fine-tuned through user interaction.

[0923] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0924] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0925] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0926] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0927] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0928] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0929] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0930] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0931] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0932] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0933] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0934] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0935] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0936] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0937] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0938] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0939] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0940] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0941] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0942] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0943] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0944] The following is further disclosed regarding the embodiments described above.

[0945] (Claim 1)

[0946] A means for receiving audio data and converting it into text data using speech recognition technology,

[0947] A means for receiving image data and extracting task-related information using image recognition technology,

[0948] A method for generating and prioritizing tasks from text data using natural language processing techniques,

[0949] A means of sending the generated task to the terminal and setting a reminder,

[0950] A means to display a task list to the user and allow them to edit tasks,

[0951] A system that includes this.

[0952] (Claim 2)

[0953] The system according to claim 1, which processes audio data and image data in real time and notifies the user of the task.

[0954] (Claim 3)

[0955] The system according to claim 1, which features a reminder function specifically for users with attention disorders and optimizes notifications for important tasks.

[0956] "Example 1"

[0957] (Claim 1)

[0958] A means for receiving audio information and converting it into text information using speech recognition technology,

[0959] A means for receiving image information and extracting business-related information using image recognition technology,

[0960] A method for generating and prioritizing tasks from textual information using natural language processing technology,

[0961] A means for sending the generated tasks to an information processing terminal and setting up a notification function,

[0962] A means to display a list of tasks to users and enable them to edit tasks,

[0963] A means of processing audio and image information in real time and notifying users as part of business operations,

[0964] A system that includes this.

[0965] (Claim 2)

[0966] The system according to claim 1, which has a notification function specifically for users with attention disorders and optimizes notifications for important tasks.

[0967] (Claim 3)

[0968] The system according to claim 1, which uses generational AI technology to automatically generate and adjust tasks, thereby improving the efficiency of operations.

[0969] "Application Example 1"

[0970] (Claim 1)

[0971] A means for receiving acoustic information and converting it into document information through language processing,

[0972] A means for receiving visual information and extracting work-related information using object recognition technology,

[0973] A means for generating work instructions from document information using natural language processing technology and assigning priorities,

[0974] A means for sending the generated work instructions to an information processing device and setting up notifications,

[0975] A means to display a list of tasks to the user and enable editing of work instructions,

[0976] Means for generating work instructions from acoustic and visual information and transmitting them to an automated device,

[0977] A system that includes this.

[0978] (Claim 2)

[0979] The system according to claim 1, which processes acoustic and visual information in real time and notifies the user and automated equipment of the information as work instructions.

[0980] (Claim 3)

[0981] The system according to claim 1, which has a notification function specifically for users with attention disorders and optimizes the notification of important work instructions.

[0982] "Example 2 of combining an emotion engine"

[0983] (Claim 1)

[0984] A means for receiving audio data and converting it into text information using acoustic analysis technology,

[0985] A means for receiving image information and extracting work-related information using image analysis technology,

[0986] A means of generating tasks from textual information using natural language processing technology and setting priorities,

[0987] A means of evaluating the user's emotional state using emotion analysis technology and adjusting task priorities based on those emotions,

[0988] A means to send the generated task and appropriate notification message to the user's device and configure notification settings,

[0989] A means to display a list of tasks to the user and allow them to modify the tasks,

[0990] A system that includes this.

[0991] (Claim 2)

[0992] The system according to claim 1, which simultaneously processes audio and image information and notifies the user of tasks based on emotions.

[0993] (Claim 3)

[0994] The system according to claim 1, which includes a notification function tailored to users with attention disorders and optimizes notifications for important tasks while taking into account their emotional state.

[0995] "Application example 2 when combining with an emotional engine"

[0996] (Claim 1)

[0997] A means for receiving audio data and converting it into text data using speech recognition technology,

[0998] A means of receiving image data and extracting information using image recognition technology,

[0999] A means of analyzing emotions from speech using an emotion engine and adjusting priorities based on emotional state,

[1000] A method for generating tasks from text data using natural language processing technology, and then adjusting and prioritizing them based on sentiment analysis results,

[1001] A means of sending the generated tasks to the device and setting emotionally sensitive reminders and suggestion messages,

[1002] A means to display a task list to the user and enable task editing based on emotion,

[1003] A system that includes this.

[1004] (Claim 2)

[1005] The system according to claim 1, which processes audio data and image data in real time, prioritizes tasks based on sentiment analysis, and notifies the user accordingly.

[1006] (Claim 3)

[1007] The system according to claim 1, which includes a reminder function that takes emotional state into consideration and optimizes notifications for important tasks and stress reduction suggestions. [Explanation of Symbols]

[1008] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving audio data and converting it into text data using speech recognition technology, A means for receiving image data and extracting task-related information using image recognition technology, A method for generating and prioritizing tasks from text data using natural language processing techniques, A means of sending the generated task to the terminal and setting a reminder, A means to display a task list to the user and allow them to edit tasks, A system that includes this.

2. The system according to claim 1, which processes audio data and image data in real time and notifies the user of the task.

3. The system according to claim 1, which includes a reminder function specifically for users with attention disorders and optimizes notifications for important tasks.

Images