system

Abstract

Provide a system. 【Solution means】 A receiving device that receives voice information and character information, An information collection device that collects and stores the activity information of all family members, An information analysis device that analyzes the collected information and assigns priorities to schedules and tasks, A notification device that sets alerts and notifications based on priorities, An interactive device for promoting information exchange within the home, A health management device that collects and analyzes health information to provide health guidance and exercise recommendations, A generation device that generates the activity status of the entire family, A voice synthesis device that provides information by voice output, A system including a recommendation device that provides recommendations suitable for an individual based on healthcare information.

Description

Technical Field

[0005] , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Means for Solving the Problems

[0005] ​​​​​​​​​​​​​​​​​​​This invention provides a system that accepts voice and text input and automatically collects and stores behavioral data of all family members. This enables centralized management of family schedules and tasks. It also provides a notification system that analyzes the collected data, prioritizes schedules and tasks, and sets necessary reminders and notifications for each family member. Furthermore, by providing a chat function to promote smooth communication within the family and a health management system that collects health data of the elderly and provides health management and exercise suggestions, it realizes improvements in family well-being and productivity. This effectively solves the challenges related to complex family schedule management and elderly health management.

[0006] "Voice input" is a method of providing information and commands to a computer system using voice.

[0007] "Text input" is the process of supplying textual information to a computer using a keyboard, touchscreen, or other means.

[0008] "Input means" refers to a device or interface that has the function of receiving voice or text information from a user.

[0009] A "data collection means" is a mechanism for acquiring and recording information about user behavior.

[0010] "Data analysis means" refers to methods and devices for evaluating collected data, extracting useful information, and processing it.

[0011] A "notification mechanism" is a system that has the function of communicating important information or reminders to the user.

[0012] A "chat system" is a system equipped with functions for real-time text-based communication.

[0013] "Health management tools" refer to technologies or methods for monitoring and analyzing an individual's health information and for diagnosing their health status and suggesting improvements.

[0014] A "reporting mechanism" is a system that provides users with information compiled based on collected data.

[0015] A "smart device" is a portable electronic device that provides various functions through computer control.

[0016] A "wearable device" is an electronic device that a user can wear to collect information and receive notifications.

[0017] A "machine learning algorithm" is a computational method that learns from past data and uses that learning to make predictions and analyses about the future. [Brief explanation of the drawing]

[0018] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] Shows an emotion map to which a plurality of emotions are mapped. [Figure 10] Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Modes for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] One embodiment of the present invention is a system for efficiently managing family schedules and tasks. This system operates in cooperation with a server, terminals, and users.

[0040] First, the user enters their daily schedule and tasks into the device via voice input or text input. The device then formats the entered information as digital data and sends it to the server in the appropriate format.

[0041] Next, the server receives this data and stores it in the database. The server checks the integrity of the received data and removes duplicates or corrects the content as needed. The stored data is centrally managed as schedules and tasks for each family member and is shared with other members.

[0042] Furthermore, the server periodically analyzes the stored data and automatically sets priorities for individual tasks and events. This uses machine learning algorithms, which are optimized based on past behavioral trends and current circumstances. Based on these analysis results, push notifications and reminders are sent from the server to the device for important events and tasks as needed. These notifications are adjusted according to priority and urgency to help users not forget.

[0043] Furthermore, to facilitate communication among family members, the device is equipped with a chat function. This function allows each user to instantly send messages to each other and discuss schedules and tasks. This chat simplifies information sharing among all members, prevents misunderstandings, and promotes cooperation.

[0044] In terms of health management, the server stores health information acquired from smart devices and wearables to understand the user's health status. By analyzing this data, appropriate exercise suggestions and health advice are provided. In households with elderly people, this enables early detection of health problems and appropriate responses, improving the sense of security for the entire household.

[0045] As a concrete example, consider a case where a family's daily routine is incorporated into the system. On Monday morning, the user uses the system's voice input function to enter, "I have a doctor's appointment tomorrow at 3 pm." The terminal processes this information and sends it to the server. The server saves the appointment data and reflects it in the calendars of all family members. The day before the visit, the server sends a reminder to the terminal, allowing the user to focus on other tasks with peace of mind.

[0046] Thus, the aim of this invention is to improve the quality of life for families and reduce the burden of complicated schedule management at home through the embodiments of this invention.

[0047] The following describes the processing flow.

[0048] Step 1:

[0049] Users input tasks and events into the device via voice or text. In the case of voice input, the device performs speech recognition and converts it into text data. This data is formatted in a structured format that includes date, time, location, and detailed information.

[0050] Step 2:

[0051] The terminal sends formatted data to the server. The server verifies the integrity of the received data, checking for date and time discrepancies, data errors, and other issues. If inconsistencies are found, an error message is sent to the terminal, prompting the user to make corrections.

[0052] Step 3:

[0053] The server stores the correct data in the database. During storage, tasks are appropriately organized based on each family member's schedule. Schedules and tasks are synchronized and shared across all members' calendars.

[0054] Step 4:

[0055] The server uses machine learning algorithms to analyze stored data and determine the priority of each task. This analysis takes into account past behavioral patterns, urgency, and importance data.

[0056] Step 5:

[0057] Based on the analysis results, the server sets push notifications and reminders according to the priority of tasks and events. The set reminders are sent to the user's device at the specified time.

[0058] Step 6:

[0059] The device notifies the user when it receives reminders or notifications from the server. Notifications are delivered via pop-ups, voice messages, or alert sounds to help the user remember tasks.

[0060] Step 7:

[0061] Users can use their device's chat function to communicate schedule and task changes to family members. The chat is reflected in real time for other members, enabling collaborative schedule management.

[0062] Step 8:

[0063] The server collects health data from smart devices and wearable devices. This data is used to understand the user's health status and provide exercise suggestions and advice. If there are any significant changes in the user's health status, an alert is sent to the device.

[0064] Step 9:

[0065] The server compiles weekly or monthly reports on the family's overall activities and sends them to the user's device. These reports are used to review past activities and help plan for the future.

[0066] (Example 1)

[0067] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0068] Managing complex information within the home, such as scheduling, prioritizing tasks, and health management, is extremely cumbersome due to the lack of integration among various systems and applications. This makes it difficult to efficiently manage all information and facilitate smooth communication among family members. Furthermore, providing real-time advice based on individual health conditions is also challenging.

[0069] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0070] In this invention, the server includes information input means, information collection means, information analysis means, notification generation means, messaging means, health management means, support means, and report generation means. This enables centralized management of information within the home, allowing for efficient schedule management, task prioritization, and provision of health advice.

[0071] "Information input means" refers to technology that provides an interface that allows users to input information in voice or text format.

[0072] "Information gathering means" refers to technologies that have the function of aggregating and storing various data such as user behavior and health status.

[0073] "Information analysis means" refers to technology that analyzes collected data and provides a function to determine priorities based on the user's schedule and tasks.

[0074] A "notification generation method" is a technology for generating and sending reminders and notifications to users based on the results of analyzed data.

[0075] A "messaging tool" is a technology that has a chat function to support information communication between users and facilitate dialogue.

[0076] "Health management tools" refer to technologies that collect users' health data, manage their health status based on that data, and provide exercise suggestions.

[0077] "Support measures" refer to technologies that provide personalized health advice based on individual health data.

[0078] A "report generation method" is a technology that generates reports on the activities of the entire organization based on data accumulated within the system.

[0079] This invention is a system that provides integrated support for schedule management, health management, and communication within the home. This system efficiently processes information through the cooperation of a server, terminals, and users.

[0080] Users first input their schedules and tasks using a device with voice recognition capabilities. This utilizes technology that converts voice input into text. Specifically, smartphones and tablets are used, and voice recognition software such as Google Assistant and Siri supports this.

[0081] The terminal formats the input information as digital data and sends it to the server in an appropriate format. The processing performed by the terminal includes natural language processing techniques and data format conversion, utilizing mobile devices or computers.

[0082] The server receives data sent from the terminal and securely stores it in a database. The stored data is analyzed using machine learning algorithms, and schedules and task priorities are automatically set. This analysis uses machine learning libraries such as Python's scikit-learn.

[0083] Furthermore, the server notifies the user of a reminder based on the analysis results. This notification is sent as a push notification to the device according to priority, utilizing the notification systems of Android® and iOS.

[0084] The device also has a messaging function, allowing users to share information with family members in real time. This streamlines communication throughout the family. For example, chat applications like Slack support this function.

[0085] In terms of health management, the server collects and analyzes health data from wearable devices to provide users with personalized health advice. Health status checks and exercise suggestions are based on data obtained from devices such as Fitbit and Apple Watch.

[0086] As a concrete example, consider a scenario where a user enters "I have a doctor's appointment tomorrow at 3 PM" into their device on a Monday morning. The device processes this information and sends it to the server, which then updates the family's calendar. The day before the appointment, a reminder is sent from the server to the device, allowing the user to focus on other tasks with peace of mind.

[0087] An example of a prompt for a generative AI model would be: "Create a program that creates a reminder and sends it to the device based on the schedule entered by the user: 'I have a doctor's appointment tomorrow at 3pm.'"

[0088] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0089] Step 1:

[0090] Users enter appointments and tasks via voice or text. The entered data is converted into digital format by the device's voice recognition and text processing technology. In the case of voice input, Google Assistant or Siri converts the voice into text and generates formatted data for use in the next step.

[0091] Step 2:

[0092] The terminal sends the formatted data to the server. Here, it is formatted as digital data and sent to the server via an API. This process involves data format conversion and transmission based on network protocols.

[0093] Step 3:

[0094] The server receives data sent from the terminal and saves it to the database. The data first undergoes an integrity check, during which duplicate data is removed and content is corrected. After saving, the data is arranged in a format suitable for analysis.

[0095] Step 4:

[0096] The server analyzes the stored data. This analysis uses machine learning libraries such as Python's scikit-learn to automatically calculate the priority of tasks and appointments. Past data and the current situation are used as input, and priority information is generated as output.

[0097] Step 5:

[0098] The server generates a notification based on the analysis results and sends a push notification to the device. The generated notification is customized according to priority and delivered to the user through the notification service on Android or iOS.

[0099] Step 6:

[0100] The device displays notifications from the server on its screen. The notification system provides users with reminders and opportunities. Users can use this information to adjust their daily activities.

[0101] (Application Example 1)

[0102] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0103] In modern households, managing the schedules and tasks of all family members, facilitating communication, and monitoring their health are important yet cumbersome tasks. This often leads to misunderstandings within the family and delays in health management, contributing to a decline in quality of life. Furthermore, there is a need to utilize household robots to provide smoother information delivery and reminder functions.

[0104] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0105] In this invention, the server includes a receiving device that accepts voice and text information, an information gathering device that collects and stores activity information of all household members, and an information analysis device that analyzes the collected information and prioritizes schedules and tasks. This enables efficient management of schedules and tasks for all family members, smooth communication, and the provision of appropriate advice based on health information.

[0106] "Voice information" refers to information acquired via voice, which converts user instructions and inquiries into a digital format.

[0107] "Textual information" refers to information obtained as text, and involves processing user instructions and inquiries as strings of characters.

[0108] A "receiving device" is a device that has the function of acquiring audio and text information and transmitting it to the system in digital format.

[0109] An "information gathering device" is a device that centrally collects information on the activities and health status of household members and stores it for later analysis.

[0110] An "information analysis device" is a device that analyzes various types of collected information to identify patterns and trends, and contributes to improving household management.

[0111] A "notification device" is a device that provides users with information, either in the form of voice or text, for the purpose of reminding them of schedules and important matters.

[0112] A "dialogue device" is a device that facilitates communication between target users and allows them to exchange messages with each other.

[0113] A "health management device" is a device that analyzes collected health information and provides appropriate health guidance and exercise recommendations.

[0114] A "generation device" is a device that creates a digital report of the overall activity status of a household, making it available for use as reference material later on.

[0115] A "speech synthesis device" is a device that has the function of converting text information into speech and providing that information to the user.

[0116] A "recommendation device" is a device that provides users with personalized recommendations based on information obtained from health management devices.

[0117] This system utilizes a combination of hardware and software to facilitate information management and communication within the home. Specifically, a home robot functions as the primary device, integrating voice recognition, data analysis, and notification functions. This robot is equipped with a microphone and speaker, allowing it to receive, convert, analyze, and output voice information using speech synthesis.

[0118] The entire system is based on Python, using the SpeechRecognition library for speech recognition, Pandas for data management, Scikit-learn for machine learning, and gTTS for speech synthesis. These tools work together to process user-inputted voice and text information in real time, enabling schedule management, health monitoring, and timely notifications.

[0119] The server stores behavioral data from all household members in the cloud and performs multidimensional analysis to generate optimal schedules and health recommendations. This data is provided as reports and notifications generated according to user requests.

[0120] As a concrete example, on a holiday, a home robot might notify the user via voice, "Good morning, today's plan is breakfast with the family and a picnic in the park this afternoon. Are you ready?" It might also suggest, "The weather forecast for tomorrow is rain. Shall we look for ideas for indoor activities this weekend?" The user responds by voice, and the robot uses that information to retrieve more detailed information from the network and provide appropriate recommendations to the user through speech synthesis.

[0121] The AI ​​model for generating prompts will be operated based on instructions such as, "Generate a scenario where a family robot provides voice notifications about schedules and health advice. Include specific examples of dialogue for prioritizing and setting reminders."

[0122] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0123] Step 1:

[0124] The user enters a voice command. The device's microphone captures the voice, and the SpeechRecognition library is used to convert the voice data into text data. Voice input is performed in the form of "Schedule a meeting for 3pm tomorrow," and the output is a text-based instruction.

[0125] Step 2:

[0126] The terminal sends the converted character data to the server for information analysis. Upon receiving this information, the server uses Pandas to store the character data as structured data and processes it into a format that is easy to analyze. The input is character data, and the output is structured data stored as a database entry.

[0127] Step 3:

[0128] The server uses Scikit-learn's machine learning algorithms to analyze trends in the stored data and prioritize schedules. The input is database entries, and the output is a prioritized schedule list. This analysis extracts tasks that users should prioritize.

[0129] Step 4:

[0130] The server generates necessary reminders and notifications using speech synthesis, taking priority into consideration. This process converts text data into audio files using the gTTS library and sends them to the terminal. The input is a prioritized schedule list, and the output is an audio file.

[0131] Step 5:

[0132] The device's speaker plays the generated audio files to notify the user of appointments and reminders verbally. This allows the user to confirm schedules and tasks verbally. As output, the user receives verbal instructions and provides voice feedback as needed.

[0133] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0134] As an embodiment of the present invention, a family schedule and task management system incorporating an emotion engine is provided. This system, in which a server, terminals, and users work together, facilitates the daily lives of families.

[0135] Users input schedules and tasks via voice or text through their device. In the case of voice input, the device converts the input voice into text data, while simultaneously a background emotion engine analyzes the intonation and rhythm of the voice to evaluate the user's emotions. This emotion data is formatted along with the input data and sent to the server.

[0136] The server extracts emotional information from the received data and stores it in a database. This allows the server not only to manage schedules but also to dynamically adjust task priorities based on the user's emotions and individual circumstances. For example, if a user is feeling stressed, the server can set a lower priority for tasks and generate alerts encouraging them to take appropriate rest.

[0137] Furthermore, based on the emotions recognized by the emotion engine, the server can individually edit notifications and reminders, enabling communication tailored to the user's state. For example, if a user is feeling depressed, an encouraging message can be added to the notification, providing psychological support.

[0138] In terms of health management, the system integrates health and emotional data collected from smart devices and wearable devices to provide a more accurate understanding of health status and exercise suggestions. This system supports health management tailored to the user's emotions, for example, by suggesting relaxation exercises when the user is experiencing high levels of stress.

[0139] As a concrete example, suppose a user types "I'm going to dance class tomorrow" into their device. If the emotion engine detects that the user is enjoying themselves, the server will give that activity a high priority. Conversely, if anxiety is detected from the voice, support can be provided by sending helpful information and advice in advance.

[0140] Thus, the present invention aims to improve the quality of life within the home by utilizing an emotional engine to provide schedule management and support that is more tailored to each individual family member.

[0141] The following describes the processing flow.

[0142] Step 1:

[0143] Users enter schedules and tasks using their device via voice or text. In the case of voice input, the device converts the voice data into text, while simultaneously an emotion engine analyzes the characteristics of the voice to identify the user's emotions.

[0144] Step 2:

[0145] The device sends formatted data, including emotions, to the server. This data includes the date and time, location, task details, and emotion information.

[0146] Step 3:

[0147] The server analyzes the received data and stores it in a database. At this time, sentiment data is recorded along with schedule data, which can be used for future analysis.

[0148] Step 4:

[0149] The server analyzes the collected data using machine learning algorithms. By taking sentiment data into consideration, it is possible to dynamically set the priority of each task.

[0150] Step 5:

[0151] Based on the analysis results of the emotion engine, the server creates notifications and reminders. For example, if the user's emotions indicate stress, the server lowers the task's priority and sets a notification that includes suggestions to promote relaxation.

[0152] Step 6:

[0153] The device displays notifications and reminders sent from the server to the user. The content of the notifications is customized according to the user's emotional state.

[0154] Step 7:

[0155] Users can communicate with their families using the device's chat function. Appropriate communication is possible through timing and methods adjusted based on emotional information.

[0156] Step 8:

[0157] The server continuously collects health data from smart devices and wearable devices and monitors health status by combining it with emotional data. If the emotional engine detects significant stress, it will suggest health management measures that require special attention.

[0158] Step 9:

[0159] The server generates weekly or monthly reports based on activity data for the entire family and sends them to the device. These reports also reflect emotional states and include information that can help improve individual well-being.

[0160] (Example 2)

[0161] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0162] In modern family life, managing each member's schedule and tasks, as well as monitoring their health, is complex and difficult to address individually. Furthermore, flexible planning and support that takes into account each member's emotional state are required, but traditional systems have been ineffective in achieving this. Therefore, a system capable of improving the efficiency and quality of life for the entire family is desired.

[0163] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0164] In this invention, the server includes data analysis means for analyzing emotional data and evaluating the user's emotional state, priority setting means for prioritizing schedules and tasks, and emotional support means for providing psychological support based on the user's emotions. This makes it possible to manage individualized schedules and tasks that take into account the user's emotional state, and to provide effective psychological support.

[0165] "Information input means" refers to a device or function that allows a user to input schedules and tasks using voice or text.

[0166] "Data analysis means" refers to a device or program that has the processing capability to analyze information entered by a user and, in particular, to evaluate the user's emotional state based on emotional data.

[0167] A "memory device" is a system component that has a storage function for storing collected information.

[0168] A "priority setting means" is a function or device for setting priorities for schedules and tasks based on their importance and urgency.

[0169] "Notification generation means" refers to a function or device for generating and providing appropriate notification information to the user based on a set priority.

[0170] "Communication methods" refer to system components, including chat and messaging functions, used to facilitate the exchange of information within a household.

[0171] A "health management device" is a device or system that has the function of monitoring a person's health status and, based on that, suggesting exercises and lifestyle improvements for maintaining good health.

[0172] An "emotional support device" is a device or program that provides individualized support based on the user's emotional state.

[0173] A "portable information processing device" is an electronic device that is portable and equipped with the functions of data collection and processing.

[0174] A "wearable information processing device" is an electronic device that can be worn by a user and has the function of collecting and processing biometric and environmental information.

[0175] An "artificial intelligence algorithm" is a program or technology that enables a machine to perform analysis and inference from data, and to personalize future suggestions based on the user's past behavior.

[0176] This invention functions as a family schedule and task management system incorporating an emotion engine. Users input schedules and tasks using voice or text via a device such as a smartphone or tablet. The input voice data is converted to text using speech recognition software, and emotion analysis software simultaneously evaluates the user's emotions. Specific software used includes a "Voice Input API" for speech recognition and an "Emotion Analysis API" for emotion analysis.

[0177] The device sends this data to the server. Based on the received sentiment data and schedule information, the server dynamically adjusts task priorities using a priority setting algorithm. Furthermore, based on the user's emotional state, it utilizes a notification generation function to provide appropriate reminders and support messages.

[0178] In health management, biometric information obtained from wearable information devices worn by the user is analyzed and integrated with the user's emotional state to provide appropriate suggestions for exercise and health maintenance. For this purpose, data such as heart rate and activity levels are collected.

[0179] For example, if a user enters "I'm going jogging tomorrow" into their device, the emotion engine can detect the user's motivation, and the server will prioritize that activity. Conversely, if the emotion data indicates the user is feeling stressed, the system can provide a notification recommending exercise that promotes relaxation.

[0180] Generative AI models can also be used to generate specific messages that respond to the user's emotions. An example of a prompt might be, "Generate a motivational message based on the user's emotional state."

[0181] This system enables schedule management that takes into account each user's emotions and health condition, aiming to improve the quality of life for families.

[0182] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0183] Step 1:

[0184] Users input schedule and task information via voice or text through their device. In the case of voice input, the device uses speech recognition software to convert the voice into text. Here, voice data is input and converted into text data. This text data forms the basis for the next processing step.

[0185] Step 2:

[0186] The device uses emotion analysis software to evaluate the user's emotions from voice or text data. In this process, the device extracts emotion data from the intonation and rhythm of the voice and the content of the text, and outputs the emotion analysis results. This emotion data plays a crucial role in schedule management.

[0187] Step 3:

[0188] The terminal combines the converted text data and sentiment data, formats it, and sends it to the server. The input consists of text data and sentiment data, and structured data (e.g., in JSON format) containing these is transmitted to the server.

[0189] Step 4:

[0190] The server extracts schedule and sentiment information from the received data. A data analysis module then operates to analyze the user's state and the tasks they need. The output is the analysis results for prioritization and notification generation.

[0191] Step 5:

[0192] The server dynamically sets priorities based on user sentiment data. The priority setting algorithm lowers the priority of tasks when the user is experiencing stress, and conversely, gives higher priority to activities when positive emotions are detected. The input is the analysis results from earlier, and the output is updated priority information.

[0193] Step 6:

[0194] The server generates and sends notifications and reminders to users based on priority. Specifically, it uses a generative AI model to create messages and provides them to users as notifications. Priority information and sentiment data are used as input, and notification messages are generated as output.

[0195] Step 7:

[0196] The server acquires health data from wearable information processing devices and integrates it with emotional information. This data integration process involves a detailed assessment of the user's health status and the generation of exercise suggestions tailored to their needs. The inputs to this process are health information and emotional information, and the output is customized health suggestions.

[0197] (Application Example 2)

[0198] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0199] In daily family life, managing each individual's schedule and tasks often becomes complicated, and there is a lack of appropriate responses and communication tailored to each member's emotional state. Furthermore, health management often fails to consider the emotional state of each individual, creating challenges for the entire family to lead a richer and healthier life. Therefore, flexible schedule and health management that accommodates the emotional state of each member is necessary.

[0200] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0201] In this invention, the server includes a terminal device that accepts voice and text input, a data collection device that collects and stores behavioral data of family members, a data analysis device that analyzes the collected data to extract emotional data and prioritize schedules and tasks, a notification device that individually edits and sets reminders and notifications based on priority and emotional data, a dialogue device for facilitating electronic communication within the home, a health management device that collects and analyzes health-related data to provide health management and exercise suggestions, a reporting device that generates activity reports for the entire family, and an emotion response device that generates notifications and advice according to the user's emotional state. This enables schedule management tailored to each family member, as well as health management and appropriate communication according to their emotional state.

[0202] A "terminal device" is a device that receives input data from users via voice and text.

[0203] A "data collection device" is a device that has the function of collecting behavioral data of family members and storing that data.

[0204] A "data analysis device" is a device that extracts emotional data from collected behavioral data and processes it to prioritize schedules and tasks.

[0205] A "notification device" is a device that has the function of individually editing and setting reminders and notifications based on priority and sentiment data.

[0206] A "dialogue device" is a device that has functions to facilitate electronic dialogue within the home and to make communication smoother.

[0207] A "health management device" is a device that collects health-related data, analyzes it, and provides health management and exercise suggestions to the user.

[0208] A "reporting device" is a device that has the function of generating activity reports for the entire family.

[0209] An "emotional response device" is a device that generates notifications and advice according to the user's emotional state.

[0210] This system is a comprehensive management system designed to support the lives of family members. The terminal accepts data from users through voice and text input. In the case of voice input, an emotion engine is used to analyze the intonation and rhythm of the input voice data and evaluate the user's emotional state. The data collection device centrally collects and stores behavioral and emotional data from family members.

[0211] The server analyzes the collected data using a data analysis device and extracts sentiment data. Based on this analysis, it prioritizes schedules and tasks. The notification device then individually edits and delivers reminders and notifications to the user based on this priority and sentiment data.

[0212] The dialogue device has the function of generating natural conversations using a generative AI model to facilitate electronic conversations within the home. The health management device analyzes health-related data collected from portable and wearable devices to more accurately understand the user's health status and provides exercise suggestions based on emotions.

[0213] For example, if a user voice-inputs "I'm going jogging tomorrow," the device can sense that the user is enjoying the activity. The server then sets a high priority for that activity and sends information to the user via a notification device. In this way, appropriate support tailored to the user's state is provided.

[0214] An example of a prompt for a generative AI model is: "Design a program that uses an emotion engine to implement a way to manage household schedules and tasks. Provide an example where the robot suggests specific actions based on the emotional state of each family member."

[0215] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0216] Step 1:

[0217] Users input schedule information via voice or text through their device. Voice input is converted to text using speech recognition. During this process, the device utilizes an emotion engine to analyze the intonation and rhythm of the voice and generate emotion data. The input data is then prepared as both text and emotion data.

[0218] Step 2:

[0219] The terminal transmits prepared text information and sentiment data to the data collection device. The data collection device stores the input behavioral and sentiment data in a database. This stored data, which also includes each user's past behavioral history, is used for later analysis.

[0220] Step 3:

[0221] The server uses a data analysis device to analyze behavioral and emotional data in the database. This data analysis employs machine learning algorithms to prioritize user schedules and tasks based on their emotions. The analysis uses past behavioral data and new emotional data as input, and outputs schedule data with assigned priorities.

[0222] Step 4:

[0223] The server generates reminders and notifications through the notification device based on the analysis results. These generated notifications include personalized messages tailored to the user's emotional state. For example, if the user is feeling stressed, a message such as "Take it easy today and get some rest" might be sent.

[0224] Step 5:

[0225] The dialogue device uses a generative AI model to revitalize communication within the home. The generative AI model utilizes prompts to generate natural dialogue. During this process, the dialogue output incorporates emotional data, providing the user with appropriate feedback and advice.

[0226] Step 6:

[0227] The health management device analyzes health-related data obtained from portable and wearable devices. The server integrates emotional data and health-related data to obtain information to understand the user's health status. Based on this information, appropriate exercise suggestions are provided to the user via a notification device.

[0228] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0229] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0230] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0231] [Second Embodiment]

[0232] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0233] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0234] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0235] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0236] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0237] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0238] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0239] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0240] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0241] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0242] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0243] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0244] One embodiment of the present invention is a system for efficiently managing family schedules and tasks. This system operates in cooperation with a server, terminals, and users.

[0245] First, the user enters their daily schedule and tasks into the device via voice input or text input. The device then formats the entered information as digital data and sends it to the server in the appropriate format.

[0246] Next, the server receives this data and stores it in the database. The server checks the integrity of the received data and removes duplicates or corrects the content as needed. The stored data is centrally managed as schedules and tasks for each family member and is shared with other members.

[0247] Furthermore, the server periodically analyzes the stored data and automatically sets priorities for individual tasks and events. This uses machine learning algorithms, which are optimized based on past behavioral trends and current circumstances. Based on these analysis results, push notifications and reminders are sent from the server to the device for important events and tasks as needed. These notifications are adjusted according to priority and urgency to help users not forget.

[0248] Furthermore, to facilitate communication among family members, the device is equipped with a chat function. This function allows each user to instantly send messages to each other and discuss schedules and tasks. This chat simplifies information sharing among all members, prevents misunderstandings, and promotes cooperation.

[0249] In terms of health management, the server stores health information acquired from smart devices and wearables to understand the user's health status. By analyzing this data, appropriate exercise suggestions and health advice are provided. In households with elderly people, this enables early detection of health problems and appropriate responses, improving the sense of security for the entire household.

[0250] As a concrete example, consider a case where a family's daily routine is incorporated into the system. On Monday morning, the user uses the system's voice input function to enter, "I have a doctor's appointment tomorrow at 3 pm." The terminal processes this information and sends it to the server. The server saves the appointment data and reflects it in the calendars of all family members. The day before the visit, the server sends a reminder to the terminal, allowing the user to focus on other tasks with peace of mind.

[0251] Thus, the aim of this invention is to improve the quality of life for families and reduce the burden of complicated schedule management at home through the embodiments of this invention.

[0252] The following describes the processing flow.

[0253] Step 1:

[0254] Users input tasks and events into the device via voice or text. In the case of voice input, the device performs speech recognition and converts it into text data. This data is formatted in a structured format that includes date, time, location, and detailed information.

[0255] Step 2:

[0256] The terminal sends formatted data to the server. The server verifies the integrity of the received data, checking for date and time discrepancies, data errors, and other issues. If inconsistencies are found, an error message is sent to the terminal, prompting the user to make corrections.

[0257] Step 3:

[0258] The server stores the correct data in the database. During storage, tasks are appropriately organized based on each family member's schedule. Schedules and tasks are synchronized and shared across all members' calendars.

[0259] Step 4:

[0260] The server uses machine learning algorithms to analyze stored data and determine the priority of each task. This analysis takes into account past behavioral patterns, urgency, and importance data.

[0261] Step 5:

[0262] Based on the analysis results, the server sets push notifications and reminders according to the priority of tasks and events. The set reminders are sent to the user's device at the specified time.

[0263] Step 6:

[0264] The device notifies the user when it receives reminders or notifications from the server. Notifications are delivered via pop-ups, voice messages, or alert sounds to help the user remember tasks.

[0265] Step 7:

[0266] Users can use their device's chat function to communicate schedule and task changes to family members. The chat is reflected in real time for other members, enabling collaborative schedule management.

[0267] Step 8:

[0268] The server collects health data from smart devices and wearable devices. This data is used to understand the user's health status and provide exercise suggestions and advice. If there are any significant changes in the user's health status, an alert is sent to the device.

[0269] Step 9:

[0270] The server compiles weekly or monthly reports on the family's overall activities and sends them to the user's device. These reports are used to review past activities and help plan for the future.

[0271] (Example 1)

[0272] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0273] Managing complex information within the home, such as scheduling, prioritizing tasks, and health management, is extremely cumbersome due to the lack of integration among various systems and applications. This makes it difficult to efficiently manage all information and facilitate smooth communication among family members. Furthermore, providing real-time advice based on individual health conditions is also challenging.

[0274] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0275] In this invention, the server includes information input means, information collection means, information analysis means, notification generation means, messaging means, health management means, support means, and report generation means. This enables centralized management of information within the home, allowing for efficient schedule management, task prioritization, and provision of health advice.

[0276] "Information input means" refers to technology that provides an interface that allows users to input information in voice or text format.

[0277] "Information gathering means" refers to technologies that have the function of aggregating and storing various data such as user behavior and health status.

[0278] "Information analysis means" refers to technology that analyzes collected data and provides a function to determine priorities based on the user's schedule and tasks.

[0279] A "notification generation method" is a technology for generating and sending reminders and notifications to users based on the results of analyzed data.

[0280] A "messaging tool" is a technology that has a chat function to support information communication between users and facilitate dialogue.

[0281] The "health management means" is a technology for collecting users' health data, managing their health status based on it, and providing exercise suggestions.

[0282] The "support means" is a technology for providing individual health advice based on individual health data.

[0283] The "report generation means" is a technology for generating a report on the activities of the entire group based on the data accumulated in the system.

[0284] This invention is a system that integrally supports schedule management, health management, and communication within a household. This system performs efficient information processing through the cooperation of a server, terminals, and users.

[0285] First, the user uses a terminal with a voice recognition function to input schedules and tasks. For this, a technology for converting voice input into text is used. Specifically, smartphones or tablets are utilized, and voice recognition software such as Google Assistant or Siri supports this.

[0286] The terminal formats the input information as digital data and sends it to the server in a suitable format. The processing performed by the terminal includes natural language processing technology and data format conversion, and mobile devices or computers are used.

[0287] The server receives the data sent from the terminal and securely stores it in a database. The stored data is analyzed using machine learning algorithms, and the priorities of schedules and tasks are automatically set. Machine learning libraries such as Python's scikit-learn are used for this analysis.

[0288] Furthermore, based on the analysis results, the server notifies the user of reminders. This notification is sent as a push notification to the terminal according to the priority, utilizing the notification systems of Android and iOS.

[0289] The device also has a messaging function, allowing users to share information with family members in real time. This streamlines communication throughout the family. For example, chat applications like Slack support this function.

[0290] In terms of health management, the server collects and analyzes health data from wearable devices to provide users with personalized health advice. Health status checks and exercise suggestions are based on data obtained from devices such as Fitbit and Apple Watch.

[0291] As a concrete example, consider a scenario where a user enters "I have a doctor's appointment tomorrow at 3 PM" into their device on a Monday morning. The device processes this information and sends it to the server, which then updates the family's calendar. The day before the appointment, a reminder is sent from the server to the device, allowing the user to focus on other tasks with peace of mind.

[0292] An example of a prompt for a generative AI model would be: "Create a program that creates a reminder and sends it to the device based on the schedule entered by the user: 'I have a doctor's appointment tomorrow at 3pm.'"

[0293] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0294] Step 1:

[0295] Users enter appointments and tasks via voice or text. The entered data is converted into digital format by the device's voice recognition and text processing technology. In the case of voice input, Google Assistant or Siri converts the voice into text and generates formatted data for use in the next step.

[0296] Step 2:

[0297] The terminal sends the formatted data to the server. Here, it is formatted as digital data and sent to the server via an API. This process involves data format conversion and transmission based on network protocols.

[0298] Step 3:

[0299] The server receives data sent from the terminal and saves it to the database. The data first undergoes an integrity check, during which duplicate data is removed and content is corrected. After saving, the data is arranged in a format suitable for analysis.

[0300] Step 4:

[0301] The server analyzes the stored data. This analysis uses machine learning libraries such as Python's scikit-learn to automatically calculate the priority of tasks and appointments. Past data and the current situation are used as input, and priority information is generated as output.

[0302] Step 5:

[0303] The server generates a notification based on the analysis results and sends a push notification to the device. The generated notification is customized according to priority and delivered to the user through the notification service on Android or iOS.

[0304] Step 6:

[0305] The device displays notifications from the server on its screen. The notification system provides users with reminders and opportunities. Users can use this information to adjust their daily activities.

[0306] (Application Example 1)

[0307] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0308] In modern households, managing the schedules and tasks of all family members, promoting communication, and monitoring health conditions are important yet cumbersome tasks. This makes it easy for family members to miss each other or for health management to be delayed, which can reduce the quality of life. There is also a demand for leveraging household robots to achieve smooth information provision and reminder functions.

[0309] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0310] In this invention, the server includes a receiving device that receives voice information and character information, an information collection device that collects and stores the activity information of all household members, and an information analysis device that analyzes the collected information and assigns priorities to schedules and tasks. This enables efficient management of the schedules and tasks of all family members, smooth communication, and provision of appropriate advice based on health information.

[0311] "Voice information" refers to information obtained through voice, which converts user instructions and inquiries into digital form.

[0312] "Character information" refers to information obtained as text, which processes user instructions and inquiries as character strings.

[0313] The "receiving device" is a device that has the function of acquiring voice information and character information and transmitting them to the system in digital form.

[0314] The "information collection device" is a device that centrally collects the activity information, health status, etc. of household members and stores them for later analysis.

[0315] The "information analysis device" is a device that analyzes various collected information to find patterns and trends, and contributes to the improvement of household management.

[0316] A "notification device" is a device that provides users with information, either in the form of voice or text, for the purpose of reminding them of schedules and important matters.

[0317] A "dialogue device" is a device that facilitates communication between target users and allows them to exchange messages with each other.

[0318] A "health management device" is a device that analyzes collected health information and provides appropriate health guidance and exercise recommendations.

[0319] A "generation device" is a device that creates a digital report of the overall activity status of a household, making it available for use as reference material later on.

[0320] A "speech synthesis device" is a device that has the function of converting text information into speech and providing that information to the user.

[0321] A "recommendation device" is a device that provides users with personalized recommendations based on information obtained from health management devices.

[0322] This system utilizes a combination of hardware and software to facilitate information management and communication within the home. Specifically, a home robot functions as the primary device, integrating voice recognition, data analysis, and notification functions. This robot is equipped with a microphone and speaker, allowing it to receive, convert, analyze, and output voice information using speech synthesis.

[0323] The entire system is based on Python, using the SpeechRecognition library for speech recognition, Pandas for data management, Scikit-learn for machine learning, and gTTS for speech synthesis. These tools work together to process user-inputted voice and text information in real time, enabling schedule management, health monitoring, and timely notifications.

[0324] The server stores behavioral data from all household members in the cloud and performs multidimensional analysis to generate optimal schedules and health recommendations. This data is provided as reports and notifications generated according to user requests.

[0325] As a concrete example, on a holiday, a home robot might notify the user via voice, "Good morning, today's plan is breakfast with the family and a picnic in the park this afternoon. Are you ready?" It might also suggest, "The weather forecast for tomorrow is rain. Shall we look for ideas for indoor activities this weekend?" The user responds by voice, and the robot uses that information to retrieve more detailed information from the network and provide appropriate recommendations to the user through speech synthesis.

[0326] The AI ​​model for generating prompts will be operated based on instructions such as, "Generate a scenario where a family robot provides voice notifications about schedules and health advice. Include specific examples of dialogue for prioritizing and setting reminders."

[0327] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0328] Step 1:

[0329] The user enters a voice command. The device's microphone captures the voice, and the SpeechRecognition library is used to convert the voice data into text data. Voice input is performed in the form of "Schedule a meeting for 3pm tomorrow," and the output is a text-based instruction.

[0330] Step 2:

[0331] The terminal sends the converted character data to the server for information analysis. Upon receiving this information, the server uses Pandas to store the character data as structured data and processes it into a format that is easy to analyze. The input is character data, and the output is structured data stored as a database entry.

[0332] Step 3:

[0333] The server uses Scikit-learn's machine learning algorithms to analyze trends in the stored data and prioritize schedules. The input is database entries, and the output is a prioritized schedule list. This analysis extracts tasks that users should prioritize.

[0334] Step 4:

[0335] The server generates necessary reminders and notifications using speech synthesis, taking priority into consideration. This process converts text data into audio files using the gTTS library and sends them to the terminal. The input is a prioritized schedule list, and the output is an audio file.

[0336] Step 5:

[0337] The device's speaker plays the generated audio files to notify the user of appointments and reminders verbally. This allows the user to confirm schedules and tasks verbally. As output, the user receives verbal instructions and provides voice feedback as needed.

[0338] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0339] As an embodiment of the present invention, a family schedule and task management system incorporating an emotion engine is provided. This system, in which a server, terminals, and users work together, facilitates the daily lives of families.

[0340] Users input schedules and tasks via voice or text through their device. In the case of voice input, the device converts the input voice into text data, while simultaneously a background emotion engine analyzes the intonation and rhythm of the voice to evaluate the user's emotions. This emotion data is formatted along with the input data and sent to the server.

[0341] The server extracts emotional information from the received data and stores it in a database. This allows the server not only to manage schedules but also to dynamically adjust task priorities based on the user's emotions and individual circumstances. For example, if a user is feeling stressed, the server can set a lower priority for tasks and generate alerts encouraging them to take appropriate rest.

[0342] Furthermore, based on the emotions recognized by the emotion engine, the server can individually edit notifications and reminders, enabling communication tailored to the user's state. For example, if a user is feeling depressed, an encouraging message can be added to the notification, providing psychological support.

[0343] In terms of health management, the system integrates health and emotional data collected from smart devices and wearable devices to provide a more accurate understanding of health status and exercise suggestions. This system supports health management tailored to the user's emotions, for example, by suggesting relaxation exercises when the user is experiencing high levels of stress.

[0344] As a concrete example, suppose a user types "I'm going to dance class tomorrow" into their device. If the emotion engine detects that the user is enjoying themselves, the server will give that activity a high priority. Conversely, if anxiety is detected from the voice, support can be provided by sending helpful information and advice in advance.

[0345] Thus, the present invention aims to improve the quality of life within the home by utilizing an emotional engine to provide schedule management and support that is more tailored to each individual family member.

[0346] The following describes the processing flow.

[0347] Step 1:

[0348] Users enter schedules and tasks using their device via voice or text. In the case of voice input, the device converts the voice data into text, while simultaneously an emotion engine analyzes the characteristics of the voice to identify the user's emotions.

[0349] Step 2:

[0350] The device sends formatted data, including emotions, to the server. This data includes the date and time, location, task details, and emotion information.

[0351] Step 3:

[0352] The server analyzes the received data and stores it in a database. At this time, sentiment data is recorded along with schedule data, which can be used for future analysis.

[0353] Step 4:

[0354] The server analyzes the collected data using machine learning algorithms. By taking sentiment data into consideration, it is possible to dynamically set the priority of each task.

[0355] Step 5:

[0356] Based on the analysis results of the emotion engine, the server creates notifications and reminders. For example, if the user's emotions indicate stress, the server lowers the task's priority and sets a notification that includes suggestions to promote relaxation.

[0357] Step 6:

[0358] The device displays notifications and reminders sent from the server to the user. The content of the notifications is customized according to the user's emotional state.

[0359] Step 7:

[0360] Users can communicate with their families using the device's chat function. Appropriate communication is possible through timing and methods adjusted based on emotional information.

[0361] Step 8:

[0362] The server continuously collects health data from smart devices and wearable devices and monitors health status by combining it with emotional data. If the emotional engine detects significant stress, it will suggest health management measures that require special attention.

[0363] Step 9:

[0364] The server generates weekly or monthly reports based on activity data for the entire family and sends them to the device. These reports also reflect emotional states and include information that can help improve individual well-being.

[0365] (Example 2)

[0366] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0367] In modern family life, managing each member's schedule and tasks, as well as monitoring their health, is complex and difficult to address individually. Furthermore, flexible planning and support that takes into account each member's emotional state are required, but traditional systems have been ineffective in achieving this. Therefore, a system capable of improving the efficiency and quality of life for the entire family is desired.

[0368] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0369] In this invention, the server includes data analysis means for analyzing emotional data and evaluating the user's emotional state, priority setting means for prioritizing schedules and tasks, and emotional support means for providing psychological support based on the user's emotions. This makes it possible to manage individualized schedules and tasks that take into account the user's emotional state, and to provide effective psychological support.

[0370] "Information input means" refers to a device or function that allows a user to input schedules and tasks using voice or text.

[0371] "Data analysis means" refers to a device or program that has the processing capability to analyze information entered by a user and, in particular, to evaluate the user's emotional state based on emotional data.

[0372] A "memory device" is a system component that has a storage function for storing collected information.

[0373] A "priority setting means" is a function or device for setting priorities for schedules and tasks based on their importance and urgency.

[0374] "Notification generation means" refers to a function or device for generating and providing appropriate notification information to the user based on a set priority.

[0375] "Communication methods" refer to system components, including chat and messaging functions, used to facilitate the exchange of information within a household.

[0376] A "health management device" is a device or system that has the function of monitoring a person's health status and, based on that, suggesting exercises and lifestyle improvements for maintaining good health.

[0377] An "emotional support device" is a device or program that provides individualized support based on the user's emotional state.

[0378] A "portable information processing device" is an electronic device that is portable and equipped with the functions of data collection and processing.

[0379] A "wearable information processing device" is an electronic device that can be worn by a user and has the function of collecting and processing biometric and environmental information.

[0380] An "artificial intelligence algorithm" is a program or technology that enables a machine to perform analysis and inference from data, and to personalize future suggestions based on the user's past behavior.

[0381] This invention functions as a family schedule and task management system incorporating an emotion engine. Users input schedules and tasks using voice or text via a device such as a smartphone or tablet. The input voice data is converted to text using speech recognition software, and emotion analysis software simultaneously evaluates the user's emotions. Specific software used includes a "Voice Input API" for speech recognition and an "Emotion Analysis API" for emotion analysis.

[0382] The device sends this data to the server. Based on the received sentiment data and schedule information, the server dynamically adjusts task priorities using a priority setting algorithm. Furthermore, based on the user's emotional state, it utilizes a notification generation function to provide appropriate reminders and support messages.

[0383] In health management, biometric information obtained from wearable information devices worn by the user is analyzed and integrated with the user's emotional state to provide appropriate suggestions for exercise and health maintenance. For this purpose, data such as heart rate and activity levels are collected.

[0384] For example, if a user enters "I'm going jogging tomorrow" into their device, the emotion engine can detect the user's motivation, and the server will prioritize that activity. Conversely, if the emotion data indicates the user is feeling stressed, the system can provide a notification recommending exercise that promotes relaxation.

[0385] Generative AI models can also be used to generate specific messages that respond to the user's emotions. An example of a prompt might be, "Generate a motivational message based on the user's emotional state."

[0386] This system enables schedule management that takes into account each user's emotions and health condition, aiming to improve the quality of life for families.

[0387] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0388] Step 1:

[0389] Users input schedule and task information via voice or text through their device. In the case of voice input, the device uses speech recognition software to convert the voice into text. Here, voice data is input and converted into text data. This text data forms the basis for the next processing step.

[0390] Step 2:

[0391] The device uses emotion analysis software to evaluate the user's emotions from voice or text data. In this process, the device extracts emotion data from the intonation and rhythm of the voice and the content of the text, and outputs the emotion analysis results. This emotion data plays a crucial role in schedule management.

[0392] Step 3:

[0393] The terminal combines the converted text data and sentiment data, formats it, and sends it to the server. The input consists of text data and sentiment data, and structured data (e.g., in JSON format) containing these is transmitted to the server.

[0394] Step 4:

[0395] The server extracts schedule and sentiment information from the received data. A data analysis module then operates to analyze the user's state and the tasks they need. The output is the analysis results for prioritization and notification generation.

[0396] Step 5:

[0397] The server dynamically sets priorities based on user sentiment data. The priority setting algorithm lowers the priority of tasks when the user is experiencing stress, and conversely, gives higher priority to activities when positive emotions are detected. The input is the analysis results from earlier, and the output is updated priority information.

[0398] Step 6:

[0399] The server generates and sends notifications and reminders to users based on priority. Specifically, it uses a generative AI model to create messages and provides them to users as notifications. Priority information and sentiment data are used as input, and notification messages are generated as output.

[0400] Step 7:

[0401] The server acquires health data from wearable information processing devices and integrates it with emotional information. This data integration process involves a detailed assessment of the user's health status and the generation of exercise suggestions tailored to their needs. The inputs to this process are health information and emotional information, and the output is customized health suggestions.

[0402] (Application Example 2)

[0403] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0404] In daily family life, managing each individual's schedule and tasks often becomes complicated, and there is a lack of appropriate responses and communication tailored to each member's emotional state. Furthermore, health management often fails to consider the emotional state of each individual, creating challenges for the entire family to lead a richer and healthier life. Therefore, flexible schedule and health management that accommodates the emotional state of each member is necessary.

[0405] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0406] In this invention, the server includes a terminal device that accepts voice and text input, a data collection device that collects and stores behavioral data of family members, a data analysis device that analyzes the collected data to extract emotional data and prioritize schedules and tasks, a notification device that individually edits and sets reminders and notifications based on priority and emotional data, a dialogue device for facilitating electronic communication within the home, a health management device that collects and analyzes health-related data to provide health management and exercise suggestions, a reporting device that generates activity reports for the entire family, and an emotion response device that generates notifications and advice according to the user's emotional state. This enables schedule management tailored to each family member, as well as health management and appropriate communication according to their emotional state.

[0407] A "terminal device" is a device that receives input data from users via voice and text.

[0408] A "data collection device" is a device that has the function of collecting behavioral data of family members and storing that data.

[0409] A "data analysis device" is a device that extracts emotional data from collected behavioral data and processes it to prioritize schedules and tasks.

[0410] A "notification device" is a device that has the function of individually editing and setting reminders and notifications based on priority and sentiment data.

[0411] A "dialogue device" is a device that has functions to facilitate electronic dialogue within the home and to make communication smoother.

[0412] A "health management device" is a device that collects health-related data, analyzes it, and provides health management and exercise suggestions to the user.

[0413] A "reporting device" is a device that has the function of generating activity reports for the entire family.

[0414] An "emotional response device" is a device that generates notifications and advice according to the user's emotional state.

[0415] This system is a comprehensive management system designed to support the lives of family members. The terminal accepts data from users through voice and text input. In the case of voice input, an emotion engine is used to analyze the intonation and rhythm of the input voice data and evaluate the user's emotional state. The data collection device centrally collects and stores behavioral and emotional data from family members.

[0416] The server analyzes the collected data using a data analysis device and extracts sentiment data. Based on this analysis, it prioritizes schedules and tasks. The notification device then individually edits and delivers reminders and notifications to the user based on this priority and sentiment data.

[0417] The dialogue device has the function of generating natural conversations using a generative AI model to facilitate electronic conversations within the home. The health management device analyzes health-related data collected from portable and wearable devices to more accurately understand the user's health status and provides exercise suggestions based on emotions.

[0418] For example, if a user voice-inputs "I'm going jogging tomorrow," the device can sense that the user is enjoying the activity. The server then sets a high priority for that activity and sends information to the user via a notification device. In this way, appropriate support tailored to the user's state is provided.

[0419] An example of a prompt for a generative AI model is: "Design a program that uses an emotion engine to implement a way to manage household schedules and tasks. Provide an example where the robot suggests specific actions based on the emotional state of each family member."

[0420] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0421] Step 1:

[0422] Users input schedule information via voice or text through their device. Voice input is converted to text using speech recognition. During this process, the device utilizes an emotion engine to analyze the intonation and rhythm of the voice and generate emotion data. The input data is then prepared as both text and emotion data.

[0423] Step 2:

[0424] The terminal transmits prepared text information and sentiment data to the data collection device. The data collection device stores the input behavioral and sentiment data in a database. This stored data, which also includes each user's past behavioral history, is used for later analysis.

[0425] Step 3:

[0426] The server uses a data analysis device to analyze behavioral and emotional data in the database. This data analysis employs machine learning algorithms to prioritize user schedules and tasks based on their emotions. The analysis uses past behavioral data and new emotional data as input, and outputs schedule data with assigned priorities.

[0427] Step 4:

[0428] The server generates reminders and notifications through the notification device based on the analysis results. These generated notifications include personalized messages tailored to the user's emotional state. For example, if the user is feeling stressed, a message such as "Take it easy today and get some rest" might be sent.

[0429] Step 5:

[0430] The dialogue device uses a generative AI model to revitalize communication within the home. The generative AI model utilizes prompts to generate natural dialogue. During this process, the dialogue output incorporates emotional data, providing the user with appropriate feedback and advice.

[0431] Step 6:

[0432] The health management device analyzes health-related data obtained from portable and wearable devices. The server integrates emotional data and health-related data to obtain information to understand the user's health status. Based on this information, appropriate exercise suggestions are provided to the user via a notification device.

[0433] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0434] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0435] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0436] [Third Embodiment]

[0437] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0438] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0439] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0440] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0441] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0442] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0443] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0444] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0445] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0446] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0447] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0448] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0449] One embodiment of the present invention is a system for efficiently managing family schedules and tasks. This system operates in cooperation with a server, terminals, and users.

[0450] First, the user enters their daily schedule and tasks into the device via voice input or text input. The device then formats the entered information as digital data and sends it to the server in the appropriate format.

[0451] Next, the server receives this data and stores it in the database. The server checks the integrity of the received data and removes duplicates or corrects the content as needed. The stored data is centrally managed as schedules and tasks for each family member and is shared with other members.

[0452] Furthermore, the server periodically analyzes the stored data and automatically sets priorities for individual tasks and events. This uses machine learning algorithms, which are optimized based on past behavioral trends and current circumstances. Based on these analysis results, push notifications and reminders are sent from the server to the device for important events and tasks as needed. These notifications are adjusted according to priority and urgency to help users not forget.

[0453] Furthermore, to facilitate communication among family members, the device is equipped with a chat function. This function allows each user to instantly send messages to each other and discuss schedules and tasks. This chat simplifies information sharing among all members, prevents misunderstandings, and promotes cooperation.

[0454] In terms of health management, the server stores health information acquired from smart devices and wearables to understand the user's health status. By analyzing this data, appropriate exercise suggestions and health advice are provided. In households with elderly people, this enables early detection of health problems and appropriate responses, improving the sense of security for the entire household.

[0455] As a concrete example, consider a case where a family's daily routine is incorporated into the system. On Monday morning, the user uses the system's voice input function to enter, "I have a doctor's appointment tomorrow at 3 pm." The terminal processes this information and sends it to the server. The server saves the appointment data and reflects it in the calendars of all family members. The day before the visit, the server sends a reminder to the terminal, allowing the user to focus on other tasks with peace of mind.

[0456] Thus, the aim of this invention is to improve the quality of life for families and reduce the burden of complicated schedule management at home through the embodiments of this invention.

[0457] The following describes the processing flow.

[0458] Step 1:

[0459] Users input tasks and events into the device via voice or text. In the case of voice input, the device performs speech recognition and converts it into text data. This data is formatted in a structured format that includes date, time, location, and detailed information.

[0460] Step 2:

[0461] The terminal sends formatted data to the server. The server verifies the integrity of the received data, checking for date and time discrepancies, data errors, and other issues. If inconsistencies are found, an error message is sent to the terminal, prompting the user to make corrections.

[0462] Step 3:

[0463] The server stores the correct data in the database. During storage, tasks are appropriately organized based on each family member's schedule. Schedules and tasks are synchronized and shared across all members' calendars.

[0464] Step 4:

[0465] The server uses machine learning algorithms to analyze stored data and determine the priority of each task. This analysis takes into account past behavioral patterns, urgency, and importance data.

[0466] Step 5:

[0467] Based on the analysis results, the server sets push notifications and reminders according to the priority of tasks and events. The set reminders are sent to the user's device at the specified time.

[0468] Step 6:

[0469] The device notifies the user when it receives reminders or notifications from the server. Notifications are delivered via pop-ups, voice messages, or alert sounds to help the user remember tasks.

[0470] Step 7:

[0471] Users can use their device's chat function to communicate schedule and task changes to family members. The chat is reflected in real time for other members, enabling collaborative schedule management.

[0472] Step 8:

[0473] The server collects health data from smart devices and wearable devices. This data is used to understand the user's health status and provide exercise suggestions and advice. If there are any significant changes in the user's health status, an alert is sent to the device.

[0474] Step 9:

[0475] The server compiles weekly or monthly reports on the family's overall activities and sends them to the user's device. These reports are used to review past activities and help plan for the future.

[0476] (Example 1)

[0477] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0478] Managing complex information within the home, such as scheduling, prioritizing tasks, and health management, is extremely cumbersome due to the lack of integration among various systems and applications. This makes it difficult to efficiently manage all information and facilitate smooth communication among family members. Furthermore, providing real-time advice based on individual health conditions is also challenging.

[0479] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0480] In this invention, the server includes information input means, information collection means, information analysis means, notification generation means, messaging means, health management means, support means, and report generation means. This enables centralized management of information within the home, allowing for efficient schedule management, task prioritization, and provision of health advice.

[0481] "Information input means" refers to technology that provides an interface that allows users to input information in voice or text format.

[0482] "Information gathering means" refers to technologies that have the function of aggregating and storing various data such as user behavior and health status.

[0483] "Information analysis means" refers to technology that analyzes collected data and provides a function to determine priorities based on the user's schedule and tasks.

[0484] A "notification generation method" is a technology for generating and sending reminders and notifications to users based on the results of analyzed data.

[0485] A "messaging tool" is a technology that has a chat function to support information communication between users and facilitate dialogue.

[0486] "Health management tools" refer to technologies that collect users' health data, manage their health status based on that data, and provide exercise suggestions.

[0487] "Support measures" refer to technologies that provide personalized health advice based on individual health data.

[0488] A "report generation method" is a technology that generates reports on the activities of the entire organization based on data accumulated within the system.

[0489] This invention is a system that provides integrated support for schedule management, health management, and communication within the home. This system efficiently processes information through the cooperation of a server, terminals, and users.

[0490] Users first input their schedules and tasks using a device with voice recognition capabilities. This utilizes technology that converts voice input into text. Specifically, smartphones and tablets are used, and voice recognition software such as Google Assistant and Siri supports this process.

[0491] The terminal formats the input information as digital data and sends it to the server in an appropriate format. The processing performed by the terminal includes natural language processing techniques and data format conversion, utilizing mobile devices or computers.

[0492] The server receives data sent from the terminal and securely stores it in a database. The stored data is analyzed using machine learning algorithms, and schedules and task priorities are automatically set. This analysis uses machine learning libraries such as Python's scikit-learn.

[0493] Furthermore, the server notifies the user of a reminder based on the analysis results. This notification is sent as a push notification to the device according to priority, utilizing the notification systems of Android and iOS.

[0494] The device also has a messaging function, allowing users to share information with family members in real time. This streamlines communication throughout the family. For example, chat applications like Slack support this function.

[0495] In terms of health management, the server collects and analyzes health data from wearable devices to provide users with personalized health advice. Health status checks and exercise suggestions are based on data obtained from devices such as Fitbit and Apple Watch.

[0496] As a concrete example, consider a scenario where a user enters "I have a doctor's appointment tomorrow at 3 PM" into their device on a Monday morning. The device processes this information and sends it to the server, which then updates the family's calendar. The day before the appointment, a reminder is sent from the server to the device, allowing the user to focus on other tasks with peace of mind.

[0497] An example of a prompt for a generative AI model would be: "Create a program that creates a reminder and sends it to the device based on the schedule entered by the user: 'I have a doctor's appointment tomorrow at 3pm.'"

[0498] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0499] Step 1:

[0500] Users enter appointments and tasks via voice or text. The entered data is converted into digital format by the device's voice recognition and text processing technology. In the case of voice input, Google Assistant or Siri converts the voice into text and generates formatted data for use in the next step.

[0501] Step 2:

[0502] The terminal sends the formatted data to the server. Here, it is formatted as digital data and sent to the server via an API. This process involves data format conversion and transmission based on network protocols.

[0503] Step 3:

[0504] The server receives data sent from the terminal and saves it to the database. The data first undergoes an integrity check, during which duplicate data is removed and content is corrected. After saving, the data is arranged in a format suitable for analysis.

[0505] Step 4:

[0506] The server analyzes the stored data. This analysis uses machine learning libraries such as Python's scikit-learn to automatically calculate the priority of tasks and appointments. Past data and the current situation are used as input, and priority information is generated as output.

[0507] Step 5:

[0508] The server generates a notification based on the analysis results and sends a push notification to the device. The generated notification is customized according to priority and delivered to the user through the notification service on Android or iOS.

[0509] Step 6:

[0510] The device displays notifications from the server on its screen. The notification system provides users with reminders and opportunities. Users can use this information to adjust their daily activities.

[0511] (Application Example 1)

[0512] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0513] In modern households, managing the schedules and tasks of all family members, facilitating communication, and monitoring their health are important yet cumbersome tasks. This often leads to misunderstandings within the family and delays in health management, contributing to a decline in quality of life. Furthermore, there is a need to utilize household robots to provide smoother information delivery and reminder functions.

[0514] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0515] In this invention, the server includes a receiving device that accepts voice and text information, an information gathering device that collects and stores activity information of all household members, and an information analysis device that analyzes the collected information and prioritizes schedules and tasks. This enables efficient management of schedules and tasks for all family members, smooth communication, and the provision of appropriate advice based on health information.

[0516] "Voice information" refers to information acquired via voice, which converts user instructions and inquiries into a digital format.

[0517] "Textual information" refers to information obtained as text, and involves processing user instructions and inquiries as strings of characters.

[0518] A "receiving device" is a device that has the function of acquiring audio and text information and transmitting it to the system in digital format.

[0519] An "information gathering device" is a device that centrally collects information on the activities and health status of household members and stores it for later analysis.

[0520] An "information analysis device" is a device that analyzes various types of collected information to identify patterns and trends, and contributes to improving household management.

[0521] A "notification device" is a device that provides users with information, either in the form of voice or text, for the purpose of reminding them of schedules and important matters.

[0522] A "dialogue device" is a device that facilitates communication between target users and allows them to exchange messages with each other.

[0523] A "health management device" is a device that analyzes collected health information and provides appropriate health guidance and exercise recommendations.

[0524] A "generation device" is a device that creates a digital report of the overall activity status of a household, making it available for use as reference material later on.

[0525] A "speech synthesis device" is a device that has the function of converting text information into speech and providing that information to the user.

[0526] A "recommendation device" is a device that provides users with personalized recommendations based on information obtained from health management devices.

[0527] This system utilizes a combination of hardware and software to facilitate information management and communication within the home. Specifically, a home robot functions as the primary device, integrating voice recognition, data analysis, and notification functions. This robot is equipped with a microphone and speaker, allowing it to receive, convert, analyze, and output voice information using speech synthesis.

[0528] The entire system is based on Python, using the SpeechRecognition library for speech recognition, Pandas for data management, Scikit-learn for machine learning, and gTTS for speech synthesis. These tools work together to process user-inputted voice and text information in real time, enabling schedule management, health monitoring, and timely notifications.

[0529] The server stores behavioral data from all household members in the cloud and performs multidimensional analysis to generate optimal schedules and health recommendations. This data is provided as reports and notifications generated according to user requests.

[0530] As a concrete example, on a holiday, a home robot might notify the user via voice, "Good morning, today's plan is breakfast with the family and a picnic in the park this afternoon. Are you ready?" It might also suggest, "The weather forecast for tomorrow is rain. Shall we look for ideas for indoor activities this weekend?" The user responds by voice, and the robot uses that information to retrieve more detailed information from the network and provide appropriate recommendations to the user through speech synthesis.

[0531] The AI ​​model for generating prompts will be operated based on instructions such as, "Generate a scenario where a family robot provides voice notifications about schedules and health advice. Include specific examples of dialogue for prioritizing and setting reminders."

[0532] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0533] Step 1:

[0534] The user enters a voice command. The device's microphone captures the voice, and the SpeechRecognition library is used to convert the voice data into text data. Voice input is performed in the form of "Schedule a meeting for 3pm tomorrow," and the output is a text-based instruction.

[0535] Step 2:

[0536] The terminal sends the converted character data to the server for information analysis. Upon receiving this information, the server uses Pandas to store the character data as structured data and processes it into a format that is easy to analyze. The input is character data, and the output is structured data stored as a database entry.

[0537] Step 3:

[0538] The server uses Scikit-learn's machine learning algorithms to analyze trends in the stored data and prioritize schedules. The input is database entries, and the output is a prioritized schedule list. This analysis extracts tasks that users should prioritize.

[0539] Step 4:

[0540] The server generates necessary reminders and notifications using speech synthesis, taking priority into consideration. This process converts text data into audio files using the gTTS library and sends them to the terminal. The input is a prioritized schedule list, and the output is an audio file.

[0541] Step 5:

[0542] The device's speaker plays the generated audio files to notify the user of appointments and reminders verbally. This allows the user to confirm schedules and tasks verbally. As output, the user receives verbal instructions and provides voice feedback as needed.

[0543] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0544] As an embodiment of the present invention, a family schedule and task management system incorporating an emotion engine is provided. This system, in which a server, terminals, and users work together, facilitates the daily lives of families.

[0545] Users input schedules and tasks via voice or text through their device. In the case of voice input, the device converts the input voice into text data, while simultaneously a background emotion engine analyzes the intonation and rhythm of the voice to evaluate the user's emotions. This emotion data is formatted along with the input data and sent to the server.

[0546] The server extracts emotional information from the received data and stores it in a database. This allows the server not only to manage schedules but also to dynamically adjust task priorities based on the user's emotions and individual circumstances. For example, if a user is feeling stressed, the server can set a lower priority for tasks and generate alerts encouraging them to take appropriate rest.

[0547] Furthermore, based on the emotions recognized by the emotion engine, the server can individually edit notifications and reminders, enabling communication tailored to the user's state. For example, if a user is feeling depressed, an encouraging message can be added to the notification, providing psychological support.

[0548] In terms of health management, the system integrates health and emotional data collected from smart devices and wearable devices to provide a more accurate understanding of health status and exercise suggestions. This system supports health management tailored to the user's emotions, for example, by suggesting relaxation exercises when the user is experiencing high levels of stress.

[0549] As a concrete example, suppose a user types "I'm going to dance class tomorrow" into their device. If the emotion engine detects that the user is enjoying themselves, the server will give that activity a high priority. Conversely, if anxiety is detected from the voice, support can be provided by sending helpful information and advice in advance.

[0550] Thus, the present invention aims to improve the quality of life within the home by utilizing an emotional engine to provide schedule management and support that is more tailored to each individual family member.

[0551] The following describes the processing flow.

[0552] Step 1:

[0553] Users enter schedules and tasks using their device via voice or text. In the case of voice input, the device converts the voice data into text, while simultaneously an emotion engine analyzes the characteristics of the voice to identify the user's emotions.

[0554] Step 2:

[0555] The device sends formatted data, including emotions, to the server. This data includes the date and time, location, task details, and emotion information.

[0556] Step 3:

[0557] The server analyzes the received data and stores it in a database. At this time, sentiment data is recorded along with schedule data, which can be used for future analysis.

[0558] Step 4:

[0559] The server analyzes the collected data using machine learning algorithms. By taking sentiment data into consideration, it is possible to dynamically set the priority of each task.

[0560] Step 5:

[0561] Based on the analysis results of the emotion engine, the server creates notifications and reminders. For example, if the user's emotions indicate stress, the server lowers the task's priority and sets a notification that includes suggestions to promote relaxation.

[0562] Step 6:

[0563] The device displays notifications and reminders sent from the server to the user. The content of the notifications is customized according to the user's emotional state.

[0564] Step 7:

[0565] Users can communicate with their families using the device's chat function. Appropriate communication is possible through timing and methods adjusted based on emotional information.

[0566] Step 8:

[0567] The server continuously collects health data from smart devices and wearable devices and monitors health status by combining it with emotional data. If the emotional engine detects significant stress, it will suggest health management measures that require special attention.

[0568] Step 9:

[0569] The server generates weekly or monthly reports based on activity data for the entire family and sends them to the device. These reports also reflect emotional states and include information that can help improve individual well-being.

[0570] (Example 2)

[0571] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0572] In modern family life, managing each member's schedule and tasks, as well as monitoring their health, is complex and difficult to address individually. Furthermore, flexible planning and support that takes into account each member's emotional state are required, but traditional systems have been ineffective in achieving this. Therefore, a system capable of improving the efficiency and quality of life for the entire family is desired.

[0573] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0574] In this invention, the server includes data analysis means for analyzing emotional data and evaluating the user's emotional state, priority setting means for prioritizing schedules and tasks, and emotional support means for providing psychological support based on the user's emotions. This makes it possible to manage individualized schedules and tasks that take into account the user's emotional state, and to provide effective psychological support.

[0575] "Information input means" refers to a device or function that allows a user to input schedules and tasks using voice or text.

[0576] "Data analysis means" refers to a device or program that has the processing capability to analyze information entered by a user and, in particular, to evaluate the user's emotional state based on emotional data.

[0577] A "memory device" is a system component that has a storage function for storing collected information.

[0578] A "priority setting means" is a function or device for setting priorities for schedules and tasks based on their importance and urgency.

[0579] "Notification generation means" refers to a function or device for generating and providing appropriate notification information to the user based on a set priority.

[0580] "Communication methods" refer to system components, including chat and messaging functions, used to facilitate the exchange of information within a household.

[0581] A "health management device" is a device or system that has the function of monitoring a person's health status and, based on that, suggesting exercises and lifestyle improvements for maintaining good health.

[0582] An "emotional support device" is a device or program that provides individualized support based on the user's emotional state.

[0583] A "portable information processing device" is an electronic device that is portable and equipped with the functions of data collection and processing.

[0584] A "wearable information processing device" is an electronic device that can be worn by a user and has the function of collecting and processing biometric and environmental information.

[0585] An "artificial intelligence algorithm" is a program or technology that enables a machine to perform analysis and inference from data, and to personalize future suggestions based on the user's past behavior.

[0586] This invention functions as a family schedule and task management system incorporating an emotion engine. Users input schedules and tasks using voice or text via a device such as a smartphone or tablet. The input voice data is converted to text using speech recognition software, and emotion analysis software simultaneously evaluates the user's emotions. Specific software used includes a "Voice Input API" for speech recognition and an "Emotion Analysis API" for emotion analysis.

[0587] The device sends this data to the server. Based on the received sentiment data and schedule information, the server dynamically adjusts task priorities using a priority setting algorithm. Furthermore, based on the user's emotional state, it utilizes a notification generation function to provide appropriate reminders and support messages.

[0588] In health management, biometric information obtained from wearable information devices worn by the user is analyzed and integrated with the user's emotional state to provide appropriate suggestions for exercise and health maintenance. For this purpose, data such as heart rate and activity levels are collected.

[0589] For example, if a user enters "I'm going jogging tomorrow" into their device, the emotion engine can detect the user's motivation, and the server will prioritize that activity. Conversely, if the emotion data indicates the user is feeling stressed, the system can provide a notification recommending exercise that promotes relaxation.

[0590] Generative AI models can also be used to generate specific messages that respond to the user's emotions. An example of a prompt might be, "Generate a motivational message based on the user's emotional state."

[0591] This system enables schedule management that takes into account each user's emotions and health condition, aiming to improve the quality of life for families.

[0592] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0593] Step 1:

[0594] Users input schedule and task information via voice or text through their device. In the case of voice input, the device uses speech recognition software to convert the voice into text. Here, voice data is input and converted into text data. This text data forms the basis for the next processing step.

[0595] Step 2:

[0596] The device uses emotion analysis software to evaluate the user's emotions from voice or text data. In this process, the device extracts emotion data from the intonation and rhythm of the voice and the content of the text, and outputs the emotion analysis results. This emotion data plays a crucial role in schedule management.

[0597] Step 3:

[0598] The terminal combines the converted text data and sentiment data, formats it, and sends it to the server. The input consists of text data and sentiment data, and structured data (e.g., in JSON format) containing these is transmitted to the server.

[0599] Step 4:

[0600] The server extracts schedule and sentiment information from the received data. A data analysis module then operates to analyze the user's state and the tasks they need. The output is the analysis results for prioritization and notification generation.

[0601] Step 5:

[0602] The server dynamically sets priorities based on user sentiment data. The priority setting algorithm lowers the priority of tasks when the user is experiencing stress, and conversely, gives higher priority to activities when positive emotions are detected. The input is the analysis results from earlier, and the output is updated priority information.

[0603] Step 6:

[0604] The server generates and sends notifications and reminders to users based on priority. Specifically, it uses a generative AI model to create messages and provides them to users as notifications. Priority information and sentiment data are used as input, and notification messages are generated as output.

[0605] Step 7:

[0606] The server acquires health data from wearable information processing devices and integrates it with emotional information. This data integration process involves a detailed assessment of the user's health status and the generation of exercise suggestions tailored to their needs. The inputs to this process are health information and emotional information, and the output is customized health suggestions.

[0607] (Application Example 2)

[0608] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0609] In daily family life, managing each individual's schedule and tasks often becomes complicated, and there is a lack of appropriate responses and communication tailored to each member's emotional state. Furthermore, health management often fails to consider the emotional state of each individual, creating challenges for the entire family to lead a richer and healthier life. Therefore, flexible schedule and health management that accommodates the emotional state of each member is necessary.

[0610] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0611] In this invention, the server includes a terminal device that accepts voice and text input, a data collection device that collects and stores behavioral data of family members, a data analysis device that analyzes the collected data to extract emotional data and prioritize schedules and tasks, a notification device that individually edits and sets reminders and notifications based on priority and emotional data, a dialogue device for facilitating electronic communication within the home, a health management device that collects and analyzes health-related data to provide health management and exercise suggestions, a reporting device that generates activity reports for the entire family, and an emotion response device that generates notifications and advice according to the user's emotional state. This enables schedule management tailored to each family member, as well as health management and appropriate communication according to their emotional state.

[0612] A "terminal device" is a device that receives input data from users via voice and text.

[0613] A "data collection device" is a device that has the function of collecting behavioral data of family members and storing that data.

[0614] A "data analysis device" is a device that extracts emotional data from collected behavioral data and processes it to prioritize schedules and tasks.

[0615] A "notification device" is a device that has the function of individually editing and setting reminders and notifications based on priority and sentiment data.

[0616] A "dialogue device" is a device that has functions to facilitate electronic dialogue within the home and to make communication smoother.

[0617] A "health management device" is a device that collects health-related data, analyzes it, and provides health management and exercise suggestions to the user.

[0618] A "reporting device" is a device that has the function of generating activity reports for the entire family.

[0619] An "emotional response device" is a device that generates notifications and advice according to the user's emotional state.

[0620] This system is a comprehensive management system designed to support the lives of family members. The terminal accepts data from users through voice and text input. In the case of voice input, an emotion engine is used to analyze the intonation and rhythm of the input voice data and evaluate the user's emotional state. The data collection device centrally collects and stores behavioral and emotional data from family members.

[0621] The server analyzes the collected data using a data analysis device and extracts sentiment data. Based on this analysis, it prioritizes schedules and tasks. The notification device then individually edits and delivers reminders and notifications to the user based on this priority and sentiment data.

[0622] The dialogue device has the function of generating natural conversations using a generative AI model to facilitate electronic conversations within the home. The health management device analyzes health-related data collected from portable and wearable devices to more accurately understand the user's health status and provides exercise suggestions based on emotions.

[0623] For example, if a user voice-inputs "I'm going jogging tomorrow," the device can sense that the user is enjoying the activity. The server then sets a high priority for that activity and sends information to the user via a notification device. In this way, appropriate support tailored to the user's state is provided.

[0624] An example of a prompt for a generative AI model is: "Design a program that uses an emotion engine to implement a way to manage household schedules and tasks. Provide an example where the robot suggests specific actions based on the emotional state of each family member."

[0625] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0626] Step 1:

[0627] Users input schedule information via voice or text through their device. Voice input is converted to text using speech recognition. During this process, the device utilizes an emotion engine to analyze the intonation and rhythm of the voice and generate emotion data. The input data is then prepared as both text and emotion data.

[0628] Step 2:

[0629] The terminal transmits prepared text information and sentiment data to the data collection device. The data collection device stores the input behavioral and sentiment data in a database. This stored data, which also includes each user's past behavioral history, is used for later analysis.

[0630] Step 3:

[0631] The server uses a data analysis device to analyze behavioral and emotional data in the database. This data analysis employs machine learning algorithms to prioritize user schedules and tasks based on their emotions. The analysis uses past behavioral data and new emotional data as input, and outputs schedule data with assigned priorities.

[0632] Step 4:

[0633] The server generates reminders and notifications through the notification device based on the analysis results. These generated notifications include personalized messages tailored to the user's emotional state. For example, if the user is feeling stressed, a message such as "Take it easy today and get some rest" might be sent.

[0634] Step 5:

[0635] The dialogue device uses a generative AI model to revitalize communication within the home. The generative AI model utilizes prompts to generate natural dialogue. During this process, the dialogue output incorporates emotional data, providing the user with appropriate feedback and advice.

[0636] Step 6:

[0637] The health management device analyzes health-related data obtained from portable and wearable devices. The server integrates emotional data and health-related data to obtain information to understand the user's health status. Based on this information, appropriate exercise suggestions are provided to the user via a notification device.

[0638] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0639] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0640] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0641] [Fourth Embodiment]

[0642] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0643] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0644] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0645] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0646] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0647] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0648] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0649] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0650] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0651] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0652] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0653] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0654] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0655] One embodiment of the present invention is a system for efficiently managing family schedules and tasks. This system operates in cooperation with a server, terminals, and users.

[0656] First, the user enters their daily schedule and tasks into the device via voice input or text input. The device then formats the entered information as digital data and sends it to the server in the appropriate format.

[0657] Next, the server receives this data and stores it in the database. The server checks the integrity of the received data and removes duplicates or corrects the content as needed. The stored data is centrally managed as schedules and tasks for each family member and is shared with other members.

[0658] Furthermore, the server periodically analyzes the stored data and automatically sets priorities for individual tasks and events. This uses machine learning algorithms, which are optimized based on past behavioral trends and current circumstances. Based on these analysis results, push notifications and reminders are sent from the server to the device for important events and tasks as needed. These notifications are adjusted according to priority and urgency to help users not forget.

[0659] Furthermore, to facilitate communication among family members, the device is equipped with a chat function. This function allows each user to instantly send messages to each other and discuss schedules and tasks. This chat simplifies information sharing among all members, prevents misunderstandings, and promotes cooperation.

[0660] In terms of health management, the server stores health information acquired from smart devices and wearables to understand the user's health status. By analyzing this data, appropriate exercise suggestions and health advice are provided. In households with elderly people, this enables early detection of health problems and appropriate responses, improving the sense of security for the entire household.

[0661] As a concrete example, consider a case where a family's daily routine is incorporated into the system. On Monday morning, the user uses the system's voice input function to enter, "I have a doctor's appointment tomorrow at 3 pm." The terminal processes this information and sends it to the server. The server saves the appointment data and reflects it in the calendars of all family members. The day before the visit, the server sends a reminder to the terminal, allowing the user to focus on other tasks with peace of mind.

[0662] Thus, the aim of this invention is to improve the quality of life for families and reduce the burden of complicated schedule management at home through the embodiments of this invention.

[0663] The following describes the processing flow.

[0664] Step 1:

[0665] Users input tasks and events into the device via voice or text. In the case of voice input, the device performs speech recognition and converts it into text data. This data is formatted in a structured format that includes date, time, location, and detailed information.

[0666] Step 2:

[0667] The terminal sends formatted data to the server. The server verifies the integrity of the received data, checking for date and time discrepancies, data errors, and other issues. If inconsistencies are found, an error message is sent to the terminal, prompting the user to make corrections.

[0668] Step 3:

[0669] The server stores the correct data in the database. During storage, tasks are appropriately organized based on each family member's schedule. Schedules and tasks are synchronized and shared across all members' calendars.

[0670] Step 4:

[0671] The server uses machine learning algorithms to analyze stored data and determine the priority of each task. This analysis takes into account past behavioral patterns, urgency, and importance data.

[0672] Step 5:

[0673] Based on the analysis results, the server sets push notifications and reminders according to the priority of tasks and events. The set reminders are sent to the user's device at the specified time.

[0674] Step 6:

[0675] The device notifies the user when it receives reminders or notifications from the server. Notifications are delivered via pop-ups, voice messages, or alert sounds to help the user remember tasks.

[0676] Step 7:

[0677] Users can use their device's chat function to communicate schedule and task changes to family members. The chat is reflected in real time for other members, enabling collaborative schedule management.

[0678] Step 8:

[0679] The server collects health data from smart devices and wearable devices. This data is used to understand the user's health status and provide exercise suggestions and advice. If there are any significant changes in the user's health status, an alert is sent to the device.

[0680] Step 9:

[0681] The server compiles weekly or monthly reports on the family's overall activities and sends them to the user's device. These reports are used to review past activities and help plan for the future.

[0682] (Example 1)

[0683] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0684] Managing complex information within the home, such as scheduling, prioritizing tasks, and health management, is extremely cumbersome due to the lack of integration among various systems and applications. This makes it difficult to efficiently manage all information and facilitate smooth communication among family members. Furthermore, providing real-time advice based on individual health conditions is also challenging.

[0685] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0686] In this invention, the server includes information input means, information collection means, information analysis means, notification generation means, messaging means, health management means, support means, and report generation means. This enables centralized management of information within the home, allowing for efficient schedule management, task prioritization, and provision of health advice.

[0687] "Information input means" refers to technology that provides an interface that allows users to input information in voice or text format.

[0688] "Information gathering means" refers to technologies that have the function of aggregating and storing various data such as user behavior and health status.

[0689] "Information analysis means" refers to technology that analyzes collected data and provides a function to determine priorities based on the user's schedule and tasks.

[0690] A "notification generation method" is a technology for generating and sending reminders and notifications to users based on the results of analyzed data.

[0691] A "messaging tool" is a technology that has a chat function to support information communication between users and facilitate dialogue.

[0692] "Health management tools" refer to technologies that collect users' health data, manage their health status based on that data, and provide exercise suggestions.

[0693] "Support measures" refer to technologies that provide personalized health advice based on individual health data.

[0694] A "report generation method" is a technology that generates reports on the activities of the entire organization based on data accumulated within the system.

[0695] This invention is a system that provides integrated support for schedule management, health management, and communication within the home. This system efficiently processes information through the cooperation of a server, terminals, and users.

[0696] Users first input their schedules and tasks using a device with voice recognition capabilities. This utilizes technology that converts voice input into text. Specifically, smartphones and tablets are used, and voice recognition software such as Google Assistant and Siri supports this process.

[0697] The terminal formats the input information as digital data and sends it to the server in an appropriate format. The processing performed by the terminal includes natural language processing techniques and data format conversion, utilizing mobile devices or computers.

[0698] The server receives data sent from the terminal and securely stores it in a database. The stored data is analyzed using machine learning algorithms, and schedules and task priorities are automatically set. This analysis uses machine learning libraries such as Python's scikit-learn.

[0699] Furthermore, the server notifies the user of a reminder based on the analysis results. This notification is sent as a push notification to the device according to priority, utilizing the notification systems of Android and iOS.

[0700] The device also has a messaging function, allowing users to share information with family members in real time. This streamlines communication throughout the family. For example, chat applications like Slack support this function.

[0701] In terms of health management, the server collects and analyzes health data from wearable devices to provide users with personalized health advice. Health status checks and exercise suggestions are based on data obtained from devices such as Fitbit and Apple Watch.

[0702] As a concrete example, consider a scenario where a user enters "I have a doctor's appointment tomorrow at 3 PM" into their device on a Monday morning. The device processes this information and sends it to the server, which then updates the family's calendar. The day before the appointment, a reminder is sent from the server to the device, allowing the user to focus on other tasks with peace of mind.

[0703] An example of a prompt for a generative AI model would be: "Create a program that creates a reminder and sends it to the device based on the schedule entered by the user: 'I have a doctor's appointment tomorrow at 3pm.'"

[0704] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0705] Step 1:

[0706] Users enter appointments and tasks via voice or text. The entered data is converted into digital format by the device's voice recognition and text processing technology. In the case of voice input, Google Assistant or Siri converts the voice into text and generates formatted data for use in the next step.

[0707] Step 2:

[0708] The terminal sends the formatted data to the server. Here, it is formatted as digital data and sent to the server via an API. This process involves data format conversion and transmission based on network protocols.

[0709] Step 3:

[0710] The server receives data sent from the terminal and saves it to the database. The data first undergoes an integrity check, during which duplicate data is removed and content is corrected. After saving, the data is arranged in a format suitable for analysis.

[0711] Step 4:

[0712] The server analyzes the stored data. This analysis uses machine learning libraries such as Python's scikit-learn to automatically calculate the priority of tasks and appointments. Past data and the current situation are used as input, and priority information is generated as output.

[0713] Step 5:

[0714] The server generates a notification based on the analysis results and sends a push notification to the device. The generated notification is customized according to priority and delivered to the user through the notification service on Android or iOS.

[0715] Step 6:

[0716] The device displays notifications from the server on its screen. The notification system provides users with reminders and opportunities. Users can use this information to adjust their daily activities.

[0717] (Application Example 1)

[0718] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0719] In modern households, managing the schedules and tasks of all family members, facilitating communication, and monitoring their health are important yet cumbersome tasks. This often leads to misunderstandings within the family and delays in health management, contributing to a decline in quality of life. Furthermore, there is a need to utilize household robots to provide smoother information delivery and reminder functions.

[0720] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0721] In this invention, the server includes a receiving device that accepts voice and text information, an information gathering device that collects and stores activity information of all household members, and an information analysis device that analyzes the collected information and prioritizes schedules and tasks. This enables efficient management of schedules and tasks for all family members, smooth communication, and the provision of appropriate advice based on health information.

[0722] "Voice information" refers to information acquired via voice, which converts user instructions and inquiries into a digital format.

[0723] "Textual information" refers to information obtained as text, and involves processing user instructions and inquiries as strings of characters.

[0724] A "receiving device" is a device that has the function of acquiring audio and text information and transmitting it to the system in digital format.

[0725] An "information gathering device" is a device that centrally collects information on the activities and health status of household members and stores it for later analysis.

[0726] An "information analysis device" is a device that analyzes various types of collected information to identify patterns and trends, and contributes to improving household management.

[0727] A "notification device" is a device that provides users with information, either in the form of voice or text, for the purpose of reminding them of schedules and important matters.

[0728] A "dialogue device" is a device that facilitates communication between target users and allows them to exchange messages with each other.

[0729] A "health management device" is a device that analyzes collected health information and provides appropriate health guidance and exercise recommendations.

[0730] A "generation device" is a device that creates a digital report of the overall activity status of a household, making it available for use as reference material later on.

[0731] A "speech synthesis device" is a device that has the function of converting text information into speech and providing that information to the user.

[0732] A "recommendation device" is a device that provides users with personalized recommendations based on information obtained from health management devices.

[0733] This system utilizes a combination of hardware and software to facilitate information management and communication within the home. Specifically, a home robot functions as the primary device, integrating voice recognition, data analysis, and notification functions. This robot is equipped with a microphone and speaker, allowing it to receive, convert, analyze, and output voice information using speech synthesis.

[0734] The entire system is based on Python, using the SpeechRecognition library for speech recognition, Pandas for data management, Scikit-learn for machine learning, and gTTS for speech synthesis. These tools work together to process user-inputted voice and text information in real time, enabling schedule management, health monitoring, and timely notifications.

[0735] The server stores behavioral data from all household members in the cloud and performs multidimensional analysis to generate optimal schedules and health recommendations. This data is provided as reports and notifications generated according to user requests.

[0736] As a concrete example, on a holiday, a home robot might notify the user via voice, "Good morning, today's plan is breakfast with the family and a picnic in the park this afternoon. Are you ready?" It might also suggest, "The weather forecast for tomorrow is rain. Shall we look for ideas for indoor activities this weekend?" The user responds by voice, and the robot uses that information to retrieve more detailed information from the network and provide appropriate recommendations to the user through speech synthesis.

[0737] The AI ​​model for generating prompts will be operated based on instructions such as, "Generate a scenario where a family robot provides voice notifications about schedules and health advice. Include specific examples of dialogue for prioritizing and setting reminders."

[0738] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0739] Step 1:

[0740] The user enters a voice command. The device's microphone captures the voice, and the SpeechRecognition library is used to convert the voice data into text data. Voice input is performed in the form of "Schedule a meeting for 3pm tomorrow," and the output is a text-based instruction.

[0741] Step 2:

[0742] The terminal sends the converted character data to the server for information analysis. Upon receiving this information, the server uses Pandas to store the character data as structured data and processes it into a format that is easy to analyze. The input is character data, and the output is structured data stored as a database entry.

[0743] Step 3:

[0744] The server uses Scikit-learn's machine learning algorithms to analyze trends in the stored data and prioritize schedules. The input is database entries, and the output is a prioritized schedule list. This analysis extracts tasks that users should prioritize.

[0745] Step 4:

[0746] The server generates necessary reminders and notifications using speech synthesis, taking priority into consideration. This process converts text data into audio files using the gTTS library and sends them to the terminal. The input is a prioritized schedule list, and the output is an audio file.

[0747] Step 5:

[0748] The device's speaker plays the generated audio files to notify the user of appointments and reminders verbally. This allows the user to confirm schedules and tasks verbally. As output, the user receives verbal instructions and provides voice feedback as needed.

[0749] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0750] As an embodiment of the present invention, a family schedule and task management system incorporating an emotion engine is provided. This system, in which a server, terminals, and users work together, facilitates the daily lives of families.

[0751] Users input schedules and tasks via voice or text through their device. In the case of voice input, the device converts the input voice into text data, while simultaneously a background emotion engine analyzes the intonation and rhythm of the voice to evaluate the user's emotions. This emotion data is formatted along with the input data and sent to the server.

[0752] The server extracts emotional information from the received data and stores it in a database. This allows the server not only to manage schedules but also to dynamically adjust task priorities based on the user's emotions and individual circumstances. For example, if a user is feeling stressed, the server can set a lower priority for tasks and generate alerts encouraging them to take appropriate rest.

[0753] Furthermore, based on the emotions recognized by the emotion engine, the server can individually edit notifications and reminders, enabling communication tailored to the user's state. For example, if a user is feeling depressed, an encouraging message can be added to the notification, providing psychological support.

[0754] In terms of health management, the system integrates health and emotional data collected from smart devices and wearable devices to provide a more accurate understanding of health status and exercise suggestions. This system supports health management tailored to the user's emotions, for example, by suggesting relaxation exercises when the user is experiencing high levels of stress.

[0755] As a concrete example, suppose a user types "I'm going to dance class tomorrow" into their device. If the emotion engine detects that the user is enjoying themselves, the server will give that activity a high priority. Conversely, if anxiety is detected from the voice, support can be provided by sending helpful information and advice in advance.

[0756] Thus, the present invention aims to improve the quality of life within the home by utilizing an emotional engine to provide schedule management and support that is more tailored to each individual family member.

[0757] The following describes the processing flow.

[0758] Step 1:

[0759] Users enter schedules and tasks using their device via voice or text. In the case of voice input, the device converts the voice data into text, while simultaneously an emotion engine analyzes the characteristics of the voice to identify the user's emotions.

[0760] Step 2:

[0761] The device sends formatted data, including emotions, to the server. This data includes the date and time, location, task details, and emotion information.

[0762] Step 3:

[0763] The server analyzes the received data and stores it in a database. At this time, sentiment data is recorded along with schedule data, which can be used for future analysis.

[0764] Step 4:

[0765] The server analyzes the collected data using machine learning algorithms. By taking sentiment data into consideration, it is possible to dynamically set the priority of each task.

[0766] Step 5:

[0767] Based on the analysis results of the emotion engine, the server creates notifications and reminders. For example, if the user's emotions indicate stress, the server lowers the task's priority and sets a notification that includes suggestions to promote relaxation.

[0768] Step 6:

[0769] The device displays notifications and reminders sent from the server to the user. The content of the notifications is customized according to the user's emotional state.

[0770] Step 7:

[0771] Users can communicate with their families using the device's chat function. Appropriate communication is possible through timing and methods adjusted based on emotional information.

[0772] Step 8:

[0773] The server continuously collects health data from smart devices and wearable devices and monitors health status by combining it with emotional data. If the emotional engine detects significant stress, it will suggest health management measures that require special attention.

[0774] Step 9:

[0775] The server generates weekly or monthly reports based on activity data for the entire family and sends them to the device. These reports also reflect emotional states and include information that can help improve individual well-being.

[0776] (Example 2)

[0777] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0778] In modern family life, managing each member's schedule and tasks, as well as monitoring their health, is complex and difficult to address individually. Furthermore, flexible planning and support that takes into account each member's emotional state are required, but traditional systems have been ineffective in achieving this. Therefore, a system capable of improving the efficiency and quality of life for the entire family is desired.

[0779] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0780] In this invention, the server includes data analysis means for analyzing emotional data and evaluating the user's emotional state, priority setting means for prioritizing schedules and tasks, and emotional support means for providing psychological support based on the user's emotions. This makes it possible to manage individualized schedules and tasks that take into account the user's emotional state, and to provide effective psychological support.

[0781] "Information input means" refers to a device or function that allows a user to input schedules and tasks using voice or text.

[0782] "Data analysis means" refers to a device or program that has the processing capability to analyze information entered by a user and, in particular, to evaluate the user's emotional state based on emotional data.

[0783] A "memory device" is a system component that has a storage function for storing collected information.

[0784] A "priority setting means" is a function or device for setting priorities for schedules and tasks based on their importance and urgency.

[0785] "Notification generation means" refers to a function or device for generating and providing appropriate notification information to the user based on a set priority.

[0786] "Communication methods" refer to system components, including chat and messaging functions, used to facilitate the exchange of information within a household.

[0787] A "health management device" is a device or system that has the function of monitoring a person's health status and, based on that, suggesting exercises and lifestyle improvements for maintaining good health.

[0788] An "emotional support device" is a device or program that provides individualized support based on the user's emotional state.

[0789] A "portable information processing device" is an electronic device that is portable and equipped with the functions of data collection and processing.

[0790] A "wearable information processing device" is an electronic device that can be worn by a user and has the function of collecting and processing biometric and environmental information.

[0791] An "artificial intelligence algorithm" is a program or technology that enables a machine to perform analysis and inference from data, and to personalize future suggestions based on the user's past behavior.

[0792] This invention functions as a family schedule and task management system incorporating an emotion engine. Users input schedules and tasks using voice or text via a device such as a smartphone or tablet. The input voice data is converted to text using speech recognition software, and emotion analysis software simultaneously evaluates the user's emotions. Specific software used includes a "Voice Input API" for speech recognition and an "Emotion Analysis API" for emotion analysis.

[0793] The device sends this data to the server. Based on the received sentiment data and schedule information, the server dynamically adjusts task priorities using a priority setting algorithm. Furthermore, based on the user's emotional state, it utilizes a notification generation function to provide appropriate reminders and support messages.

[0794] In health management, biometric information obtained from wearable information devices worn by the user is analyzed and integrated with the user's emotional state to provide appropriate suggestions for exercise and health maintenance. For this purpose, data such as heart rate and activity levels are collected.

[0795] For example, if a user enters "I'm going jogging tomorrow" into their device, the emotion engine can detect the user's motivation, and the server will prioritize that activity. Conversely, if the emotion data indicates the user is feeling stressed, the system can provide a notification recommending exercise that promotes relaxation.

[0796] Generative AI models can also be used to generate specific messages that respond to the user's emotions. An example of a prompt might be, "Generate a motivational message based on the user's emotional state."

[0797] This system enables schedule management that takes into account each user's emotions and health condition, aiming to improve the quality of life for families.

[0798] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0799] Step 1:

[0800] Users input schedule and task information via voice or text through their device. In the case of voice input, the device uses speech recognition software to convert the voice into text. Here, voice data is input and converted into text data. This text data forms the basis for the next processing step.

[0801] Step 2:

[0802] The device uses emotion analysis software to evaluate the user's emotions from voice or text data. In this process, the device extracts emotion data from the intonation and rhythm of the voice and the content of the text, and outputs the emotion analysis results. This emotion data plays a crucial role in schedule management.

[0803] Step 3:

[0804] The terminal combines the converted text data and sentiment data, formats it, and sends it to the server. The input consists of text data and sentiment data, and structured data (e.g., in JSON format) containing these is transmitted to the server.

[0805] Step 4:

[0806] The server extracts schedule and sentiment information from the received data. A data analysis module then operates to analyze the user's state and the tasks they need. The output is the analysis results for prioritization and notification generation.

[0807] Step 5:

[0808] The server dynamically sets priorities based on user sentiment data. The priority setting algorithm lowers the priority of tasks when the user is experiencing stress, and conversely, gives higher priority to activities when positive emotions are detected. The input is the analysis results from earlier, and the output is updated priority information.

[0809] Step 6:

[0810] The server generates and sends notifications and reminders to users based on priority. Specifically, it uses a generative AI model to create messages and provides them to users as notifications. Priority information and sentiment data are used as input, and notification messages are generated as output.

[0811] Step 7:

[0812] The server acquires health data from wearable information processing devices and integrates it with emotional information. This data integration process involves a detailed assessment of the user's health status and the generation of exercise suggestions tailored to their needs. The inputs to this process are health information and emotional information, and the output is customized health suggestions.

[0813] (Application Example 2)

[0814] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0815] In daily family life, managing each individual's schedule and tasks often becomes complicated, and there is a lack of appropriate responses and communication tailored to each member's emotional state. Furthermore, health management often fails to consider the emotional state of each individual, creating challenges for the entire family to lead a richer and healthier life. Therefore, flexible schedule and health management that accommodates the emotional state of each member is necessary.

[0816] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0817] In this invention, the server includes a terminal device that accepts voice and text input, a data collection device that collects and stores behavioral data of family members, a data analysis device that analyzes the collected data to extract emotional data and prioritize schedules and tasks, a notification device that individually edits and sets reminders and notifications based on priority and emotional data, a dialogue device for facilitating electronic communication within the home, a health management device that collects and analyzes health-related data to provide health management and exercise suggestions, a reporting device that generates activity reports for the entire family, and an emotion response device that generates notifications and advice according to the user's emotional state. This enables schedule management tailored to each family member, as well as health management and appropriate communication according to their emotional state.

[0818] A "terminal device" is a device that receives input data from users via voice and text.

[0819] A "data collection device" is a device that has the function of collecting behavioral data of family members and storing that data.

[0820] A "data analysis device" is a device that extracts emotional data from collected behavioral data and processes it to prioritize schedules and tasks.

[0821] A "notification device" is a device that has the function of individually editing and setting reminders and notifications based on priority and sentiment data.

[0822] A "dialogue device" is a device that has functions to facilitate electronic dialogue within the home and to make communication smoother.

[0823] A "health management device" is a device that collects health-related data, analyzes it, and provides health management and exercise suggestions to the user.

[0824] A "reporting device" is a device that has the function of generating activity reports for the entire family.

[0825] An "emotional response device" is a device that generates notifications and advice according to the user's emotional state.

[0826] This system is a comprehensive management system designed to support the lives of family members. The terminal accepts data from users through voice and text input. In the case of voice input, an emotion engine is used to analyze the intonation and rhythm of the input voice data and evaluate the user's emotional state. The data collection device centrally collects and stores behavioral and emotional data from family members.

[0827] The server analyzes the collected data using a data analysis device and extracts sentiment data. Based on this analysis, it prioritizes schedules and tasks. The notification device then individually edits and delivers reminders and notifications to the user based on this priority and sentiment data.

[0828] The dialogue device has the function of generating natural conversations using a generative AI model to facilitate electronic conversations within the home. The health management device analyzes health-related data collected from portable and wearable devices to more accurately understand the user's health status and provides exercise suggestions based on emotions.

[0829] For example, if a user voice-inputs "I'm going jogging tomorrow," the device can sense that the user is enjoying the activity. The server then sets a high priority for that activity and sends information to the user via a notification device. In this way, appropriate support tailored to the user's state is provided.

[0830] An example of a prompt for a generative AI model is: "Design a program that uses an emotion engine to implement a way to manage household schedules and tasks. Provide an example where the robot suggests specific actions based on the emotional state of each family member."

[0831] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0832] Step 1:

[0833] Users input schedule information via voice or text through their device. Voice input is converted to text using speech recognition. During this process, the device utilizes an emotion engine to analyze the intonation and rhythm of the voice and generate emotion data. The input data is then prepared as both text and emotion data.

[0834] Step 2:

[0835] The terminal transmits prepared text information and sentiment data to the data collection device. The data collection device stores the input behavioral and sentiment data in a database. This stored data, which also includes each user's past behavioral history, is used for later analysis.

[0836] Step 3:

[0837] The server uses a data analysis device to analyze behavioral and emotional data in the database. This data analysis employs machine learning algorithms to prioritize user schedules and tasks based on their emotions. The analysis uses past behavioral data and new emotional data as input, and outputs schedule data with assigned priorities.

[0838] Step 4:

[0839] The server generates reminders and notifications through the notification device based on the analysis results. These generated notifications include personalized messages tailored to the user's emotional state. For example, if the user is feeling stressed, a message such as "Take it easy today and get some rest" might be sent.

[0840] Step 5:

[0841] The dialogue device uses a generative AI model to revitalize communication within the home. The generative AI model utilizes prompts to generate natural dialogue. During this process, the dialogue output incorporates emotional data, providing the user with appropriate feedback and advice.

[0842] Step 6:

[0843] The health management device analyzes health-related data obtained from portable and wearable devices. The server integrates emotional data and health-related data to obtain information to understand the user's health status. Based on this information, appropriate exercise suggestions are provided to the user via a notification device.

[0844] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0845] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0846] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0847] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0848] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0849] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0850] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0851] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0852] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0853] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0854] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0855] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0856] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0857] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0858] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0859] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0860] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0861] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0862] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0863] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0864] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0865] The following is further disclosed regarding the embodiments described above.

[0866] (Claim 1)

[0867] An input method that accepts voice input and text input,

[0868] A data collection method for collecting and storing behavioral data for all family members,

[0869] A data analysis method that analyzes collected data and prioritizes schedules and tasks,

[0870] A notification method that sets reminders and notifications based on priority,

[0871] A chat tool to facilitate communication within the family,

[0872] A health management tool that collects and analyzes health data to provide health management and exercise suggestions,

[0873] A system that includes a reporting mechanism for generating activity reports for the entire family.

[0874] (Claim 2)

[0875] The system according to claim 1, wherein health data is collected from smart devices and wearable devices.

[0876] (Claim 3)

[0877] The system according to claim 1, comprising a machine learning algorithm that customizes the suggested content using the user's past behavior data.

[0878] "Example 1"

[0879] (Claim 1)

[0880] An information input method that accepts voice input and text input,

[0881] Information gathering means for collecting and storing information on the actions of all individuals,

[0882] Information analysis tools that analyze collected information and prioritize schedules and tasks,

[0883] A notification generation method that sets reminders and notifications based on priority,

[0884] A messaging method to facilitate information transmission within a group,

[0885] A health management tool that collects and analyzes health information to provide health management and exercise suggestions,

[0886] Support tools that provide health advice based on individual health information,

[0887] A system that includes a report generation method for generating activity reports for the entire organization.

[0888] (Claim 2)

[0889] The system according to claim 1, wherein health information is collected from smart devices and wearable devices.

[0890] (Claim 3)

[0891] The system according to claim 1, comprising machine learning technology that customizes the suggested content using an individual's past behavioral information.

[0892] "Application Example 1"

[0893] (Claim 1)

[0894] A receiving device that accepts audio and text information,

[0895] An information collection device that collects and stores activity information of all household members,

[0896] An information analysis device that analyzes collected information and prioritizes schedules and tasks,

[0897] A notification device that sets alerts and notifications based on priority,

[0898] A dialogue device to facilitate information exchange within the home,

[0899] A health management device that collects and analyzes health information to provide health guidance and exercise recommendations,

[0900] A generator that generates activity data for the entire household,

[0901] A speech synthesis device that provides information through voice output,

[0902] A system that includes a recommendation device that provides personalized recommendations based on healthcare information.

[0903] (Claim 2)

[0904] The system according to claim 1, wherein health information is collected from a mobile device and a wearable electronic device.

[0905] (Claim 3)

[0906] The system according to claim 1, comprising an artificial intelligence algorithm that personalizes recommendations using the user's past activity information.

[0907] "Example 2 of combining an emotion engine"

[0908] (Claim 1)

[0909] An information input method that accepts voice input and text input,

[0910] A data analysis method that analyzes emotional data and evaluates the user's emotional state,

[0911] A storage means for saving the collected information,

[0912] Prioritization methods for setting priorities for schedules and tasks,

[0913] A notification generation means that generates notification information based on priority,

[0914] Communication means to facilitate communication within the home,

[0915] A health management tool that collects and analyzes health information to manage health status and propose exercise,

[0916] A system that includes emotional support tools to provide psychological support based on the user's emotions.

[0917] (Claim 2)

[0918] The system according to claim 1, wherein health information is collected from a portable information processing device and a wearable information processing device.

[0919] (Claim 3)

[0920] The system according to claim 1, comprising an artificial intelligence algorithm that personalizes the suggested content using the user's past behavior information.

[0921] "Application example 2 when combining with an emotional engine"

[0922] (Claim 1)

[0923] A terminal device that accepts voice input and text input,

[0924] A data collection device that collects and stores behavioral data of family members,

[0925] A data analysis device that analyzes collected data to extract emotional data and prioritizes schedules and tasks,

[0926] A notification device that allows users to individually edit and set reminders and notifications based on priority and sentiment data,

[0927] A dialogue device to facilitate electronic communication within the home,

[0928] A health management device that collects and analyzes health-related data to provide health management and exercise suggestions,

[0929] A reporting device that generates activity reports for the entire family,

[0930] A system that includes an emotion-responsive device that generates notifications and advice according to the user's emotional state.

[0931] (Claim 2)

[0932] The system according to claim 1, wherein health-related data is collected from portable devices and wearable devices.

[0933] (Claim 3)

[0934] The system according to claim 1, comprising a predictive algorithm that personalizes the suggested content based on the user's past behavioral data, and dynamically adjusts the notification content using sentiment data. [Explanation of Symbols]

[0935] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A receiving device that accepts audio and text information, An information collection device that collects and stores activity information of all household members, An information analysis device that analyzes collected information and prioritizes schedules and tasks, A notification device that sets alerts and notifications based on priority, A dialogue device to facilitate information exchange within the home, A health management device that collects and analyzes health information to provide health guidance and exercise recommendations, A generator that generates activity data for the entire household, A speech synthesis device that provides information through voice output, A system that includes a recommendation device that provides personalized recommendations based on healthcare information.

2. The system according to claim 1, wherein health information is collected from a mobile device and a wearable electronic device.

3. The system according to claim 1, comprising an artificial intelligence algorithm that personalizes recommendations using the user's past activity information.

Images