system

Abstract

We provide the system. [Solution] A means of inputting user schedule information and saving it to a database, A means for generating notifications at a predetermined time based on a database and sending them to each user terminal, A means of presenting notifications received by the user's device via voice or text, prompting the user to take appropriate action, A means of collecting user movement data using sensors and location information technology, A means for analyzing collected operational data to detect anomalies and generating a warning when an anomaly is detected, A means of sending warnings to users and their administrators to ensure security, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to the description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Dementia patients have a poor sense of time and tend to forget important daily activities and schedules. Also, they tend to become confused when they don't know what action to take next, and their self-sufficiency in daily life may decline. Furthermore, ensuring safety is important, especially paying attention to the possibility of unexpected going out and accidents. These problems impose a great burden not only on the patients but also on their families and caregivers. Therefore, a system for supporting the daily life of dementia patients is required.

Means for Solving the Problems

[0005] This invention provides a means for supporting dementia patients by storing schedule information in a database and generating and sending notifications to the user's terminal at predetermined times. The user's terminal also presents notifications via voice or text to prompt the user to take appropriate action. Furthermore, it ensures safety by collecting user movement data using sensors and location information technology, analyzing the data with an AI algorithm, and generating warnings when an anomaly is detected, notifying the user and their administrator. In addition, it includes means for breaking down and storing daily tasks and guiding the user to the next task via voice or video as needed, thereby supporting the user in living an independent daily life.

[0006] "Users" refers to dementia patients and their supporters (family members or caregivers) who use the system.

[0007] "Schedule information" refers to information about the user's daily schedule and activities, and specifically refers to tasks that require time management, such as medication times and going out.

[0008] A "database" is an information management system that systematically stores schedule information and daily task procedures, and retrieves them as needed.

[0009] A "notification" is a scheduled alert or reminder message, which is information sent to prompt a user to take a specific action.

[0010] "User device" refers to electronic devices such as smartphones and wearable devices carried by the user, which are responsible for receiving and displaying notifications from the system.

[0011] A "sensor" is a device or technology installed in a user's terminal to collect location information and behavioral data of the user.

[0012] "Location information technology" refers to technologies that use GPS and other methods to determine a user's current location.

[0013] "Action data" refers to data about the user's behavior and movement, and is necessary information for detecting abnormal behavior.

[0014] An "AI algorithm" is an artificial intelligence technology used to analyze collected behavioral data and detect patterns that are different from the norm.

[0015] "Detecting anomalies" refers to the process of identifying behaviors that deviate from established normal behavioral patterns.

[0016] A "warning" is a message or notification sent to users or administrators to draw their attention to an anomaly that has been detected.

[0017] "Daily tasks" refer to a series of tasks related to the user's daily activities and behaviors, and in particular, to actions that should be supported in a standardized manner.

[0018] "Means" refers to the technical elements or methods used to achieve a specific function within a system. [Brief explanation of the drawing]

[0019] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0021] First, the language used in the following description will be explained.

[0022] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0023] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0024] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0025] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0027] [First Embodiment]

[0028] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0029] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0032] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0035] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0039] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0040] This invention provides a system to support the lives of dementia patients and enhance their independence, and offers an application that can be implemented on smartphones and wearable devices. The following describes each component of the system and the related processes.

[0041] First, the user enters their schedule information into the application. This information includes medication times, meal times, and daily appointments. This information is sent to and stored on a server in the cloud. The server generates notifications based on the set times and sends them to the user's device.

[0042] Next, the device presents received notifications to the user via voice or text. For example, when it's time to take medication, it will give a voice notification such as, "It's time to take your medicine." This helps users remember to perform important daily tasks.

[0043] Furthermore, as a guide for daily life, users can register the steps for their daily tasks on the device. The server receives this information, breaks down the task into specific steps, and saves them. When requested by the user, the device will guide them through the next task using voice or video. For example, it might say, "Let's wash your face," indicating the next step.

[0044] Furthermore, to ensure security, the device uses sensors and location technology to collect user behavior data. This data is periodically sent to a server and analyzed by an AI algorithm. If an abnormal behavior pattern (for example, going out late at night) is detected, the server will issue a warning to the user and their administrator.

[0045] This allows users to live safely and independently, reducing the burden on family members and caregivers. Furthermore, the system is flexibly customizable, allowing it to provide appropriate support according to the user's needs.

[0046] The following describes the processing flow.

[0047] Step 1:

[0048] Users input their daily schedule information into applications on their smartphones or wearable devices. This includes things like medication times and planned outings.

[0049] Step 2:

[0050] The terminal sends the entered schedule information to a server in the cloud. The transmitted data is stored in the server's database.

[0051] Step 3:

[0052] The server monitors the stored schedule information and prepares to generate notifications as a specific time approaches.

[0053] Step 4:

[0054] When the scheduled time arrives, the server generates a notification based on the schedule and sends it to each user's device.

[0055] Step 5:

[0056] The device notifies the user of received notifications via voice or text. For example, it might display an alert such as, "It's time to take your medicine."

[0057] Step 6:

[0058] The user will take the necessary action (e.g., take medication) according to the notification.

[0059] Step 7:

[0060] Users register the steps for their daily tasks (e.g., getting ready in the morning) on ​​their device.

[0061] Step 8:

[0062] The server breaks down registered tasks into steps and stores them in the database. This process clarifies the specific action steps.

[0063] Step 9:

[0064] When a user requests guidance for a specific task, the device will guide them through the next steps using voice or video. For example, it might say, "Next, let's wash your face."

[0065] Step 10:

[0066] To monitor the user's daily activities, the device collects location information and motion data using sensors and GPS.

[0067] Step 11:

[0068] The collected data is periodically sent to a server. The server uses AI algorithms to analyze and detect abnormal behavioral patterns.

[0069] Step 12:

[0070] If an anomaly is detected, the server will issue a warning to the user and their administrator. This warning will be presented via voice or message through the terminal.

[0071] (Example 1)

[0072] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0073] In modern society, with the advancement of an aging society, there is a growing need for people with cognitive decline to live safely and independently. However, these individuals are more likely to forget important tasks in their daily lives or exhibit abnormal behavior, which increases the burden on both the individuals themselves and their caregivers. The present invention aims to provide a system that supports the daily lives of people with cognitive decline and ensures their safety.

[0074] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0075] In this invention, the server includes means for inputting user activity plan information and storing it in an information storage unit, means for generating notifications at a predetermined time based on the information storage unit and transmitting them to each user device, and means for analyzing collected operation information to detect abnormalities and generating warnings when abnormalities are detected. This enables users to act independently without forgetting important daily tasks and to live safely by responding immediately when abnormal behavior is detected.

[0076] "User" refers to an individual who uses the system of the present invention, and may particularly include individuals with cognitive impairment.

[0077] "Activity plan information" refers to information about the plans and tasks that users intend to carry out in their daily lives.

[0078] The term "information storage unit" refers to a storage system that stores and manages input data in chronological order.

[0079] "Notification" refers to messages or information that are sent to users based on a set schedule.

[0080] "User device" refers to a terminal device or wearable device carried by the user, used to receive notifications from the system.

[0081] "Motion information" refers to data related to the user's movement and activities, and is collected using sensors and location information technology.

[0082] A "warning" refers to a cautionary message sent to users and their administrators when the system detects an anomaly.

[0083] A "machine learning algorithm" refers to a computational method used to analyze patterns based on collected behavioral information and detect anomalies.

[0084] This invention is a system designed to support the daily lives of people with cognitive impairment and ensure their safety. The system primarily functions as a server, terminals, and users.

[0085] The server receives user activity plan information and stores it in a cloud-based information storage unit. Typically, a cloud storage system is used for this purpose. Based on the stored information, notifications are generated according to specific timeframes and sent to each terminal. A messaging protocol prioritizing real-time delivery is used for sending these notifications.

[0086] The terminal receives notifications from the server and presents them to the user in audio or text format. It can also prompt the user for the next necessary action based on their response. For this purpose, the terminal is equipped with speech recognition and speech synthesis capabilities. For example, to help users remember when to take their medication, it can provide reminders such as, "It's 8 o'clock. It's time to take your medicine."

[0087] Users manually input their activity plan information into the device. This allows them to visually check their daily schedule and manage their actions through notifications from the device. Activity plan information could include basic daily routines such as "8:00: Take medicine" and "12:00: Have lunch."

[0088] Furthermore, the system collects user behavior information using sensors and GPS built into the device. This behavior information is analyzed on a server, and if an abnormal pattern is detected, a warning is immediately generated. Machine learning algorithms are used to analyze behavior patterns, enabling early detection of anomalies.

[0089] For example, when managing a user's daily tasks in a structured manner, if the user enters "morning preparations" as a task, the device can provide voice guidance such as "Let's wash your face" and "Next, let's brush our teeth."

[0090] Examples of prompts include, "Please provide an overview of the daily life support system for dementia patients," and "Please explain in detail how user schedule information is processed." These prompts clarify the information that stakeholders will handle through the generated AI model, thereby improving the system's overall quality.

[0091] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0092] Step 1:

[0093] Users input activity plan information using an application installed on their smartphone or wearable device. This input information includes things like medication times and meal times. Specifically, they enter their schedule in text format into a form on the screen. Once input is complete, the data is sent from the device to the server. The output is the entered activity plan information data.

[0094] Step 2:

[0095] The server stores the received activity plan information in a cloud-based information storage unit. The storage system used is a general-purpose database, and normalization is performed to ensure data integrity and availability. After being stored in the database, this data becomes the basis for generating notifications based on the set time. The output is the plan information stored in the database.

[0096] Step 3:

[0097] The server generates a notification as a predetermined time approaches, based on the activity plan information stored in the information storage unit. The notification content is processed into a message related to the target task and sent to each terminal using a real-time messaging protocol. The output is the notification message sent to the user's terminal.

[0098] Step 4:

[0099] The terminal immediately displays notifications received from the server to the user. The display method is either voice or text information, selected according to the user's settings. A speech synthesis engine is used, and the voice message "It's time to take your medicine" is delivered. The output is notification information for the user.

[0100] Step 5:

[0101] The device periodically collects user activity information using built-in sensors and GPS functionality. This includes distance traveled and current location coordinates. This data is encrypted and sent to a server. The output is activity data collected from sensors and GPS.

[0102] Step 6:

[0103] The server receives collected operational data and uses machine learning algorithms to analyze abnormal patterns. The boundary between normal and abnormal is dynamically set based on past analysis data. As a result, if an abnormality such as leaving the premises late at night is detected, a warning is generated. The output consists of the abnormality detection result and the warning message.

[0104] Step 7:

[0105] The server sends out warnings to users and their administrators when an anomaly is detected. These warnings are sent via email or SMS, ensuring immediate notification. This warning system facilitates a quick response and ensures security.

[0106] (Application Example 1)

[0107] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0108] The objective of this invention is to provide support that enables elderly people and dementia patients to live their daily lives more safely and independently. In particular, it is necessary to prevent confusion and dangerous situations during shopping activities in physical stores, and to enable users to conduct purchasing activities efficiently. Furthermore, it is also important to enable immediate and appropriate responses in the event of abnormal behavior.

[0109] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0110] In this invention, the server includes means for inputting user activity information and storing it in an information management device, means for generating notifications at predetermined times based on the information management device and transmitting them to each user device, and means for monitoring the user's behavior within the store and notifying the store manager if an abnormality occurs. This makes it possible to improve the safety and efficiency of shopping activities for the elderly and dementia patients in physical stores.

[0111] "User" refers to an individual who uses the system to receive support for their daily activities.

[0112] "Activity information" refers to data about the plans and necessary tasks that users perform in their daily lives and purchasing activities.

[0113] An "information management device" is a digital device that manages a user's schedule and tasks and provides information when needed.

[0114] "Generating a notification" refers to the process of creating information that prompts a user to take a specific action.

[0115] "Devices used" refers to electronic devices such as smartphones, tablets, and wearable devices that users carry with them.

[0116] A "sensor" refers to a sensor device used to monitor a user's physical movements and health condition.

[0117] "Location data technology" refers to technology used to determine the current geographical location of a user or object.

[0118] An "analysis device" refers to a computing device used to analyze collected data and detect anomalies.

[0119] A "warning signal" refers to warning information generated when the system detects an anomaly.

[0120] A "purchase catalog" refers to a list of items that a user intends to purchase at a physical store.

[0121] "Product information" refers to information that provides users with directions or location information for products they intend to purchase, either through audio or video.

[0122] A "store manager" refers to the person responsible for overseeing operations at a physical store.

[0123] "Machine learning technology" is a method of learning patterns using large amounts of data, and is a technology used for anomaly detection and other applications.

[0124] A system implementing this invention includes a series of functions for managing user activity information and ensuring safety and efficiency during purchasing activities.

[0125] The server stores user-entered activity information and purchase lists in a cloud-based information management system. This ensures data security and accessibility when needed. The server then generates and sends notifications to user devices based on a predefined schedule. These notifications include information to support purchasing activities; for example, they can inform users of the time when they should pick up a specific product.

[0126] The device being used, such as a smartphone or wearable device, will present this notification to the user via voice or text. This allows the user to know when to take the specified action. In particular, in physical stores, location data technology can be used to guide users to the location of specific products. This allows users to effectively find the products they are looking for.

[0127] Furthermore, the sensors collect user activity information in real time and transmit it to a server. The server analyzes this information using an analysis device, and if an abnormal pattern is detected, it immediately generates a warning signal using machine learning technology and notifies the store manager. In this process, AI models are used to achieve highly accurate anomaly detection.

[0128] As a concrete example, consider a scenario where a user is shopping at a supermarket. If the user is looking for milk, the device will guide them by saying, "The milk section is on the refrigerated shelves on the left." If the user remains stationary for an extended period, the server analyzes this information and, if necessary, sends a warning signal to store staff.

[0129] An example of a prompt to use with a generative AI model is: "Suggest a way to guide the user to effectively find items on their shopping list. How would you support the user using in-store location data?"

[0130] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0131] Step 1:

[0132] Users input daily activity information and purchase lists into their devices. The input data is then transmitted from the device to a server. The server stores this data in a cloud-based information management system. The input data includes schedule information and product lists, which are stored in a database for use in subsequent processing.

[0133] Step 2:

[0134] The server generates notifications to support the user's purchasing behavior based on stored activity information. Schedule data is used to generate notifications, configuring instructions for picking up specific products at set times. The generated notifications are sent to the user's device. The output here is a timely notification regarding purchasing activity.

[0135] Step 3:

[0136] The device presents notifications received from the server to the user via voice or text. This presentation is done through the device's built-in voice output function or display. The information presented is intended to help the user know the location and purchase timing of a specific product, and the output is in the form of voice messages or text displays.

[0137] Step 4:

[0138] User movement information is acquired through sensors connected to the terminal. In this process, user movement and location data are collected in real time by sensors and recorded on the terminal. The input consists of physical location and movement information, which is stored and processed into data to be sent to the server.

[0139] Step 5:

[0140] The server analyzes the behavioral information transmitted from the terminal using an analysis device. Here, an abnormal pattern is detected using a generative AI model. For example, if the device remains motionless for a long period of time or if an unusual movement pattern is observed, it is judged to be abnormal. The input for the analysis is behavioral data from sensors, and the output is a judgment result indicating whether or not an abnormality is present.

[0141] Step 6:

[0142] If an anomaly is detected, the server generates an alert signal and sends a notification to the store manager. This is to prompt immediate action upon anomaly detection, allowing store staff to provide verification and assistance as needed. The input is the anomaly detection result, and the output is the alarm message.

[0143] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0144] This invention provides a system that combines an emotion engine with a system for supporting dementia patients, thereby offering appropriate support and safety supervision tailored to the user's mental state. The system is provided as an application implemented on smartphones and wearable devices.

[0145] When using the system, users enter their schedule information into the application. This information is sent to the server via the terminal and stored in a database. Based on the entered schedule information, the server generates notifications at the necessary times and sends them to the user's terminal.

[0146] The device that receives the notification will present the information to the user via voice or text, guiding them on what to do next. For example, if it's mealtime, it might say, "It's time to start eating." These notifications prevent user confusion and promote independence in daily life.

[0147] This system incorporates a function to collect user behavior data using sensors and location technology. The terminal periodically sends this data to a server, where it is analyzed using AI algorithms. If an abnormal behavior pattern is detected, the server issues a warning to the user and their administrator to ensure security.

[0148] Another distinctive feature is the emotion engine built into the device. The emotion engine analyzes the user's facial expressions and voice tone through the camera and voice input, collecting and storing emotional data. Using this data, the server determines the user's emotional state and provides notifications and support accordingly.

[0149] For example, if the emotion engine determines that a user is feeling down, the device will provide an encouraging message such as, "Let's take a short walk to cheer you up." Emotional data is also integrated with anomaly detection algorithms to enable more accurate safety monitoring.

[0150] This allows for comprehensive support that takes into account the emotional well-being of users, helping them to achieve a safe and independent life.

[0151] The following describes the processing flow.

[0152] Step 1:

[0153] Users input their daily schedule information (e.g., medication times, planned outings) into an application on their smartphone or wearable device.

[0154] Step 2:

[0155] The terminal sends the entered schedule information to the server and stores it in the database.

[0156] Step 3:

[0157] The server manages schedule information, generates notifications according to predetermined times, and prepares them to be sent to the user's terminal.

[0158] Step 4:

[0159] The device notifies the user of received notifications via voice or text. For example, it might announce, "It's time to take your medicine."

[0160] Step 5:

[0161] At the same time, the device uses its camera and microphone to collect the user's facial expressions and voice data.

[0162] Step 6:

[0163] The emotion engine built into the device analyzes the collected data and evaluates the user's emotional state (e.g., joy, sadness, stress).

[0164] Step 7:

[0165] The emotion engine evaluates the user's emotional state and sends the results to the server.

[0166] Step 8:

[0167] Based on the received emotional data, the server determines how to respond to the user and what additional notifications to send. For example, if the user is feeling stressed, it might suggest, "Take a deep breath to relax."

[0168] Step 9:

[0169] Meanwhile, the device continues to collect operational data using sensors and location information technology and transmit it to the server.

[0170] Step 10:

[0171] The server uses AI algorithms to integrate behavioral and emotional data to detect abnormal behavior.

[0172] Step 11:

[0173] If an anomaly is detected, the server immediately sends a warning to the user and their administrator, and displays a message on the terminal prompting them to take specific action.

[0174] Step 12:

[0175] Users will follow the guidance provided by the device and contact family members or caregivers as needed.

[0176] Through the above process, we support the safety and emotional well-being of users simultaneously, achieving more comprehensive life support than ever before.

[0177] (Example 2)

[0178] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0179] In an aging society, a system is needed that accurately supports the daily activities and mental state of people with cognitive decline in order to help them live safe and independent lives. However, conventional technology has limitations in taking into account the emotional state of users and in its ability to immediately detect abnormal behavior and send warnings. Therefore, there is a need for means to provide more comprehensive support.

[0180] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0181] In this invention, the server includes means for inputting the user's schedule information into an information device and storing it on an information recording medium; means for generating notifications at a predetermined time based on the information recording medium and communicating them to each information terminal; and means for collecting emotional data via a camera and voice input, which the information device then analyzes. This makes it possible to comprehensively support the user's behavior and emotional state and promote a safe and independent life.

[0182] An "information device" is a digital device used by users to input schedule information and manage various types of data.

[0183] An "information recording medium" is a storage device or database used to store data entered by a user or data generated by a system.

[0184] An "information terminal" is a device used to provide users with notifications from a server, and it has the functionality to receive and display notifications.

[0185] A "detector" refers to a measuring device or sensor installed to collect user behavior data.

[0186] "Geographic information technology" is a technology used to track a user's movement and location based on location information.

[0187] A "machine learning algorithm" is an algorithm that analyzes collected data, recognizes patterns, and identifies anomalies.

[0188] "Emotional data" refers to data collected through cameras and voice input that indicates the user's emotions and mental state.

[0189] "Immediately communicating warnings" means notifying users and administrators without delay when abnormal behavior or conditions are detected.

[0190] To implement this invention, smartphones and wearable devices are used as information devices to manage users' schedule information and activity data. In this process, the data must be securely stored in a cloud database, which is an information recording medium. A server connects to the information terminal via internet communication and generates notifications based on the schedule information received from the user. These notifications are sent to the information terminal at a pre-set time, allowing the user to confirm them visually or audibly.

[0191] The information terminal incorporates accelerometers and GPS as detectors, periodically collecting the user's location information and body movements. This data is transmitted to a server in combination with geographic information technology and analyzed using machine learning algorithms. If abnormal behavior is detected as a result of the analysis, the server immediately communicates a warning to the user and administrator to ensure security.

[0192] Furthermore, the information terminal is equipped with a high-performance camera and microphone, which analyze facial expressions and voice tone to collect emotional data. This allows the server to analyze the user's emotional state and generate appropriate responses. For example, if the user is feeling down, the server might generate a prompt such as "Suggesting activities to refresh you," providing encouragement to the user through the information terminal.

[0193] For example, if a user enters a walking schedule at 2:00 PM every day, the information device will generate a notification at 1:55 PM that says, "It's almost time for your walk. Let's start getting ready." Furthermore, if sentiment data reveals that the user is feeling stressed, it will offer advice such as, "Why not try listening to your favorite music while you walk?" An example of a prompt message would be an instruction such as, "Suggesting the next action to take."

[0194] With the above configuration, it is possible to comprehensively support the user's behavior and emotional state, enabling them to live a safe and independent life.

[0195] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0196] Step 1:

[0197] Users input their schedule information using smartphones or wearable devices. This input includes the date, time, and details of the schedule. The entered data is temporarily stored within the information device and prepared for transmission to the server.

[0198] Step 2:

[0199] The terminal transmits the schedule information entered by the user to the server via the internet. The server receives this data and stores it in a database for each user. The stored data is used for subsequent processing based on the scheduled time.

[0200] Step 3:

[0201] The server references the stored schedule information and generates notifications at the specified times. As part of the data processing, it creates notification content from the schedule information and adds it to the notification registration list. The generated notifications are formatted according to the notification format (audio or text) specified by the user.

[0202] Step 4:

[0203] The server sends the generated notification to the device. The device receives this notification and presents it to the user visually or audibly. Specifically, it displays a pop-up notification on the device screen and, if necessary, provides an audio alert. This allows the user to confirm and appropriately perform the next action.

[0204] Step 5:

[0205] The device uses built-in sensors and GPS technology to collect user movement data. This data includes motion information such as location, speed, and direction. The collected data is periodically transmitted to a server.

[0206] Step 6:

[0207] The server receives behavioral data sent from the terminal and analyzes it using machine learning algorithms. It identifies behavioral patterns from the input data and performs data calculations to identify abnormal behavior. If an anomaly is detected, it generates a warning for immediate action.

[0208] Step 7:

[0209] The server sends the generated warning to the user and their supervisor. The terminal displays this warning as a notification, prompting the user to check for safety. This allows users and supervisors to take prompt action.

[0210] Step 8:

[0211] The device uses a camera and voice input device to collect information about the user's emotions. It analyzes facial expressions and voice tone to determine the emotional state in real time. The emotional data is sent to a server.

[0212] Step 9:

[0213] The server analyzes emotional data and generates responses best suited to the user's mental state. Using a generative AI model, it creates appropriate prompts and suggestions based on the input emotional information. As a result, it recommends actions that contribute to improving the user's quality of life.

[0214] Step 10:

[0215] The server sends the generated responses and suggestions to the terminal and notifies the user. The terminal collects the user's responses and sends the feedback to the server to further improve the system's accuracy. This enables flexible support tailored to the user's needs.

[0216] (Application Example 2)

[0217] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0218] For dementia patients and the elderly, managing their daily lives and ensuring safety are significant challenges. Furthermore, the inability to respond appropriately to emotional fluctuations can exacerbate mental anxiety. Conventional support systems are limited to collecting movement data and providing schedule notifications, failing to adequately offer flexible support tailored to the user's emotional state.

[0219] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0220] In this invention, the server includes means for analyzing video and audio data using an emotion engine to estimate the user's emotional state, means for providing guidance to the user based on the estimated emotional state, and means for monitoring changes in the emotional state in real time and providing support for ensuring safety. This enables detailed support and safety supervision in response to fluctuations in the user's emotions.

[0221] "A means of inputting user schedule information and saving it to a database" refers to a function that provides an interface for users to input their schedules and activities, and efficiently stores that information in a database.

[0222] "A means of generating notifications at a predetermined time based on a database and sending them to each user's device" refers to a function that creates notifications at the appropriate time based on a pre-registered schedule and sends them to devices such as smart devices.

[0223] "A means of presenting notifications received by the user's device in audio or text and prompting the user to take appropriate action" refers to an interface that displays instructions received by the user on their device in audio or text format, guiding them to the next action.

[0224] "Means for collecting user motion data using sensors and location information technology" refers to a function that uses technologies such as sensors and GPS to acquire information about the user's body movements and location.

[0225] "Means for analyzing collected operational data to detect anomalies and generating warnings when an anomaly is detected" refers to a function that analyzes operational data to identify patterns that are different from the norm and creates a warning message when necessary.

[0226] "Means of sending warnings to users and their administrators to ensure security" refers to communication methods for notifying users and administrators of urgent situations and prompting a swift response.

[0227] "A means of analyzing video and audio data using an emotion engine to estimate the user's emotional state" refers to a function that analyzes data acquired from cameras and microphones to evaluate and judge the user's emotions.

[0228] "Means of providing guidance to users based on estimated emotional states" refers to functions that provide appropriate advice and instructions in accordance with the user's emotions.

[0229] "A means of monitoring changes in emotional state in real time and providing support for ensuring safety" refers to a system that quickly detects fluctuations in emotions and provides support for ensuring safety based on those fluctuations.

[0230] The system implementing this invention consists of a group of smart devices used by dementia patients and the elderly to ensure self-management and safety. At the core of the system is a server equipped with an emotion engine. The server receives the user's schedule information and stores it in a database. Based on the schedule, it generates notifications at predetermined times and sends them to the user's mobile device. This process enables the user to perform their daily activities without forgetting.

[0231] On the device, received notifications are presented to the user in voice or text format, prompting them to take their planned actions. The device incorporates sensors and location technology to collect user activity data. The server analyzes the collected data using AI algorithms and immediately generates a warning and notifies the administrator if an anomaly is detected.

[0232] Furthermore, because it is equipped with an emotion engine, it can estimate the user's emotional state in real time based on video and audio data acquired by the camera and microphone of smart glasses or wearable devices. Based on the estimated emotional state, the device provides personalized advice and guidance to the user. Technologies used in this process include OpenCV for image recognition and speech recognition APIs for speech analysis, while communication services such as Twilio are used for notifications.

[0233] For example, if a user appears lost while walking in a park, the device will ask, "Where do you want to go now?" and guide them in the correct direction by referring to their location information. Another example of a prompt using the generated AI model is, "Tell me a phrase to use when I forget where I want to go." This helps users to act with confidence even in difficult situations.

[0234] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0235] Step 1:

[0236] The user enters schedule information via a smart device. The device sends the entered data to the server. The server receives this data and stores it in a database. The input consists of schedule details and time, and the output is storage in the database.

[0237] Step 2:

[0238] The server generates notifications at predetermined times based on stored schedules. These notifications include the next tasks to be performed. The generated notifications are sent to the terminal. The input is schedule information from the database, and the output is a notification in the appropriate format.

[0239] Step 3:

[0240] The device presents received notifications to the user as audio or text, prompting the user to take the next action. The input is notification data from the server, and the output is the presentation of information to the user.

[0241] Step 4:

[0242] The device's sensors collect user movement data and location information in real time. This data is periodically transmitted to a server. The input is movement and location data from the sensors, and the output is the data transmission to the server.

[0243] Step 5:

[0244] The server analyzes the collected behavioral data using an AI algorithm. When an abnormal behavioral pattern is detected, it generates a warning and sends a notification to the user and their administrator. The input is behavioral and location data, and the output is a warning message.

[0245] Step 6:

[0246] The device collects video and audio data from the user using its camera and microphone and transmits it to the emotion engine. The emotion engine analyzes this data and estimates the emotional state. The input is video and audio data, and the output is an evaluation of the emotional state.

[0247] Step 7:

[0248] The server generates a guidance message based on the estimated emotional state and sends it to the terminal. The terminal then presents this guidance to the user. The input is emotional state data, and the output is the guidance message.

[0249] Step 8:

[0250] Users receive guidance through their devices and modify their actions as needed. Specifically, users can use the presented information to move around and complete tasks with confidence. The input is guidance messages, and the output is modified behavior.

[0251] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0252] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0253] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0254] [Second Embodiment]

[0255] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0256] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0257] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0258] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0259] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0260] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0261] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0262] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0263] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0264] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0265] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0266] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0267] This invention provides a system to support the lives of dementia patients and enhance their independence, and offers an application that can be implemented on smartphones and wearable devices. The following describes each component of the system and the related processes.

[0268] First, the user enters their schedule information into the application. This information includes medication times, meal times, and daily appointments. This information is sent to and stored on a server in the cloud. The server generates notifications based on the set times and sends them to the user's device.

[0269] Next, the device presents received notifications to the user via voice or text. For example, when it's time to take medication, it will give a voice notification such as, "It's time to take your medicine." This helps users remember to perform important daily tasks.

[0270] Furthermore, as a guide for daily life, users can register the steps for their daily tasks on the device. The server receives this information, breaks down the task into specific steps, and saves them. When requested by the user, the device will guide them through the next task using voice or video. For example, it might say, "Let's wash your face," indicating the next step.

[0271] Furthermore, to ensure security, the device uses sensors and location technology to collect user behavior data. This data is periodically sent to a server and analyzed by an AI algorithm. If an abnormal behavior pattern (for example, going out late at night) is detected, the server will issue a warning to the user and their administrator.

[0272] This allows users to live safely and independently, reducing the burden on family members and caregivers. Furthermore, the system is flexibly customizable, allowing it to provide appropriate support according to the user's needs.

[0273] The following describes the processing flow.

[0274] Step 1:

[0275] Users input their daily schedule information into applications on their smartphones or wearable devices. This includes things like medication times and planned outings.

[0276] Step 2:

[0277] The terminal sends the entered schedule information to a server in the cloud. The transmitted data is stored in the server's database.

[0278] Step 3:

[0279] The server monitors the saved schedule information and prepares to generate a notification when a specific time approaches.

[0280] Step 4:

[0281] When the set time arrives, the server generates a notification based on the schedule and sends it to each user terminal.

[0282] Step 5:

[0283] The terminal notifies the user of the received notification by voice or text. For example, it displays an alert such as "It's time to take your medicine."

[0284] Step 6:

[0285] The user performs the necessary actions (e.g., taking medicine) according to the notification.

[0286] Step 7:

[0287] The user registers the procedures for daily tasks (e.g., morning grooming) on the terminal.

[0288] Step 8:

[0289] The server decomposes the registered tasks into steps and saves them in the database. This process clarifies the specific action steps.

[0290] Step 9:

[0291] When the user requests guidance on a specific task, the terminal guides the next steps to be taken by voice or video. It provides instructions such as "Next, let's wash your face."

[0292] Step 10:

[0293] To monitor the user's daily activities, the device collects location information and motion data using sensors and GPS.

[0294] Step 11:

[0295] The collected data is periodically sent to a server. The server uses AI algorithms to analyze and detect abnormal behavioral patterns.

[0296] Step 12:

[0297] If an anomaly is detected, the server will issue a warning to the user and their administrator. This warning will be presented via voice or message through the terminal.

[0298] (Example 1)

[0299] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0300] In modern society, with the advancement of an aging society, there is a growing need for people with cognitive decline to live safely and independently. However, these individuals are more likely to forget important tasks in their daily lives or exhibit abnormal behavior, which increases the burden on both the individuals themselves and their caregivers. The present invention aims to provide a system that supports the daily lives of people with cognitive decline and ensures their safety.

[0301] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0302] In this invention, the server includes means for inputting the activity plan information of the user and storing it in the information storage unit, means for generating a notification at a predetermined time based on the information storage unit and transmitting it to each user device, and means for analyzing the collected operation information to detect an abnormality and generating a warning when an abnormality is detected. Thereby, it supports the user to be able to act independently without forgetting important daily tasks, and by immediately responding when an abnormal behavior is detected, it becomes possible to lead a safe life.

[0303] The "user" refers to an individual who uses the system of the present invention, and may particularly include people with a decline in cognitive function.

[0304] The "activity plan information" refers to information regarding the plans and tasks that the user intends to perform in daily life.

[0305] The "information storage unit" refers to a storage system that stores the input data and manages it in chronological order.

[0306] The "notification" refers to a message or information that notifies the user based on a set schedule.

[0307] The "user device" refers to a terminal device or wearable device carried by the user and is used to receive notifications from the system.

[0308] The "operation information" refers to data regarding the movement and activities of the user and is collected by sensors and location information technologies.

[0309] The "warning" refers to a warning message transmitted to the user and their administrator when the system detects an abnormality.

[0310] The "machine learning algorithm" refers to a computational method used to analyze patterns and detect abnormalities based on the collected operation information.

[0311] This invention is a system designed to support the daily lives of people with cognitive impairment and ensure their safety. The system primarily functions as a server, terminals, and users.

[0312] The server receives user activity plan information and stores it in a cloud-based information storage unit. Typically, a cloud storage system is used for this purpose. Based on the stored information, notifications are generated according to specific timeframes and sent to each terminal. A messaging protocol prioritizing real-time delivery is used for sending these notifications.

[0313] The terminal receives notifications from the server and presents them to the user in audio or text format. It can also prompt the user for the next necessary action based on their response. For this purpose, the terminal is equipped with speech recognition and speech synthesis capabilities. For example, to help users remember when to take their medication, it can provide reminders such as, "It's 8 o'clock. It's time to take your medicine."

[0314] Users manually input their activity plan information into the device. This allows them to visually check their daily schedule and manage their actions through notifications from the device. Activity plan information could include basic daily routines such as "8:00: Take medicine" and "12:00: Have lunch."

[0315] Furthermore, the system collects user behavior information using sensors and GPS built into the device. This behavior information is analyzed on a server, and if an abnormal pattern is detected, a warning is immediately generated. Machine learning algorithms are used to analyze behavior patterns, enabling early detection of anomalies.

[0316] For example, when managing a user's daily tasks in a structured manner, if the user enters "morning preparations" as a task, the device can provide voice guidance such as "Let's wash your face" and "Next, let's brush our teeth."

[0317] Examples of prompts include, "Please provide an overview of the daily life support system for dementia patients," and "Please explain in detail how user schedule information is processed." These prompts clarify the information that stakeholders will handle through the generated AI model, thereby improving the system's overall quality.

[0318] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0319] Step 1:

[0320] Users input activity plan information using an application installed on their smartphone or wearable device. This input information includes things like medication times and meal times. Specifically, they enter their schedule in text format into a form on the screen. Once input is complete, the data is sent from the device to the server. The output is the entered activity plan information data.

[0321] Step 2:

[0322] The server stores the received activity plan information in a cloud-based information storage unit. The storage system used is a general-purpose database, and normalization is performed to ensure data integrity and availability. After being stored in the database, this data becomes the basis for generating notifications based on the set time. The output is the plan information stored in the database.

[0323] Step 3:

[0324] The server generates a notification as a predetermined time approaches, based on the activity plan information stored in the information storage unit. The notification content is processed into a message related to the target task and sent to each terminal using a real-time messaging protocol. The output is the notification message sent to the user's terminal.

[0325] Step 4:

[0326] The terminal immediately displays notifications received from the server to the user. The display method is either voice or text information, selected according to the user's settings. A speech synthesis engine is used, and the voice message "It's time to take your medicine" is delivered. The output is notification information for the user.

[0327] Step 5:

[0328] The device periodically collects user activity information using built-in sensors and GPS functionality. This includes distance traveled and current location coordinates. This data is encrypted and sent to a server. The output is activity data collected from sensors and GPS.

[0329] Step 6:

[0330] The server receives collected operational data and uses machine learning algorithms to analyze abnormal patterns. The boundary between normal and abnormal is dynamically set based on past analysis data. As a result, if an abnormality such as leaving the premises late at night is detected, a warning is generated. The output consists of the abnormality detection result and the warning message.

[0331] Step 7:

[0332] The server sends out warnings to users and their administrators when an anomaly is detected. These warnings are sent via email or SMS, ensuring immediate notification. This warning system facilitates a quick response and ensures security.

[0333] (Application Example 1)

[0334] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0335] The objective of this invention is to provide support that enables elderly people and dementia patients to live their daily lives more safely and independently. In particular, it is necessary to prevent confusion and dangerous situations during shopping activities in physical stores, and to enable users to conduct purchasing activities efficiently. Furthermore, it is also important to enable immediate and appropriate responses in the event of abnormal behavior.

[0336] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0337] In this invention, the server includes means for inputting user activity information and storing it in an information management device, means for generating notifications at predetermined times based on the information management device and transmitting them to each user device, and means for monitoring the user's behavior within the store and notifying the store manager if an abnormality occurs. This makes it possible to improve the safety and efficiency of shopping activities for the elderly and dementia patients in physical stores.

[0338] "User" refers to an individual who uses the system to receive support for their daily activities.

[0339] "Activity information" refers to data about the plans and necessary tasks that users perform in their daily lives and purchasing activities.

[0340] An "information management device" is a digital device that manages a user's schedule and tasks and provides information when needed.

[0341] "Generating a notification" refers to the process of creating information that prompts a user to take a specific action.

[0342] "Devices used" refers to electronic devices such as smartphones, tablets, and wearable devices that users carry with them.

[0343] A "sensor" refers to a sensor device used to monitor a user's physical movements and health condition.

[0344] "Location data technology" refers to technology used to determine the current geographical location of a user or object.

[0345] An "analysis device" refers to a computing device used to analyze collected data and detect anomalies.

[0346] A "warning signal" refers to warning information generated when the system detects an anomaly.

[0347] A "purchase catalog" refers to a list of items that a user intends to purchase at a physical store.

[0348] "Product information" refers to information that provides users with directions or location information for products they intend to purchase, either through audio or video.

[0349] A "store manager" refers to the person responsible for overseeing operations at a physical store.

[0350] "Machine learning technology" is a method of learning patterns using large amounts of data, and is a technology used for anomaly detection and other applications.

[0351] A system implementing this invention includes a series of functions for managing user activity information and ensuring safety and efficiency during purchasing activities.

[0352] The server stores user-entered activity information and purchase lists in a cloud-based information management system. This ensures data security and accessibility when needed. The server then generates and sends notifications to user devices based on a predefined schedule. These notifications include information to support purchasing activities; for example, they can inform users of the time when they should pick up a specific product.

[0353] The device being used, such as a smartphone or wearable device, will present this notification to the user via voice or text. This allows the user to know when to take the specified action. In particular, in physical stores, location data technology can be used to guide users to the location of specific products. This allows users to effectively find the products they are looking for.

[0354] Furthermore, the sensors collect user activity information in real time and transmit it to a server. The server analyzes this information using an analysis device, and if an abnormal pattern is detected, it immediately generates a warning signal using machine learning technology and notifies the store manager. In this process, AI models are used to achieve highly accurate anomaly detection.

[0355] As a concrete example, consider a scenario where a user is shopping at a supermarket. If the user is looking for milk, the device will guide them by saying, "The milk section is on the refrigerated shelves on the left." If the user remains stationary for an extended period, the server analyzes this information and, if necessary, sends a warning signal to store staff.

[0356] An example of a prompt to use with a generative AI model is: "Suggest a way to guide the user to effectively find items on their shopping list. How would you support the user using in-store location data?"

[0357] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0358] Step 1:

[0359] Users input daily activity information and purchase lists into their devices. The input data is then transmitted from the device to a server. The server stores this data in a cloud-based information management system. The input data includes schedule information and product lists, which are stored in a database for use in subsequent processing.

[0360] Step 2:

[0361] The server generates notifications to support the user's purchasing behavior based on stored activity information. Schedule data is used to generate notifications, configuring instructions for picking up specific products at set times. The generated notifications are sent to the user's device. The output here is a timely notification regarding purchasing activity.

[0362] Step 3:

[0363] The device presents notifications received from the server to the user via voice or text. This presentation is done through the device's built-in voice output function or display. The information presented is intended to help the user know the location and purchase timing of a specific product, and the output is in the form of voice messages or text displays.

[0364] Step 4:

[0365] User movement information is acquired through sensors connected to the terminal. In this process, user movement and location data are collected in real time by sensors and recorded on the terminal. The input consists of physical location and movement information, which is stored and processed into data to be sent to the server.

[0366] Step 5:

[0367] The server analyzes the behavioral information transmitted from the terminal using an analysis device. Here, an abnormal pattern is detected using a generative AI model. For example, if the device remains motionless for a long period of time or if an unusual movement pattern is observed, it is judged to be abnormal. The input for the analysis is behavioral data from sensors, and the output is a judgment result indicating whether or not an abnormality is present.

[0368] Step 6:

[0369] If an anomaly is detected, the server generates an alert signal and sends a notification to the store manager. This is to prompt immediate action upon anomaly detection, allowing store staff to provide verification and assistance as needed. The input is the anomaly detection result, and the output is the alarm message.

[0370] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0371] This invention provides a system that combines an emotion engine with a system for supporting dementia patients, thereby offering appropriate support and safety supervision tailored to the user's mental state. The system is provided as an application implemented on smartphones and wearable devices.

[0372] When using the system, users enter their schedule information into the application. This information is sent to the server via the terminal and stored in a database. Based on the entered schedule information, the server generates notifications at the necessary times and sends them to the user's terminal.

[0373] The device that receives the notification will present the information to the user via voice or text, guiding them on what to do next. For example, if it's mealtime, it might say, "It's time to start eating." These notifications prevent user confusion and promote independence in daily life.

[0374] This system incorporates a function to collect user behavior data using sensors and location technology. The terminal periodically sends this data to a server, where it is analyzed using AI algorithms. If an abnormal behavior pattern is detected, the server issues a warning to the user and their administrator to ensure security.

[0375] Another distinctive feature is the emotion engine built into the device. The emotion engine analyzes the user's facial expressions and voice tone through the camera and voice input, collecting and storing emotional data. Using this data, the server determines the user's emotional state and provides notifications and support accordingly.

[0376] For example, if the emotion engine determines that a user is feeling down, the device will provide an encouraging message such as, "Let's take a short walk to cheer you up." Emotional data is also integrated with anomaly detection algorithms to enable more accurate safety monitoring.

[0377] This allows for comprehensive support that takes into account the emotional well-being of users, helping them to achieve a safe and independent life.

[0378] The following describes the processing flow.

[0379] Step 1:

[0380] Users input their daily schedule information (e.g., medication times, planned outings) into an application on their smartphone or wearable device.

[0381] Step 2:

[0382] The terminal sends the entered schedule information to the server and stores it in the database.

[0383] Step 3:

[0384] The server manages schedule information, generates notifications according to predetermined times, and prepares them to be sent to the user's terminal.

[0385] Step 4:

[0386] The device notifies the user of received notifications via voice or text. For example, it might announce, "It's time to take your medicine."

[0387] Step 5:

[0388] At the same time, the device uses its camera and microphone to collect the user's facial expressions and voice data.

[0389] Step 6:

[0390] The emotion engine built into the device analyzes the collected data and evaluates the user's emotional state (e.g., joy, sadness, stress).

[0391] Step 7:

[0392] The emotion engine evaluates the user's emotional state and sends the results to the server.

[0393] Step 8:

[0394] Based on the received emotional data, the server determines how to respond to the user and what additional notifications to send. For example, if the user is feeling stressed, it might suggest, "Take a deep breath to relax."

[0395] Step 9:

[0396] Meanwhile, the device continues to collect operational data using sensors and location information technology and transmit it to the server.

[0397] Step 10:

[0398] The server uses AI algorithms to integrate behavioral and emotional data to detect abnormal behavior.

[0399] Step 11:

[0400] If an anomaly is detected, the server immediately sends a warning to the user and their administrator, and displays a message on the terminal prompting them to take specific action.

[0401] Step 12:

[0402] Users will follow the guidance provided by the device and contact family members or caregivers as needed.

[0403] Through the above process, we support the safety and emotional well-being of users simultaneously, achieving more comprehensive life support than ever before.

[0404] (Example 2)

[0405] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0406] In an aging society, a system is needed that accurately supports the daily activities and mental state of people with cognitive decline in order to help them live safe and independent lives. However, conventional technology has limitations in taking into account the emotional state of users and in its ability to immediately detect abnormal behavior and send warnings. Therefore, there is a need for means to provide more comprehensive support.

[0407] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0408] In this invention, the server includes means for inputting the user's schedule information into an information device and storing it on an information recording medium; means for generating notifications at a predetermined time based on the information recording medium and communicating them to each information terminal; and means for collecting emotional data via a camera and voice input, which the information device then analyzes. This makes it possible to comprehensively support the user's behavior and emotional state and promote a safe and independent life.

[0409] An "information device" is a digital device used by users to input schedule information and manage various types of data.

[0410] An "information recording medium" is a storage device or database used to store data entered by a user or data generated by a system.

[0411] An "information terminal" is a device used to provide users with notifications from a server, and it has the functionality to receive and display notifications.

[0412] A "detector" refers to a measuring device or sensor installed to collect user behavior data.

[0413] "Geographic information technology" is a technology used to track a user's movement and location based on location information.

[0414] A "machine learning algorithm" is an algorithm that analyzes collected data, recognizes patterns, and identifies anomalies.

[0415] "Emotional data" refers to data collected through cameras and voice input that indicates the user's emotions and mental state.

[0416] "Immediately communicating warnings" means notifying users and administrators without delay when abnormal behavior or conditions are detected.

[0417] To implement this invention, smartphones and wearable devices are used as information devices to manage users' schedule information and activity data. In this process, the data must be securely stored in a cloud database, which is an information recording medium. A server connects to the information terminal via internet communication and generates notifications based on the schedule information received from the user. These notifications are sent to the information terminal at a pre-set time, allowing the user to confirm them visually or audibly.

[0418] The information terminal incorporates accelerometers and GPS as detectors, periodically collecting the user's location information and body movements. This data is transmitted to a server in combination with geographic information technology and analyzed using machine learning algorithms. If abnormal behavior is detected as a result of the analysis, the server immediately communicates a warning to the user and administrator to ensure security.

[0419] Furthermore, the information terminal is equipped with a high-performance camera and microphone, which analyze facial expressions and voice tone to collect emotional data. This allows the server to analyze the user's emotional state and generate appropriate responses. For example, if the user is feeling down, the server might generate a prompt such as "Suggesting activities to refresh you," providing encouragement to the user through the information terminal.

[0420] For example, if a user enters a walking schedule at 2:00 PM every day, the information device will generate a notification at 1:55 PM that says, "It's almost time for your walk. Let's start getting ready." Furthermore, if sentiment data reveals that the user is feeling stressed, it will offer advice such as, "Why not try listening to your favorite music while you walk?" An example of a prompt message would be an instruction such as, "Suggesting the next action to take."

[0421] With the above configuration, it is possible to comprehensively support the user's behavior and emotional state, enabling them to live a safe and independent life.

[0422] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0423] Step 1:

[0424] Users input their schedule information using smartphones or wearable devices. This input includes the date, time, and details of the schedule. The entered data is temporarily stored within the information device and prepared for transmission to the server.

[0425] Step 2:

[0426] The terminal transmits the schedule information entered by the user to the server via the internet. The server receives this data and stores it in a database for each user. The stored data is used for subsequent processing based on the scheduled time.

[0427] Step 3:

[0428] The server references the stored schedule information and generates notifications at the specified times. As part of the data processing, it creates notification content from the schedule information and adds it to the notification registration list. The generated notifications are formatted according to the notification format (audio or text) specified by the user.

[0429] Step 4:

[0430] The server sends the generated notification to the device. The device receives this notification and presents it to the user visually or audibly. Specifically, it displays a pop-up notification on the device screen and, if necessary, provides an audio alert. This allows the user to confirm and appropriately perform the next action.

[0431] Step 5:

[0432] The device uses built-in sensors and GPS technology to collect user movement data. This data includes motion information such as location, speed, and direction. The collected data is periodically transmitted to a server.

[0433] Step 6:

[0434] The server receives behavioral data sent from the terminal and analyzes it using machine learning algorithms. It identifies behavioral patterns from the input data and performs data calculations to identify abnormal behavior. If an anomaly is detected, it generates a warning for immediate action.

[0435] Step 7:

[0436] The server sends the generated warning to the user and their supervisor. The terminal displays this warning as a notification, prompting the user to check for safety. This allows users and supervisors to take prompt action.

[0437] Step 8:

[0438] The device uses a camera and voice input device to collect information about the user's emotions. It analyzes facial expressions and voice tone to determine the emotional state in real time. The emotional data is sent to a server.

[0439] Step 9:

[0440] The server analyzes emotional data and generates responses best suited to the user's mental state. Using a generative AI model, it creates appropriate prompts and suggestions based on the input emotional information. As a result, it recommends actions that contribute to improving the user's quality of life.

[0441] Step 10:

[0442] The server sends the generated responses and suggestions to the terminal and notifies the user. The terminal collects the user's responses and sends the feedback to the server to further improve the system's accuracy. This enables flexible support tailored to the user's needs.

[0443] (Application Example 2)

[0444] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0445] For dementia patients and the elderly, managing their daily lives and ensuring safety are significant challenges. Furthermore, the inability to respond appropriately to emotional fluctuations can exacerbate mental anxiety. Conventional support systems are limited to collecting movement data and providing schedule notifications, failing to adequately offer flexible support tailored to the user's emotional state.

[0446] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0447] In this invention, the server includes means for analyzing video and audio data using an emotion engine to estimate the user's emotional state, means for providing guidance to the user based on the estimated emotional state, and means for monitoring changes in the emotional state in real time and providing support for ensuring safety. This enables detailed support and safety supervision in response to fluctuations in the user's emotions.

[0448] "A means of inputting user schedule information and saving it to a database" refers to a function that provides an interface for users to input their schedules and activities, and efficiently stores that information in a database.

[0449] "A means of generating notifications at a predetermined time based on a database and sending them to each user's device" refers to a function that creates notifications at the appropriate time based on a pre-registered schedule and sends them to devices such as smart devices.

[0450] "A means of presenting notifications received by the user's device in audio or text and prompting the user to take appropriate action" refers to an interface that displays instructions received by the user on their device in audio or text format, guiding them to the next action.

[0451] "Means for collecting user motion data using sensors and location information technology" refers to a function that uses technologies such as sensors and GPS to acquire information about the user's body movements and location.

[0452] "Means for analyzing collected operational data to detect anomalies and generating warnings when an anomaly is detected" refers to a function that analyzes operational data to identify patterns that are different from the norm and creates a warning message when necessary.

[0453] "Means of sending warnings to users and their administrators to ensure security" refers to communication methods for notifying users and administrators of urgent situations and prompting a swift response.

[0454] "A means of analyzing video and audio data using an emotion engine to estimate the user's emotional state" refers to a function that analyzes data acquired from cameras and microphones to evaluate and judge the user's emotions.

[0455] "Means of providing guidance to users based on estimated emotional states" refers to functions that provide appropriate advice and instructions in accordance with the user's emotions.

[0456] "A means of monitoring changes in emotional state in real time and providing support for ensuring safety" refers to a system that quickly detects fluctuations in emotions and provides support for ensuring safety based on those fluctuations.

[0457] The system implementing this invention consists of a group of smart devices used by dementia patients and the elderly to ensure self-management and safety. At the core of the system is a server equipped with an emotion engine. The server receives the user's schedule information and stores it in a database. Based on the schedule, it generates notifications at predetermined times and sends them to the user's mobile device. This process enables the user to perform their daily activities without forgetting.

[0458] On the device, received notifications are presented to the user in voice or text format, prompting them to take their planned actions. The device incorporates sensors and location technology to collect user activity data. The server analyzes the collected data using AI algorithms and immediately generates a warning and notifies the administrator if an anomaly is detected.

[0459] Furthermore, because it is equipped with an emotion engine, it can estimate the user's emotional state in real time based on video and audio data acquired by the camera and microphone of smart glasses or wearable devices. Based on the estimated emotional state, the device provides personalized advice and guidance to the user. Technologies used in this process include OpenCV for image recognition and speech recognition APIs for speech analysis, while communication services such as Twilio are used for notifications.

[0460] For example, if a user appears lost while walking in a park, the device will ask, "Where do you want to go now?" and guide them in the correct direction by referring to their location information. Another example of a prompt using the generated AI model is, "Tell me a phrase to use when I forget where I want to go." This helps users to act with confidence even in difficult situations.

[0461] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0462] Step 1:

[0463] The user enters schedule information via a smart device. The device sends the entered data to the server. The server receives this data and stores it in a database. The input consists of schedule details and time, and the output is storage in the database.

[0464] Step 2:

[0465] The server generates notifications at predetermined times based on stored schedules. These notifications include the next tasks to be performed. The generated notifications are sent to the terminal. The input is schedule information from the database, and the output is a notification in the appropriate format.

[0466] Step 3:

[0467] The device presents received notifications to the user as audio or text, prompting the user to take the next action. The input is notification data from the server, and the output is the presentation of information to the user.

[0468] Step 4:

[0469] The device's sensors collect user movement data and location information in real time. This data is periodically transmitted to a server. The input is movement and location data from the sensors, and the output is the data transmission to the server.

[0470] Step 5:

[0471] The server analyzes the collected behavioral data using an AI algorithm. When an abnormal behavioral pattern is detected, it generates a warning and sends a notification to the user and their administrator. The input is behavioral and location data, and the output is a warning message.

[0472] Step 6:

[0473] The device collects video and audio data from the user using its camera and microphone and transmits it to the emotion engine. The emotion engine analyzes this data and estimates the emotional state. The input is video and audio data, and the output is an evaluation of the emotional state.

[0474] Step 7:

[0475] The server generates a guidance message based on the estimated emotional state and sends it to the terminal. The terminal then presents this guidance to the user. The input is emotional state data, and the output is the guidance message.

[0476] Step 8:

[0477] Users receive guidance through their devices and modify their actions as needed. Specifically, users can use the presented information to move around and complete tasks with confidence. The input is guidance messages, and the output is modified behavior.

[0478] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0479] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0480] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0481] [Third Embodiment]

[0482] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0483] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0484] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0485] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0486] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0487] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0488] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0489] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0490] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0491] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0492] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0493] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0494] This invention provides a system to support the lives of dementia patients and enhance their independence, and offers an application that can be implemented on smartphones and wearable devices. The following describes each component of the system and the related processes.

[0495] First, the user enters their schedule information into the application. This information includes medication times, meal times, and daily appointments. This information is sent to and stored on a server in the cloud. The server generates notifications based on the set times and sends them to the user's device.

[0496] Next, the device presents received notifications to the user via voice or text. For example, when it's time to take medication, it will give a voice notification such as, "It's time to take your medicine." This helps users remember to perform important daily tasks.

[0497] Furthermore, as a guide for daily life, users can register the steps for their daily tasks on the device. The server receives this information, breaks down the task into specific steps, and saves them. When requested by the user, the device will guide them through the next task using voice or video. For example, it might say, "Let's wash your face," indicating the next step.

[0498] Furthermore, to ensure security, the device uses sensors and location technology to collect user behavior data. This data is periodically sent to a server and analyzed by an AI algorithm. If an abnormal behavior pattern (for example, going out late at night) is detected, the server will issue a warning to the user and their administrator.

[0499] This allows users to live safely and independently, reducing the burden on family members and caregivers. Furthermore, the system is flexibly customizable, allowing it to provide appropriate support according to the user's needs.

[0500] The following describes the processing flow.

[0501] Step 1:

[0502] Users input their daily schedule information into applications on their smartphones or wearable devices. This includes things like medication times and planned outings.

[0503] Step 2:

[0504] The terminal sends the entered schedule information to a server in the cloud. The transmitted data is stored in the server's database.

[0505] Step 3:

[0506] The server monitors the stored schedule information and prepares to generate notifications as a specific time approaches.

[0507] Step 4:

[0508] When the scheduled time arrives, the server generates a notification based on the schedule and sends it to each user's device.

[0509] Step 5:

[0510] The device notifies the user of received notifications via voice or text. For example, it might display an alert such as, "It's time to take your medicine."

[0511] Step 6:

[0512] The user will take the necessary action (e.g., take medication) according to the notification.

[0513] Step 7:

[0514] Users register the steps for their daily tasks (e.g., getting ready in the morning) on ​​their device.

[0515] Step 8:

[0516] The server breaks down registered tasks into steps and stores them in the database. This process clarifies the specific action steps.

[0517] Step 9:

[0518] When a user requests guidance for a specific task, the device will guide them through the next steps using voice or video. For example, it might say, "Next, let's wash your face."

[0519] Step 10:

[0520] To monitor the user's daily activities, the device collects location information and motion data using sensors and GPS.

[0521] Step 11:

[0522] The collected data is periodically sent to a server. The server uses AI algorithms to analyze and detect abnormal behavioral patterns.

[0523] Step 12:

[0524] If an anomaly is detected, the server will issue a warning to the user and their administrator. This warning will be presented via voice or message through the terminal.

[0525] (Example 1)

[0526] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0527] In modern society, with the advancement of an aging society, there is a growing need for people with cognitive decline to live safely and independently. However, these individuals are more likely to forget important tasks in their daily lives or exhibit abnormal behavior, which increases the burden on both the individuals themselves and their caregivers. The present invention aims to provide a system that supports the daily lives of people with cognitive decline and ensures their safety.

[0528] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0529] In this invention, the server includes means for inputting user activity plan information and storing it in an information storage unit, means for generating notifications at a predetermined time based on the information storage unit and transmitting them to each user device, and means for analyzing collected operation information to detect abnormalities and generating warnings when abnormalities are detected. This enables users to act independently without forgetting important daily tasks and to live safely by responding immediately when abnormal behavior is detected.

[0530] "User" refers to an individual who uses the system of the present invention, and may particularly include individuals with cognitive impairment.

[0531] "Activity plan information" refers to information about the plans and tasks that users intend to carry out in their daily lives.

[0532] The term "information storage unit" refers to a storage system that stores and manages input data in chronological order.

[0533] "Notification" refers to messages or information that are sent to users based on a set schedule.

[0534] "User device" refers to a terminal device or wearable device carried by the user, used to receive notifications from the system.

[0535] "Motion information" refers to data related to the user's movement and activities, and is collected using sensors and location information technology.

[0536] A "warning" refers to a cautionary message sent to users and their administrators when the system detects an anomaly.

[0537] A "machine learning algorithm" refers to a computational method used to analyze patterns based on collected behavioral information and detect anomalies.

[0538] This invention is a system designed to support the daily lives of people with cognitive impairment and ensure their safety. The system primarily functions as a server, terminals, and users.

[0539] The server receives user activity plan information and stores it in a cloud-based information storage unit. Typically, a cloud storage system is used for this purpose. Based on the stored information, notifications are generated according to specific timeframes and sent to each terminal. A messaging protocol prioritizing real-time delivery is used for sending these notifications.

[0540] The terminal receives notifications from the server and presents them to the user in audio or text format. It can also prompt the user for the next necessary action based on their response. For this purpose, the terminal is equipped with speech recognition and speech synthesis capabilities. For example, to help users remember when to take their medication, it can provide reminders such as, "It's 8 o'clock. It's time to take your medicine."

[0541] Users manually input their activity plan information into the device. This allows them to visually check their daily schedule and manage their actions through notifications from the device. Activity plan information could include basic daily routines such as "8:00: Take medicine" and "12:00: Have lunch."

[0542] Furthermore, the system collects user behavior information using sensors and GPS built into the device. This behavior information is analyzed on a server, and if an abnormal pattern is detected, a warning is immediately generated. Machine learning algorithms are used to analyze behavior patterns, enabling early detection of anomalies.

[0543] For example, when managing a user's daily tasks in a structured manner, if the user enters "morning preparations" as a task, the device can provide voice guidance such as "Let's wash your face" and "Next, let's brush our teeth."

[0544] Examples of prompts include, "Please provide an overview of the daily life support system for dementia patients," and "Please explain in detail how user schedule information is processed." These prompts clarify the information that stakeholders will handle through the generated AI model, thereby improving the system's overall quality.

[0545] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0546] Step 1:

[0547] Users input activity plan information using an application installed on their smartphone or wearable device. This input information includes things like medication times and meal times. Specifically, they enter their schedule in text format into a form on the screen. Once input is complete, the data is sent from the device to the server. The output is the entered activity plan information data.

[0548] Step 2:

[0549] The server stores the received activity plan information in a cloud-based information storage unit. The storage system used is a general-purpose database, and normalization is performed to ensure data integrity and availability. After being stored in the database, this data becomes the basis for generating notifications based on the set time. The output is the plan information stored in the database.

[0550] Step 3:

[0551] The server generates a notification as a predetermined time approaches, based on the activity plan information stored in the information storage unit. The notification content is processed into a message related to the target task and sent to each terminal using a real-time messaging protocol. The output is the notification message sent to the user's terminal.

[0552] Step 4:

[0553] The terminal immediately displays notifications received from the server to the user. The display method is either voice or text information, selected according to the user's settings. A speech synthesis engine is used, and the voice message "It's time to take your medicine" is delivered. The output is notification information for the user.

[0554] Step 5:

[0555] The device periodically collects user activity information using built-in sensors and GPS functionality. This includes distance traveled and current location coordinates. This data is encrypted and sent to a server. The output is activity data collected from sensors and GPS.

[0556] Step 6:

[0557] The server receives collected operational data and uses machine learning algorithms to analyze abnormal patterns. The boundary between normal and abnormal is dynamically set based on past analysis data. As a result, if an abnormality such as leaving the premises late at night is detected, a warning is generated. The output consists of the abnormality detection result and the warning message.

[0558] Step 7:

[0559] The server sends out warnings to users and their administrators when an anomaly is detected. These warnings are sent via email or SMS, ensuring immediate notification. This warning system facilitates a quick response and ensures security.

[0560] (Application Example 1)

[0561] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0562] The objective of this invention is to provide support that enables elderly people and dementia patients to live their daily lives more safely and independently. In particular, it is necessary to prevent confusion and dangerous situations during shopping activities in physical stores, and to enable users to conduct purchasing activities efficiently. Furthermore, it is also important to enable immediate and appropriate responses in the event of abnormal behavior.

[0563] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0564] In this invention, the server includes means for inputting user activity information and storing it in an information management device, means for generating notifications at predetermined times based on the information management device and transmitting them to each user device, and means for monitoring the user's behavior within the store and notifying the store manager if an abnormality occurs. This makes it possible to improve the safety and efficiency of shopping activities for the elderly and dementia patients in physical stores.

[0565] "User" refers to an individual who uses the system to receive support for their daily activities.

[0566] "Activity information" refers to data about the plans and necessary tasks that users perform in their daily lives and purchasing activities.

[0567] An "information management device" is a digital device that manages a user's schedule and tasks and provides information when needed.

[0568] "Generating a notification" refers to the process of creating information that prompts a user to take a specific action.

[0569] "Devices used" refers to electronic devices such as smartphones, tablets, and wearable devices that users carry with them.

[0570] A "sensor" refers to a sensor device used to monitor a user's physical movements and health condition.

[0571] "Location data technology" refers to technology used to determine the current geographical location of a user or object.

[0572] An "analysis device" refers to a computing device used to analyze collected data and detect anomalies.

[0573] A "warning signal" refers to warning information generated when the system detects an anomaly.

[0574] A "purchase catalog" refers to a list of items that a user intends to purchase at a physical store.

[0575] "Product information" refers to information that provides users with directions or location information for products they intend to purchase, either through audio or video.

[0576] A "store manager" refers to the person responsible for overseeing operations at a physical store.

[0577] "Machine learning technology" is a method of learning patterns using large amounts of data, and is a technology used for anomaly detection and other applications.

[0578] A system implementing this invention includes a series of functions for managing user activity information and ensuring safety and efficiency during purchasing activities.

[0579] The server stores user-entered activity information and purchase lists in a cloud-based information management system. This ensures data security and accessibility when needed. The server then generates and sends notifications to user devices based on a predefined schedule. These notifications include information to support purchasing activities; for example, they can inform users of the time when they should pick up a specific product.

[0580] The device being used, such as a smartphone or wearable device, will present this notification to the user via voice or text. This allows the user to know when to take the specified action. In particular, in physical stores, location data technology can be used to guide users to the location of specific products. This allows users to effectively find the products they are looking for.

[0581] Furthermore, the sensors collect user activity information in real time and transmit it to a server. The server analyzes this information using an analysis device, and if an abnormal pattern is detected, it immediately generates a warning signal using machine learning technology and notifies the store manager. In this process, AI models are used to achieve highly accurate anomaly detection.

[0582] As a concrete example, consider a scenario where a user is shopping at a supermarket. If the user is looking for milk, the device will guide them by saying, "The milk section is on the refrigerated shelves on the left." If the user remains stationary for an extended period, the server analyzes this information and, if necessary, sends a warning signal to store staff.

[0583] An example of a prompt to use with a generative AI model is: "Suggest a way to guide the user to effectively find items on their shopping list. How would you support the user using in-store location data?"

[0584] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0585] Step 1:

[0586] Users input daily activity information and purchase lists into their devices. The input data is then transmitted from the device to a server. The server stores this data in a cloud-based information management system. The input data includes schedule information and product lists, which are stored in a database for use in subsequent processing.

[0587] Step 2:

[0588] The server generates notifications to support the user's purchasing behavior based on stored activity information. Schedule data is used to generate notifications, configuring instructions for picking up specific products at set times. The generated notifications are sent to the user's device. The output here is a timely notification regarding purchasing activity.

[0589] Step 3:

[0590] The device presents notifications received from the server to the user via voice or text. This presentation is done through the device's built-in voice output function or display. The information presented is intended to help the user know the location and purchase timing of a specific product, and the output is in the form of voice messages or text displays.

[0591] Step 4:

[0592] User movement information is acquired through sensors connected to the terminal. In this process, user movement and location data are collected in real time by sensors and recorded on the terminal. The input consists of physical location and movement information, which is stored and processed into data to be sent to the server.

[0593] Step 5:

[0594] The server analyzes the behavioral information transmitted from the terminal using an analysis device. Here, an abnormal pattern is detected using a generative AI model. For example, if the device remains motionless for a long period of time or if an unusual movement pattern is observed, it is judged to be abnormal. The input for the analysis is behavioral data from sensors, and the output is a judgment result indicating whether or not an abnormality is present.

[0595] Step 6:

[0596] If an anomaly is detected, the server generates an alert signal and sends a notification to the store manager. This is to prompt immediate action upon anomaly detection, allowing store staff to provide verification and assistance as needed. The input is the anomaly detection result, and the output is the alarm message.

[0597] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0598] This invention provides a system that combines an emotion engine with a system for supporting dementia patients, thereby offering appropriate support and safety supervision tailored to the user's mental state. The system is provided as an application implemented on smartphones and wearable devices.

[0599] When using the system, users enter their schedule information into the application. This information is sent to the server via the terminal and stored in a database. Based on the entered schedule information, the server generates notifications at the necessary times and sends them to the user's terminal.

[0600] The device that receives the notification will present the information to the user via voice or text, guiding them on what to do next. For example, if it's mealtime, it might say, "It's time to start eating." These notifications prevent user confusion and promote independence in daily life.

[0601] This system incorporates a function to collect user behavior data using sensors and location technology. The terminal periodically sends this data to a server, where it is analyzed using AI algorithms. If an abnormal behavior pattern is detected, the server issues a warning to the user and their administrator to ensure security.

[0602] Another distinctive feature is the emotion engine built into the device. The emotion engine analyzes the user's facial expressions and voice tone through the camera and voice input, collecting and storing emotional data. Using this data, the server determines the user's emotional state and provides notifications and support accordingly.

[0603] For example, if the emotion engine determines that a user is feeling down, the device will provide an encouraging message such as, "Let's take a short walk to cheer you up." Emotional data is also integrated with anomaly detection algorithms to enable more accurate safety monitoring.

[0604] This allows for comprehensive support that takes into account the emotional well-being of users, helping them to achieve a safe and independent life.

[0605] The following describes the processing flow.

[0606] Step 1:

[0607] Users input their daily schedule information (e.g., medication times, planned outings) into an application on their smartphone or wearable device.

[0608] Step 2:

[0609] The terminal sends the entered schedule information to the server and stores it in the database.

[0610] Step 3:

[0611] The server manages schedule information, generates notifications according to predetermined times, and prepares them to be sent to the user's terminal.

[0612] Step 4:

[0613] The device notifies the user of received notifications via voice or text. For example, it might announce, "It's time to take your medicine."

[0614] Step 5:

[0615] At the same time, the device uses its camera and microphone to collect the user's facial expressions and voice data.

[0616] Step 6:

[0617] The emotion engine built into the device analyzes the collected data and evaluates the user's emotional state (e.g., joy, sadness, stress).

[0618] Step 7:

[0619] The emotion engine evaluates the user's emotional state and sends the results to the server.

[0620] Step 8:

[0621] Based on the received emotional data, the server determines how to respond to the user and what additional notifications to send. For example, if the user is feeling stressed, it might suggest, "Take a deep breath to relax."

[0622] Step 9:

[0623] Meanwhile, the device continues to collect operational data using sensors and location information technology and transmit it to the server.

[0624] Step 10:

[0625] The server uses AI algorithms to integrate behavioral and emotional data to detect abnormal behavior.

[0626] Step 11:

[0627] If an anomaly is detected, the server immediately sends a warning to the user and their administrator, and displays a message on the terminal prompting them to take specific action.

[0628] Step 12:

[0629] Users will follow the guidance provided by the device and contact family members or caregivers as needed.

[0630] Through the above process, we support the safety and emotional well-being of users simultaneously, achieving more comprehensive life support than ever before.

[0631] (Example 2)

[0632] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0633] In an aging society, a system is needed that accurately supports the daily activities and mental state of people with cognitive decline in order to help them live safe and independent lives. However, conventional technology has limitations in taking into account the emotional state of users and in its ability to immediately detect abnormal behavior and send warnings. Therefore, there is a need for means to provide more comprehensive support.

[0634] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0635] In this invention, the server includes means for inputting the user's schedule information into an information device and storing it on an information recording medium; means for generating notifications at a predetermined time based on the information recording medium and communicating them to each information terminal; and means for collecting emotional data via a camera and voice input, which the information device then analyzes. This makes it possible to comprehensively support the user's behavior and emotional state and promote a safe and independent life.

[0636] An "information device" is a digital device used by users to input schedule information and manage various types of data.

[0637] An "information recording medium" is a storage device or database used to store data entered by a user or data generated by a system.

[0638] An "information terminal" is a device used to provide users with notifications from a server, and it has the functionality to receive and display notifications.

[0639] A "detector" refers to a measuring device or sensor installed to collect user behavior data.

[0640] "Geographic information technology" is a technology used to track a user's movement and location based on location information.

[0641] A "machine learning algorithm" is an algorithm that analyzes collected data, recognizes patterns, and identifies anomalies.

[0642] "Emotional data" refers to data collected through cameras and voice input that indicates the user's emotions and mental state.

[0643] "Immediately communicating warnings" means notifying users and administrators without delay when abnormal behavior or conditions are detected.

[0644] To implement this invention, smartphones and wearable devices are used as information devices to manage users' schedule information and activity data. In this process, the data must be securely stored in a cloud database, which is an information recording medium. A server connects to the information terminal via internet communication and generates notifications based on the schedule information received from the user. These notifications are sent to the information terminal at a pre-set time, allowing the user to confirm them visually or audibly.

[0645] The information terminal incorporates accelerometers and GPS as detectors, periodically collecting the user's location information and body movements. This data is transmitted to a server in combination with geographic information technology and analyzed using machine learning algorithms. If abnormal behavior is detected as a result of the analysis, the server immediately communicates a warning to the user and administrator to ensure security.

[0646] Furthermore, the information terminal is equipped with a high-performance camera and microphone, which analyze facial expressions and voice tone to collect emotional data. This allows the server to analyze the user's emotional state and generate appropriate responses. For example, if the user is feeling down, the server might generate a prompt such as "Suggesting activities to refresh you," providing encouragement to the user through the information terminal.

[0647] For example, if a user enters a walking schedule at 2:00 PM every day, the information device will generate a notification at 1:55 PM that says, "It's almost time for your walk. Let's start getting ready." Furthermore, if sentiment data reveals that the user is feeling stressed, it will offer advice such as, "Why not try listening to your favorite music while you walk?" An example of a prompt message would be an instruction such as, "Suggesting the next action to take."

[0648] With the above configuration, it is possible to comprehensively support the user's behavior and emotional state, enabling them to live a safe and independent life.

[0649] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0650] Step 1:

[0651] Users input their schedule information using smartphones or wearable devices. This input includes the date, time, and details of the schedule. The entered data is temporarily stored within the information device and prepared for transmission to the server.

[0652] Step 2:

[0653] The terminal transmits the schedule information entered by the user to the server via the internet. The server receives this data and stores it in a database for each user. The stored data is used for subsequent processing based on the scheduled time.

[0654] Step 3:

[0655] The server references the stored schedule information and generates notifications at the specified times. As part of the data processing, it creates notification content from the schedule information and adds it to the notification registration list. The generated notifications are formatted according to the notification format (audio or text) specified by the user.

[0656] Step 4:

[0657] The server sends the generated notification to the device. The device receives this notification and presents it to the user visually or audibly. Specifically, it displays a pop-up notification on the device screen and, if necessary, provides an audio alert. This allows the user to confirm and appropriately perform the next action.

[0658] Step 5:

[0659] The device uses built-in sensors and GPS technology to collect user movement data. This data includes motion information such as location, speed, and direction. The collected data is periodically transmitted to a server.

[0660] Step 6:

[0661] The server receives behavioral data sent from the terminal and analyzes it using machine learning algorithms. It identifies behavioral patterns from the input data and performs data calculations to identify abnormal behavior. If an anomaly is detected, it generates a warning for immediate action.

[0662] Step 7:

[0663] The server sends the generated warning to the user and their supervisor. The terminal displays this warning as a notification, prompting the user to check for safety. This allows users and supervisors to take prompt action.

[0664] Step 8:

[0665] The device uses a camera and voice input device to collect information about the user's emotions. It analyzes facial expressions and voice tone to determine the emotional state in real time. The emotional data is sent to a server.

[0666] Step 9:

[0667] The server analyzes emotional data and generates responses best suited to the user's mental state. Using a generative AI model, it creates appropriate prompts and suggestions based on the input emotional information. As a result, it recommends actions that contribute to improving the user's quality of life.

[0668] Step 10:

[0669] The server sends the generated responses and suggestions to the terminal and notifies the user. The terminal collects the user's responses and sends the feedback to the server to further improve the system's accuracy. This enables flexible support tailored to the user's needs.

[0670] (Application Example 2)

[0671] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0672] For dementia patients and the elderly, managing their daily lives and ensuring safety are significant challenges. Furthermore, the inability to respond appropriately to emotional fluctuations can exacerbate mental anxiety. Conventional support systems are limited to collecting movement data and providing schedule notifications, failing to adequately offer flexible support tailored to the user's emotional state.

[0673] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0674] In this invention, the server includes means for analyzing video and audio data using an emotion engine to estimate the user's emotional state, means for providing guidance to the user based on the estimated emotional state, and means for monitoring changes in the emotional state in real time and providing support for ensuring safety. This enables detailed support and safety supervision in response to fluctuations in the user's emotions.

[0675] "A means of inputting user schedule information and saving it to a database" refers to a function that provides an interface for users to input their schedules and activities, and efficiently stores that information in a database.

[0676] "A means of generating notifications at a predetermined time based on a database and sending them to each user's device" refers to a function that creates notifications at the appropriate time based on a pre-registered schedule and sends them to devices such as smart devices.

[0677] "A means of presenting notifications received by the user's device in audio or text and prompting the user to take appropriate action" refers to an interface that displays instructions received by the user on their device in audio or text format, guiding them to the next action.

[0678] "Means for collecting user motion data using sensors and location information technology" refers to a function that uses technologies such as sensors and GPS to acquire information about the user's body movements and location.

[0679] "Means for analyzing collected operational data to detect anomalies and generating warnings when an anomaly is detected" refers to a function that analyzes operational data to identify patterns that are different from the norm and creates a warning message when necessary.

[0680] "Means of sending warnings to users and their administrators to ensure security" refers to communication methods for notifying users and administrators of urgent situations and prompting a swift response.

[0681] "A means of analyzing video and audio data using an emotion engine to estimate the user's emotional state" refers to a function that analyzes data acquired from cameras and microphones to evaluate and judge the user's emotions.

[0682] "Means of providing guidance to users based on estimated emotional states" refers to functions that provide appropriate advice and instructions in accordance with the user's emotions.

[0683] "A means of monitoring changes in emotional state in real time and providing support for ensuring safety" refers to a system that quickly detects fluctuations in emotions and provides support for ensuring safety based on those fluctuations.

[0684] The system implementing this invention consists of a group of smart devices used by dementia patients and the elderly to ensure self-management and safety. At the core of the system is a server equipped with an emotion engine. The server receives the user's schedule information and stores it in a database. Based on the schedule, it generates notifications at predetermined times and sends them to the user's mobile device. This process enables the user to perform their daily activities without forgetting.

[0685] On the device, received notifications are presented to the user in voice or text format, prompting them to take their planned actions. The device incorporates sensors and location technology to collect user activity data. The server analyzes the collected data using AI algorithms and immediately generates a warning and notifies the administrator if an anomaly is detected.

[0686] Furthermore, because it is equipped with an emotion engine, it can estimate the user's emotional state in real time based on video and audio data acquired by the camera and microphone of smart glasses or wearable devices. Based on the estimated emotional state, the device provides personalized advice and guidance to the user. Technologies used in this process include OpenCV for image recognition and speech recognition APIs for speech analysis, while communication services such as Twilio are used for notifications.

[0687] For example, if a user appears lost while walking in a park, the device will ask, "Where do you want to go now?" and guide them in the correct direction by referring to their location information. Another example of a prompt using the generated AI model is, "Tell me a phrase to use when I forget where I want to go." This helps users to act with confidence even in difficult situations.

[0688] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0689] Step 1:

[0690] The user enters schedule information via a smart device. The device sends the entered data to the server. The server receives this data and stores it in a database. The input consists of schedule details and time, and the output is storage in the database.

[0691] Step 2:

[0692] The server generates notifications at predetermined times based on stored schedules. These notifications include the next tasks to be performed. The generated notifications are sent to the terminal. The input is schedule information from the database, and the output is a notification in the appropriate format.

[0693] Step 3:

[0694] The device presents received notifications to the user as audio or text, prompting the user to take the next action. The input is notification data from the server, and the output is the presentation of information to the user.

[0695] Step 4:

[0696] The device's sensors collect user movement data and location information in real time. This data is periodically transmitted to a server. The input is movement and location data from the sensors, and the output is the data transmission to the server.

[0697] Step 5:

[0698] The server analyzes the collected behavioral data using an AI algorithm. When an abnormal behavioral pattern is detected, it generates a warning and sends a notification to the user and their administrator. The input is behavioral and location data, and the output is a warning message.

[0699] Step 6:

[0700] The device collects video and audio data from the user using its camera and microphone and transmits it to the emotion engine. The emotion engine analyzes this data and estimates the emotional state. The input is video and audio data, and the output is an evaluation of the emotional state.

[0701] Step 7:

[0702] The server generates a guidance message based on the estimated emotional state and sends it to the terminal. The terminal then presents this guidance to the user. The input is emotional state data, and the output is the guidance message.

[0703] Step 8:

[0704] Users receive guidance through their devices and modify their actions as needed. Specifically, users can use the presented information to move around and complete tasks with confidence. The input is guidance messages, and the output is modified behavior.

[0705] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0706] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0707] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0708] [Fourth Embodiment]

[0709] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0710] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0711] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0712] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0713] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0714] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0715] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0716] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0717] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0718] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0719] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0720] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0721] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0722] This invention provides a system to support the lives of dementia patients and enhance their independence, and offers an application that can be implemented on smartphones and wearable devices. The following describes each component of the system and the related processes.

[0723] First, the user enters their schedule information into the application. This information includes medication times, meal times, and daily appointments. This information is sent to and stored on a server in the cloud. The server generates notifications based on the set times and sends them to the user's device.

[0724] Next, the device presents received notifications to the user via voice or text. For example, when it's time to take medication, it will give a voice notification such as, "It's time to take your medicine." This helps users remember to perform important daily tasks.

[0725] Furthermore, as a guide for daily life, users can register the steps for their daily tasks on the device. The server receives this information, breaks down the task into specific steps, and saves them. When requested by the user, the device will guide them through the next task using voice or video. For example, it might say, "Let's wash your face," indicating the next step.

[0726] Furthermore, to ensure security, the device uses sensors and location technology to collect user behavior data. This data is periodically sent to a server and analyzed by an AI algorithm. If an abnormal behavior pattern (for example, going out late at night) is detected, the server will issue a warning to the user and their administrator.

[0727] This allows users to live safely and independently, reducing the burden on family members and caregivers. Furthermore, the system is flexibly customizable, allowing it to provide appropriate support according to the user's needs.

[0728] The following describes the processing flow.

[0729] Step 1:

[0730] Users input their daily schedule information into applications on their smartphones or wearable devices. This includes things like medication times and planned outings.

[0731] Step 2:

[0732] The terminal sends the entered schedule information to a server in the cloud. The transmitted data is stored in the server's database.

[0733] Step 3:

[0734] The server monitors the stored schedule information and prepares to generate notifications as a specific time approaches.

[0735] Step 4:

[0736] When the scheduled time arrives, the server generates a notification based on the schedule and sends it to each user's device.

[0737] Step 5:

[0738] The device notifies the user of received notifications via voice or text. For example, it might display an alert such as, "It's time to take your medicine."

[0739] Step 6:

[0740] The user will take the necessary action (e.g., take medication) according to the notification.

[0741] Step 7:

[0742] Users register the steps for their daily tasks (e.g., getting ready in the morning) on ​​their device.

[0743] Step 8:

[0744] The server breaks down registered tasks into steps and stores them in the database. This process clarifies the specific action steps.

[0745] Step 9:

[0746] When a user requests guidance for a specific task, the device will guide them through the next steps using voice or video. For example, it might say, "Next, let's wash your face."

[0747] Step 10:

[0748] To monitor the user's daily activities, the device collects location information and motion data using sensors and GPS.

[0749] Step 11:

[0750] The collected data is periodically sent to a server. The server uses AI algorithms to analyze and detect abnormal behavioral patterns.

[0751] Step 12:

[0752] If an anomaly is detected, the server will issue a warning to the user and their administrator. This warning will be presented via voice or message through the terminal.

[0753] (Example 1)

[0754] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0755] In modern society, with the advancement of an aging society, there is a growing need for people with cognitive decline to live safely and independently. However, these individuals are more likely to forget important tasks in their daily lives or exhibit abnormal behavior, which increases the burden on both the individuals themselves and their caregivers. The present invention aims to provide a system that supports the daily lives of people with cognitive decline and ensures their safety.

[0756] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0757] In this invention, the server includes means for inputting user activity plan information and storing it in an information storage unit, means for generating notifications at a predetermined time based on the information storage unit and transmitting them to each user device, and means for analyzing collected operation information to detect abnormalities and generating warnings when abnormalities are detected. This enables users to act independently without forgetting important daily tasks and to live safely by responding immediately when abnormal behavior is detected.

[0758] "User" refers to an individual who uses the system of the present invention, and may particularly include individuals with cognitive impairment.

[0759] "Activity plan information" refers to information about the plans and tasks that users intend to carry out in their daily lives.

[0760] The term "information storage unit" refers to a storage system that stores and manages input data in chronological order.

[0761] "Notification" refers to messages or information that are sent to users based on a set schedule.

[0762] "User device" refers to a terminal device or wearable device carried by the user, used to receive notifications from the system.

[0763] "Motion information" refers to data related to the user's movement and activities, and is collected using sensors and location information technology.

[0764] A "warning" refers to a cautionary message sent to users and their administrators when the system detects an anomaly.

[0765] A "machine learning algorithm" refers to a computational method used to analyze patterns based on collected behavioral information and detect anomalies.

[0766] This invention is a system designed to support the daily lives of people with cognitive impairment and ensure their safety. The system primarily functions as a server, terminals, and users.

[0767] The server receives user activity plan information and stores it in a cloud-based information storage unit. Typically, a cloud storage system is used for this purpose. Based on the stored information, notifications are generated according to specific timeframes and sent to each terminal. A messaging protocol prioritizing real-time delivery is used for sending these notifications.

[0768] The terminal receives notifications from the server and presents them to the user in audio or text format. It can also prompt the user for the next necessary action based on their response. For this purpose, the terminal is equipped with speech recognition and speech synthesis capabilities. For example, to help users remember when to take their medication, it can provide reminders such as, "It's 8 o'clock. It's time to take your medicine."

[0769] Users manually input their activity plan information into the device. This allows them to visually check their daily schedule and manage their actions through notifications from the device. Activity plan information could include basic daily routines such as "8:00: Take medicine" and "12:00: Have lunch."

[0770] Furthermore, the system collects user behavior information using sensors and GPS built into the device. This behavior information is analyzed on a server, and if an abnormal pattern is detected, a warning is immediately generated. Machine learning algorithms are used to analyze behavior patterns, enabling early detection of anomalies.

[0771] For example, when managing a user's daily tasks in a structured manner, if the user enters "morning preparations" as a task, the device can provide voice guidance such as "Let's wash your face" and "Next, let's brush our teeth."

[0772] Examples of prompts include, "Please provide an overview of the daily life support system for dementia patients," and "Please explain in detail how user schedule information is processed." These prompts clarify the information that stakeholders will handle through the generated AI model, thereby improving the system's overall quality.

[0773] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0774] Step 1:

[0775] Users input activity plan information using an application installed on their smartphone or wearable device. This input information includes things like medication times and meal times. Specifically, they enter their schedule in text format into a form on the screen. Once input is complete, the data is sent from the device to the server. The output is the entered activity plan information data.

[0776] Step 2:

[0777] The server stores the received activity plan information in a cloud-based information storage unit. The storage system used is a general-purpose database, and normalization is performed to ensure data integrity and availability. After being stored in the database, this data becomes the basis for generating notifications based on the set time. The output is the plan information stored in the database.

[0778] Step 3:

[0779] The server generates a notification as a predetermined time approaches, based on the activity plan information stored in the information storage unit. The notification content is processed into a message related to the target task and sent to each terminal using a real-time messaging protocol. The output is the notification message sent to the user's terminal.

[0780] Step 4:

[0781] The terminal immediately displays notifications received from the server to the user. The display method is either voice or text information, selected according to the user's settings. A speech synthesis engine is used, and the voice message "It's time to take your medicine" is delivered. The output is notification information for the user.

[0782] Step 5:

[0783] The device periodically collects user activity information using built-in sensors and GPS functionality. This includes distance traveled and current location coordinates. This data is encrypted and sent to a server. The output is activity data collected from sensors and GPS.

[0784] Step 6:

[0785] The server receives collected operational data and uses machine learning algorithms to analyze abnormal patterns. The boundary between normal and abnormal is dynamically set based on past analysis data. As a result, if an abnormality such as leaving the premises late at night is detected, a warning is generated. The output consists of the abnormality detection result and the warning message.

[0786] Step 7:

[0787] The server sends out warnings to users and their administrators when an anomaly is detected. These warnings are sent via email or SMS, ensuring immediate notification. This warning system facilitates a quick response and ensures security.

[0788] (Application Example 1)

[0789] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0790] The objective of this invention is to provide support that enables elderly people and dementia patients to live their daily lives more safely and independently. In particular, it is necessary to prevent confusion and dangerous situations during shopping activities in physical stores, and to enable users to conduct purchasing activities efficiently. Furthermore, it is also important to enable immediate and appropriate responses in the event of abnormal behavior.

[0791] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0792] In this invention, the server includes means for inputting user activity information and storing it in an information management device, means for generating notifications at predetermined times based on the information management device and transmitting them to each user device, and means for monitoring the user's behavior within the store and notifying the store manager if an abnormality occurs. This makes it possible to improve the safety and efficiency of shopping activities for the elderly and dementia patients in physical stores.

[0793] "User" refers to an individual who uses the system to receive support for their daily activities.

[0794] "Activity information" refers to data about the plans and necessary tasks that users perform in their daily lives and purchasing activities.

[0795] An "information management device" is a digital device that manages a user's schedule and tasks and provides information when needed.

[0796] "Generating a notification" refers to the process of creating information that prompts a user to take a specific action.

[0797] "Devices used" refers to electronic devices such as smartphones, tablets, and wearable devices that users carry with them.

[0798] A "sensor" refers to a sensor device used to monitor a user's physical movements and health condition.

[0799] "Location data technology" refers to technology used to determine the current geographical location of a user or object.

[0800] An "analysis device" refers to a computing device used to analyze collected data and detect anomalies.

[0801] A "warning signal" refers to warning information generated when the system detects an anomaly.

[0802] A "purchase catalog" refers to a list of items that a user intends to purchase at a physical store.

[0803] "Product information" refers to information that provides users with directions or location information for products they intend to purchase, either through audio or video.

[0804] A "store manager" refers to the person responsible for overseeing operations at a physical store.

[0805] "Machine learning technology" is a method of learning patterns using large amounts of data, and is a technology used for anomaly detection and other applications.

[0806] A system implementing this invention includes a series of functions for managing user activity information and ensuring safety and efficiency during purchasing activities.

[0807] The server stores user-entered activity information and purchase lists in a cloud-based information management system. This ensures data security and accessibility when needed. The server then generates and sends notifications to user devices based on a predefined schedule. These notifications include information to support purchasing activities; for example, they can inform users of the time when they should pick up a specific product.

[0808] The device being used, such as a smartphone or wearable device, will present this notification to the user via voice or text. This allows the user to know when to take the specified action. In particular, in physical stores, location data technology can be used to guide users to the location of specific products. This allows users to effectively find the products they are looking for.

[0809] Furthermore, the sensors collect user activity information in real time and transmit it to a server. The server analyzes this information using an analysis device, and if an abnormal pattern is detected, it immediately generates a warning signal using machine learning technology and notifies the store manager. In this process, AI models are used to achieve highly accurate anomaly detection.

[0810] As a concrete example, consider a scenario where a user is shopping at a supermarket. If the user is looking for milk, the device will guide them by saying, "The milk section is on the refrigerated shelves on the left." If the user remains stationary for an extended period, the server analyzes this information and, if necessary, sends a warning signal to store staff.

[0811] An example of a prompt to use with a generative AI model is: "Suggest a way to guide the user to effectively find items on their shopping list. How would you support the user using in-store location data?"

[0812] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0813] Step 1:

[0814] Users input daily activity information and purchase lists into their devices. The input data is then transmitted from the device to a server. The server stores this data in a cloud-based information management system. The input data includes schedule information and product lists, which are stored in a database for use in subsequent processing.

[0815] Step 2:

[0816] The server generates notifications to support the user's purchasing behavior based on stored activity information. Schedule data is used to generate notifications, configuring instructions for picking up specific products at set times. The generated notifications are sent to the user's device. The output here is a timely notification regarding purchasing activity.

[0817] Step 3:

[0818] The device presents notifications received from the server to the user via voice or text. This presentation is done through the device's built-in voice output function or display. The information presented is intended to help the user know the location and purchase timing of a specific product, and the output is in the form of voice messages or text displays.

[0819] Step 4:

[0820] User movement information is acquired through sensors connected to the terminal. In this process, user movement and location data are collected in real time by sensors and recorded on the terminal. The input consists of physical location and movement information, which is stored and processed into data to be sent to the server.

[0821] Step 5:

[0822] The server analyzes the behavioral information transmitted from the terminal using an analysis device. Here, an abnormal pattern is detected using a generative AI model. For example, if the device remains motionless for a long period of time or if an unusual movement pattern is observed, it is judged to be abnormal. The input for the analysis is behavioral data from sensors, and the output is a judgment result indicating whether or not an abnormality is present.

[0823] Step 6:

[0824] If an anomaly is detected, the server generates an alert signal and sends a notification to the store manager. This is to prompt immediate action upon anomaly detection, allowing store staff to provide verification and assistance as needed. The input is the anomaly detection result, and the output is the alarm message.

[0825] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0826] This invention provides a system that combines an emotion engine with a system for supporting dementia patients, thereby offering appropriate support and safety supervision tailored to the user's mental state. The system is provided as an application implemented on smartphones and wearable devices.

[0827] When using the system, users enter their schedule information into the application. This information is sent to the server via the terminal and stored in a database. Based on the entered schedule information, the server generates notifications at the necessary times and sends them to the user's terminal.

[0828] The device that receives the notification will present the information to the user via voice or text, guiding them on what to do next. For example, if it's mealtime, it might say, "It's time to start eating." These notifications prevent user confusion and promote independence in daily life.

[0829] This system incorporates a function to collect user behavior data using sensors and location technology. The terminal periodically sends this data to a server, where it is analyzed using AI algorithms. If an abnormal behavior pattern is detected, the server issues a warning to the user and their administrator to ensure security.

[0830] Another distinctive feature is the emotion engine built into the device. The emotion engine analyzes the user's facial expressions and voice tone through the camera and voice input, collecting and storing emotional data. Using this data, the server determines the user's emotional state and provides notifications and support accordingly.

[0831] For example, if the emotion engine determines that a user is feeling down, the device will provide an encouraging message such as, "Let's take a short walk to cheer you up." Emotional data is also integrated with anomaly detection algorithms to enable more accurate safety monitoring.

[0832] This allows for comprehensive support that takes into account the emotional well-being of users, helping them to achieve a safe and independent life.

[0833] The following describes the processing flow.

[0834] Step 1:

[0835] Users input their daily schedule information (e.g., medication times, planned outings) into an application on their smartphone or wearable device.

[0836] Step 2:

[0837] The terminal sends the entered schedule information to the server and stores it in the database.

[0838] Step 3:

[0839] The server manages schedule information, generates notifications according to predetermined times, and prepares them to be sent to the user's terminal.

[0840] Step 4:

[0841] The device notifies the user of received notifications via voice or text. For example, it might announce, "It's time to take your medicine."

[0842] Step 5:

[0843] At the same time, the device uses its camera and microphone to collect the user's facial expressions and voice data.

[0844] Step 6:

[0845] The emotion engine built into the device analyzes the collected data and evaluates the user's emotional state (e.g., joy, sadness, stress).

[0846] Step 7:

[0847] The emotion engine evaluates the user's emotional state and sends the results to the server.

[0848] Step 8:

[0849] Based on the received emotional data, the server determines how to respond to the user and what additional notifications to send. For example, if the user is feeling stressed, it might suggest, "Take a deep breath to relax."

[0850] Step 9:

[0851] Meanwhile, the device continues to collect operational data using sensors and location information technology and transmit it to the server.

[0852] Step 10:

[0853] The server uses AI algorithms to integrate behavioral and emotional data to detect abnormal behavior.

[0854] Step 11:

[0855] If an anomaly is detected, the server immediately sends a warning to the user and their administrator, and displays a message on the terminal prompting them to take specific action.

[0856] Step 12:

[0857] Users will follow the guidance provided by the device and contact family members or caregivers as needed.

[0858] Through the above process, we support the safety and emotional well-being of users simultaneously, achieving more comprehensive life support than ever before.

[0859] (Example 2)

[0860] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0861] In an aging society, a system is needed that accurately supports the daily activities and mental state of people with cognitive decline in order to help them live safe and independent lives. However, conventional technology has limitations in taking into account the emotional state of users and in its ability to immediately detect abnormal behavior and send warnings. Therefore, there is a need for means to provide more comprehensive support.

[0862] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0863] In this invention, the server includes means for inputting the user's schedule information into an information device and storing it on an information recording medium; means for generating notifications at a predetermined time based on the information recording medium and communicating them to each information terminal; and means for collecting emotional data via a camera and voice input, which the information device then analyzes. This makes it possible to comprehensively support the user's behavior and emotional state and promote a safe and independent life.

[0864] An "information device" is a digital device used by users to input schedule information and manage various types of data.

[0865] An "information recording medium" is a storage device or database used to store data entered by a user or data generated by a system.

[0866] An "information terminal" is a device used to provide users with notifications from a server, and it has the functionality to receive and display notifications.

[0867] A "detector" refers to a measuring device or sensor installed to collect user behavior data.

[0868] "Geographic information technology" is a technology used to track a user's movement and location based on location information.

[0869] A "machine learning algorithm" is an algorithm that analyzes collected data, recognizes patterns, and identifies anomalies.

[0870] "Emotional data" refers to data collected through cameras and voice input that indicates the user's emotions and mental state.

[0871] "Immediately communicating warnings" means notifying users and administrators without delay when abnormal behavior or conditions are detected.

[0872] To implement this invention, smartphones and wearable devices are used as information devices to manage users' schedule information and activity data. In this process, the data must be securely stored in a cloud database, which is an information recording medium. A server connects to the information terminal via internet communication and generates notifications based on the schedule information received from the user. These notifications are sent to the information terminal at a pre-set time, allowing the user to confirm them visually or audibly.

[0873] The information terminal incorporates accelerometers and GPS as detectors, periodically collecting the user's location information and body movements. This data is transmitted to a server in combination with geographic information technology and analyzed using machine learning algorithms. If abnormal behavior is detected as a result of the analysis, the server immediately communicates a warning to the user and administrator to ensure security.

[0874] Furthermore, the information terminal is equipped with a high-performance camera and microphone, which analyze facial expressions and voice tone to collect emotional data. This allows the server to analyze the user's emotional state and generate appropriate responses. For example, if the user is feeling down, the server might generate a prompt such as "Suggesting activities to refresh you," providing encouragement to the user through the information terminal.

[0875] For example, if a user enters a walking schedule at 2:00 PM every day, the information device will generate a notification at 1:55 PM that says, "It's almost time for your walk. Let's start getting ready." Furthermore, if sentiment data reveals that the user is feeling stressed, it will offer advice such as, "Why not try listening to your favorite music while you walk?" An example of a prompt message would be an instruction such as, "Suggesting the next action to take."

[0876] With the above configuration, it is possible to comprehensively support the user's behavior and emotional state, enabling them to live a safe and independent life.

[0877] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0878] Step 1:

[0879] Users input their schedule information using smartphones or wearable devices. This input includes the date, time, and details of the schedule. The entered data is temporarily stored within the information device and prepared for transmission to the server.

[0880] Step 2:

[0881] The terminal transmits the schedule information entered by the user to the server via the internet. The server receives this data and stores it in a database for each user. The stored data is used for subsequent processing based on the scheduled time.

[0882] Step 3:

[0883] The server references the stored schedule information and generates notifications at the specified times. As part of the data processing, it creates notification content from the schedule information and adds it to the notification registration list. The generated notifications are formatted according to the notification format (audio or text) specified by the user.

[0884] Step 4:

[0885] The server sends the generated notification to the device. The device receives this notification and presents it to the user visually or audibly. Specifically, it displays a pop-up notification on the device screen and, if necessary, provides an audio alert. This allows the user to confirm and appropriately perform the next action.

[0886] Step 5:

[0887] The device uses built-in sensors and GPS technology to collect user movement data. This data includes motion information such as location, speed, and direction. The collected data is periodically transmitted to a server.

[0888] Step 6:

[0889] The server receives behavioral data sent from the terminal and analyzes it using machine learning algorithms. It identifies behavioral patterns from the input data and performs data calculations to identify abnormal behavior. If an anomaly is detected, it generates a warning for immediate action.

[0890] Step 7:

[0891] The server sends the generated warning to the user and their supervisor. The terminal displays this warning as a notification, prompting the user to check for safety. This allows users and supervisors to take prompt action.

[0892] Step 8:

[0893] The device uses a camera and voice input device to collect information about the user's emotions. It analyzes facial expressions and voice tone to determine the emotional state in real time. The emotional data is sent to a server.

[0894] Step 9:

[0895] The server analyzes emotional data and generates responses best suited to the user's mental state. Using a generative AI model, it creates appropriate prompts and suggestions based on the input emotional information. As a result, it recommends actions that contribute to improving the user's quality of life.

[0896] Step 10:

[0897] The server sends the generated responses and suggestions to the terminal and notifies the user. The terminal collects the user's responses and sends the feedback to the server to further improve the system's accuracy. This enables flexible support tailored to the user's needs.

[0898] (Application Example 2)

[0899] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0900] For dementia patients and the elderly, managing their daily lives and ensuring safety are significant challenges. Furthermore, the inability to respond appropriately to emotional fluctuations can exacerbate mental anxiety. Conventional support systems are limited to collecting movement data and providing schedule notifications, failing to adequately offer flexible support tailored to the user's emotional state.

[0901] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0902] In this invention, the server includes means for analyzing video and audio data using an emotion engine to estimate the user's emotional state, means for providing guidance to the user based on the estimated emotional state, and means for monitoring changes in the emotional state in real time and providing support for ensuring safety. This enables detailed support and safety supervision in response to fluctuations in the user's emotions.

[0903] "A means of inputting user schedule information and saving it to a database" refers to a function that provides an interface for users to input their schedules and activities, and efficiently stores that information in a database.

[0904] "A means of generating notifications at a predetermined time based on a database and sending them to each user's device" refers to a function that creates notifications at the appropriate time based on a pre-registered schedule and sends them to devices such as smart devices.

[0905] "A means of presenting notifications received by the user's device in audio or text and prompting the user to take appropriate action" refers to an interface that displays instructions received by the user on their device in audio or text format, guiding them to the next action.

[0906] "Means for collecting user motion data using sensors and location information technology" refers to a function that uses technologies such as sensors and GPS to acquire information about the user's body movements and location.

[0907] "Means for analyzing collected operational data to detect anomalies and generating warnings when an anomaly is detected" refers to a function that analyzes operational data to identify patterns that are different from the norm and creates a warning message when necessary.

[0908] "Means of sending warnings to users and their administrators to ensure security" refers to communication methods for notifying users and administrators of urgent situations and prompting a swift response.

[0909] "A means of analyzing video and audio data using an emotion engine to estimate the user's emotional state" refers to a function that analyzes data acquired from cameras and microphones to evaluate and judge the user's emotions.

[0910] "Means of providing guidance to users based on estimated emotional states" refers to functions that provide appropriate advice and instructions in accordance with the user's emotions.

[0911] "A means of monitoring changes in emotional state in real time and providing support for ensuring safety" refers to a system that quickly detects fluctuations in emotions and provides support for ensuring safety based on those fluctuations.

[0912] The system implementing this invention consists of a group of smart devices used by dementia patients and the elderly to ensure self-management and safety. At the core of the system is a server equipped with an emotion engine. The server receives the user's schedule information and stores it in a database. Based on the schedule, it generates notifications at predetermined times and sends them to the user's mobile device. This process enables the user to perform their daily activities without forgetting.

[0913] On the device, received notifications are presented to the user in voice or text format, prompting them to take their planned actions. The device incorporates sensors and location technology to collect user activity data. The server analyzes the collected data using AI algorithms and immediately generates a warning and notifies the administrator if an anomaly is detected.

[0914] Furthermore, because it is equipped with an emotion engine, it can estimate the user's emotional state in real time based on video and audio data acquired by the camera and microphone of smart glasses or wearable devices. Based on the estimated emotional state, the device provides personalized advice and guidance to the user. Technologies used in this process include OpenCV for image recognition and speech recognition APIs for speech analysis, while communication services such as Twilio are used for notifications.

[0915] For example, if a user appears lost while walking in a park, the device will ask, "Where do you want to go now?" and guide them in the correct direction by referring to their location information. Another example of a prompt using the generated AI model is, "Tell me a phrase to use when I forget where I want to go." This helps users to act with confidence even in difficult situations.

[0916] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0917] Step 1:

[0918] The user enters schedule information via a smart device. The device sends the entered data to the server. The server receives this data and stores it in a database. The input consists of schedule details and time, and the output is storage in the database.

[0919] Step 2:

[0920] The server generates notifications at predetermined times based on stored schedules. These notifications include the next tasks to be performed. The generated notifications are sent to the terminal. The input is schedule information from the database, and the output is a notification in the appropriate format.

[0921] Step 3:

[0922] The device presents received notifications to the user as audio or text, prompting the user to take the next action. The input is notification data from the server, and the output is the presentation of information to the user.

[0923] Step 4:

[0924] The device's sensors collect user movement data and location information in real time. This data is periodically transmitted to a server. The input is movement and location data from the sensors, and the output is the data transmission to the server.

[0925] Step 5:

[0926] The server analyzes the collected behavioral data using an AI algorithm. When an abnormal behavioral pattern is detected, it generates a warning and sends a notification to the user and their administrator. The input is behavioral and location data, and the output is a warning message.

[0927] Step 6:

[0928] The device collects video and audio data from the user using its camera and microphone and transmits it to the emotion engine. The emotion engine analyzes this data and estimates the emotional state. The input is video and audio data, and the output is an evaluation of the emotional state.

[0929] Step 7:

[0930] The server generates a guidance message based on the estimated emotional state and sends it to the terminal. The terminal then presents this guidance to the user. The input is emotional state data, and the output is the guidance message.

[0931] Step 8:

[0932] Users receive guidance through their devices and modify their actions as needed. Specifically, users can use the presented information to move around and complete tasks with confidence. The input is guidance messages, and the output is modified behavior.

[0933] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0934] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0935] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0936] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0937] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0938] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0939] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0940] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0941] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0942] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0943] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0944] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0945] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0946] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0947] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0948] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0949] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0950] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0951] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0952] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0953] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0954] The following is further disclosed regarding the embodiments described above.

[0955] (Claim 1)

[0956] A means of inputting user schedule information and saving it to a database,

[0957] A means for generating notifications at a predetermined time based on a database and sending them to each user terminal,

[0958] A means of presenting notifications received by the user's device via voice or text, prompting the user to take appropriate action,

[0959] A means of collecting user movement data using sensors and location information technology,

[0960] A means for analyzing collected operational data to detect anomalies and generating a warning when an anomaly is detected,

[0961] A means of sending warnings to users and their administrators to ensure security,

[0962] A system that includes this.

[0963] (Claim 2)

[0964] A means of inputting a sequence of daily tasks, breaking them down, and saving them to a database,

[0965] A means of guiding the user through the next task via audio or video, upon request.

[0966] The system according to claim 1, including the following:

[0967] (Claim 3)

[0968] A means of analyzing operating patterns using an AI algorithm for anomaly detection and sending warnings in real time when anomalies occur,

[0969] The system according to claim 1, including the following:

[0970] "Example 1"

[0971] (Claim 1)

[0972] A means for inputting user activity plan information and saving it in the information storage unit,

[0973] A means for generating a notification at a predetermined time based on the information storage unit and transmitting it to each user device,

[0974] A means of presenting notifications received by the user's device in audio or text format, prompting the user to take appropriate action,

[0975] A means of collecting user movement information using sensors and location information technology,

[0976] A means for analyzing collected operational information to detect anomalies and generating a warning when an anomaly is detected,

[0977] A means of sending warnings to users and their administrators to ensure security,

[0978] A system that includes this.

[0979] (Claim 2)

[0980] A means of inputting a series of daily work procedures, breaking them down, and storing them in an information storage unit,

[0981] A means of guiding the user through the next steps to be taken, either by voice or video, upon request from the user.

[0982] The system according to claim 1, including the following:

[0983] (Claim 3)

[0984] A means of analyzing operating patterns using machine learning algorithms for anomaly detection and immediately sending warnings when anomalies occur,

[0985] The system according to claim 1, including the following:

[0986] "Application Example 1"

[0987] (Claim 1)

[0988] A means for inputting user activity information and saving it to an information management device,

[0989] A means for generating a notification at a predetermined time based on an information management device and transmitting it to each user device,

[0990] A means of prompting the user to take appropriate action by presenting notifications received by the device via voice or text,

[0991] A means of collecting user movement information using sensors and location data technology,

[0992] A means for analyzing collected operational information using an analysis device to detect abnormalities and generating a warning signal when an abnormality is detected,

[0993] A means of ensuring safety by transmitting warning signals to users and their administrators,

[0994] A method for inputting a purchase list and presenting product information via audio or video based on location information,

[0995] A means of monitoring the behavior of users within the store and notifying the store manager if an abnormality occurs,

[0996] A system that includes this.

[0997] (Claim 2)

[0998] A means of inputting the procedures for daily work, disassembling the device, and saving the information to an information management device,

[0999] The system according to claim 1, which, upon request from the user, provides voice or video guidance on the next task to be performed.

[1000] (Claim 3)

[1001] The system according to claim 1, which analyzes operating patterns using machine learning techniques for anomaly detection and transmits a warning signal in real time when an anomaly occurs.

[1002] "Example 2 of combining an emotion engine"

[1003] (Claim 1)

[1004] A means for inputting user schedule information into an information device and saving it to an information recording medium,

[1005] A means for generating a notification at a predetermined time based on an information recording medium and communicating it to each information terminal,

[1006] A means of displaying notifications received by an information terminal in audio or text, prompting the user to take appropriate action,

[1007] A means of collecting user movement data using detectors and geographic information technology,

[1008] A means for analyzing collected movement data to identify anomalies and generating a warning when an anomaly is detected,

[1009] Means for communicating warnings to users and their supervisors and ensuring safety,

[1010] A means for collecting emotional data via camera and voice input, and for an information device to analyze it,

[1011] A means of promptly suggesting actions appropriate to the user's emotional state based on collected emotional data,

[1012] A system that includes this.

[1013] (Claim 2)

[1014] A means for inputting a sequence of daily tasks, breaking it down, and saving it to an information recording medium,

[1015] The system according to claim 1, which, upon request from the user, provides instructions on the next task to be performed via audio or video.

[1016] (Claim 3)

[1017] The system according to claim 1, which analyzes operating patterns using a machine learning algorithm for anomaly identification and immediately communicates a warning when an anomaly occurs.

[1018] "Application example 2 when combining with an emotional engine"

[1019] (Claim 1)

[1020] A means of inputting user schedule information and saving it to a database,

[1021] A means for generating notifications at a predetermined time based on a database and sending them to each user terminal,

[1022] A means of presenting notifications received by the user's device via voice or text, prompting the user to take appropriate action,

[1023] A means of collecting user movement data using sensors and location information technology,

[1024] A means for analyzing collected operational data to detect anomalies and generating a warning when an anomaly is detected,

[1025] A means of sending warnings to users and their administrators to ensure security,

[1026] A means for analyzing video and audio data using an emotion engine to estimate the user's emotional state,

[1027] A means of providing guidance to users based on their estimated emotional state,

[1028] A system that includes this.

[1029] (Claim 2)

[1030] A means of inputting a sequence of daily tasks, breaking them down, and saving them to a database,

[1031] A means of guiding the user through the next task via audio or video, upon request.

[1032] A means of providing personalized voice messages based on emotional state using smart devices,

[1033] The system according to claim 1, including the following:

[1034] (Claim 3)

[1035] A means of analyzing operating patterns using an AI algorithm for anomaly detection and sending warnings in real time when anomalies occur,

[1036] A means of monitoring changes in emotional state in real time and providing support to ensure safety,

[1037] The system according to claim 1, including the following: [Explanation of Symbols]

[1038] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of inputting user schedule information and saving it to a database, A means for generating notifications at a predetermined time based on a database and sending them to each user terminal, A means of presenting notifications received by the user's device via voice or text, prompting the user to take appropriate action, A means of collecting user movement data using sensors and location information technology, A means for analyzing collected operational data to detect anomalies and generating a warning when an anomaly is detected, A means of sending warnings to users and their administrators to ensure security, A system that includes this.

2. A means of inputting a sequence of daily tasks, breaking them down, and saving them to a database, A means of guiding the user through the next task via audio or video, upon request. The system according to claim 1, including the following:

3. A means of analyzing operating patterns using an AI algorithm for anomaly detection and sending warnings in real time when anomalies occur, The system according to claim 1, including the following:

Images