System

Abstract

A system is provided.SOLUTION: The system includes a camera monitor means for monitoring the behavior and expression of a care-receiver and detecting an anxiety state, an excretion prediction means for collecting bio data of the care-receiver and predicting the necessity of excretion, a control means for determining a response to the care-receiver based on the anxiety state and the necessity of excretion, and a display means for displaying an avatar familiar to the care-receiver to divert the attention of the care-receiver from excretion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] This invention relates to a system for reducing the burden of toileting assistance. In particular, it aims to efficiently manage the toileting needs of care recipients with physical disabilities and short-term memory impairments, thereby reducing the physical and mental burden on caregivers. In conventional caregiving, when a care recipient repeatedly requests to go to the toilet, if the actual need for toileting is unclear, a significant mental and physical burden is placed on the caregiver. In addition, care recipients with short-term memory impairments have difficulty accurately recognizing and explaining their toileting needs, making it difficult to provide appropriate assistance. [Means for solving the problem]

[0005] The present invention solves the above problems by providing the following means: A system is configured including a camera monitoring means for monitoring the behavior and facial expressions of the care recipient and detecting their state of anxiety, an excretion prediction means for collecting biodata of the care recipient and predicting their need for excretion, a control means for determining how to respond to the care recipient based on their state of anxiety and the need for excretion, and a display means for displaying a friendly avatar to the care recipient to divert their attention from excretion. This system accurately determines whether the care recipient needs to use the toilet based on their behavior and biodata, and notifies the caregiver when necessary. When it is not necessary, the avatar will speak to the care recipient, thereby reducing their anxiety and allowing appropriate care to be provided.

[0006] The "camera monitoring means" is a camera system for monitoring the behavior and facial expressions of the care recipient and detecting an anxious state.

[0007] The "excretion prediction means" is a device and analysis system for collecting biodata of the care recipient and predicting the need for excretion.

[0008] "Control measures" are systems for determining how to respond to the care recipient based on their anxiety state and need for excretion.

[0009] The "display means" refers to a display and display system that displays a friendly avatar to the care recipient and diverts the care recipient's attention from excretion. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0011] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0012] First, the terms used in the following description will be explained.

[0013] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0014] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0015] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0016] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0017] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0018] [First embodiment]

[0019] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0020] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0021] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0022] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0023] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0024] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0025] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0026] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0027] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0028] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0029] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0030] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0031] This invention is a system for efficiently managing excretion assistance in the living space of a care recipient and reducing the burden on caregivers. The system includes a "camera monitoring means" that monitors the behavior and facial expression of the care recipient, an "excretion prediction means" that collects biodata of the care recipient and predicts the need for excretion, a "control means" that determines how to respond to the care recipient based on the care recipient's anxiety state and the need for excretion, and a "display means" that displays a friendly avatar to the care recipient to distract the care recipient from excretion.

[0032] The program processing in this system is as follows.

[0033] Program processing

[0034] Operation of camera monitoring means

[0035] A camera installed in the living space of the user (care recipient) captures the behavior and facial expressions of the care recipient in real time. This video data is analyzed in real time by the device, and if the care recipient appears anxious (e.g., looking around restlessly, showing anxious expressions), that state of anxiety is detected.

[0036] Operation of the excretion prediction means

[0037] The excretion prediction device attached to the care recipient continuously measures biodata such as heart rate and skin temperature and transmits the data to a terminal, which then analyzes the data with an excretion prediction algorithm to determine whether the care recipient needs to go to the toilet.

[0038] Operation of the control means

[0039] The control means provided by the server determines how to respond to the care recipient based on the anxiety state data and excretion prediction data sent from the terminal. Specifically, when the care recipient needs to go to the toilet, the server sends a notification to the caregiver's terminal informing them that toilet assistance is required.

[0040] Display operation

[0041] When there is no need to use the toilet, the device displays a friendly avatar on the screen, which diverts the care recipient's attention by talking about topics that interest them (e.g., recent family news, hobbies, etc.).

[0042] Specific examples

[0043] When the care recipient becomes anxious on the sofa

[0044] 1. Operation of camera monitoring means

[0045] The user, who is being cared for, is watching TV on the sofa. The camera monitors the situation, and the device detects that the user is looking around anxiously.

[0046] 2. Operation of the excretion prediction means

[0047] The excretion prediction device detects increases in the care recipient's heart rate and changes in skin temperature and sends this data to a terminal, which then uses this data to predict when the care recipient needs to excrete and sends the prediction to a server.

[0048] 3. Operation of the control means

[0049] The server analyzes the need for excretion, and if it determines that the user needs a toilet, it sends a notification to the caregiver's smartphone saying that toilet assistance is required. On the other hand, if it determines that the user does not need a toilet, it sends a command to the device to display an avatar.

[0050] 4. Display operation

[0051] The device displays an avatar on the screen and speaks to the care recipient in a friendly manner, for example, "Would you like to see some photos you took recently with your family?", helping to ease their anxiety.

[0052] System Effects

[0053] This system can efficiently manage the toileting needs of care recipients, reduce the occurrence of unnecessary toileting assistance, and alleviate the anxiety of care recipients, improving their quality of life and significantly reducing the physical and mental burden on caregivers.

[0054] The processing flow will be explained below.

[0055] Step 1: Obtain camera footage (device)

[0056] The device acquires real-time video data from a camera installed in the living space of the care recipient, including the care recipient's behavior and facial expressions.

[0057] Step 2: Anxiety state detection (device)

[0058] The device analyzes the captured video data in real time and detects anxiety based on the care recipient's behavior and facial expressions, such as restlessness and anxious facial expressions.

[0059] Step 3: Biodata collection (terminal)

[0060] The excretion prediction device measures biomarkers such as the care recipient's heart rate and skin temperature in real time, and the terminal receives the measured biodata.

[0061] Step 4: Transfer data (device)

[0062] The device transmits anxiety state data and biodata to the server, which prepares the data for analysis.

[0063] Step 5: Excretion prediction calculation (server)

[0064] The server analyzes the transmitted data and applies a toileting prediction algorithm to determine whether the care recipient needs to go to the toilet. The server generates a result of "needs toilet" or "no toilet."

[0065] Step 6: Restroom Need Notification (Server)

[0066] The server sends the results of the excretion prediction to the device. If the user "needs to use the toilet," the server sends notification data to the device saying "needs to use the toilet."

[0067] Step 7: Caregiver notification (device)

[0068] When the device receives a notification that the user needs to go to the toilet, it sends a notification to the caregiver's smartphone saying, "Toilet assistance required." The caregiver receives the notification and takes the necessary measures.

[0069] Step 8: Avatar Appearance and Response (Device)

[0070] If it is determined that the user does not need to use the toilet, the device will instruct an avatar to appear on the display, which will then provide friendly conversation and encourage the user to ease their anxiety.

[0071] Step 9: Select and execute conversation content (terminal)

[0072] The device selects appropriate conversation topics based on the care recipient's interests and past data, and sends instructions to the avatar, which then starts a conversation with the care recipient and makes them forget about the toilet.

[0073] In this way, the system uses a camera monitoring means, an excretion prediction means, a control means, and a display means to reduce the anxiety of the care recipient and efficiently manage the need for excretion, thereby significantly reducing the burden on the caregiver.

[0074] Example 1

[0075] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0076] In the daily lives of elderly people and physically disabled care recipients, they often lose track of when to go to the toilet, which can lead to increased anxiety and stress. It has also been pointed out that frequent toilet assistance is required of caregivers, placing a heavy physical and mental burden on them. There is a need for a system that can solve these problems, improve the quality of life of care recipients, and reduce the burden on caregivers.

[0077] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0078] In this invention, the server includes a camera means for monitoring the behavior and facial expression of the care recipient and detecting their state of anxiety, a prediction means for collecting biometric data of the care recipient and predicting their need for excretion, and a control means for determining how to respond to the care recipient based on their state of anxiety and their need for excretion. This makes it possible to efficiently manage the toilet needs of the care recipient, reduce the need for unnecessary toilet assistance, and alleviate their anxiety.

[0079] "Care recipients" refers to elderly people who require care or those who are physically unable to live independently.

[0080] "Camera means" refers to a device that captures the actions and expressions of the care recipient in real time and acquires that data.

[0081] "Biometric data" refers to data that indicates the physical condition of the care recipient, and includes information such as heart rate and skin temperature.

[0082] "Prediction means" refers to a device or program that analyzes biological data and predicts the need for excretion of the care recipient.

[0083] The "analysis means" refers to a device or program that analyzes the video data obtained from the camera means in real time and detects an anxiety state.

[0084] "Controlling means" refers to a device or program that determines how to respond to a care recipient based on their anxiety state and need to eliminate.

[0085] "Notification means" refers to a device or program that notifies a caregiver when it is determined that an individual needs to toilet.

[0086] "Display device" refers to a device for providing visual or audio information to a care recipient.

[0087] "Presentation means" refers to a device or program that displays an avatar or the like to divert the care recipient's attention from toileting.

[0088] "Anxious state" refers to a mentally unsettled state of the care recipient, such as frequently looking around or showing an anxious expression.

[0089] The present invention is a system for efficiently managing excretion assistance in the living space of a care recipient and reducing the burden on caregivers. The system includes a "camera means" that monitors the behavior and facial expression of the care recipient, a "prediction means" that collects biometric data of the care recipient and predicts the need for excretion, a "control means" that determines how to respond to the care recipient based on the care recipient's anxiety state and the need for excretion, and a "presentation means" that uses a display device that is familiar to the care recipient to divert the care recipient's attention from excretion.

[0090] Camera Means Details

[0091] A camera installed in the living space of the care recipient captures the care recipient's behavior and facial expressions in real time. This video data is analyzed by a device (e.g., Raspberry Pi or NVIDIA Jetson Nano). The device uses an open-source computer vision library (e.g., OpenCV) to perform facial recognition and facial expression analysis, and detects anxiety states (e.g., frequently looking around, showing anxious expressions, etc.).

[0092] Details of the forecasting method

[0093] A toilet prediction device (e.g., Fitbit or Apple Watch) worn by the care recipient continuously measures biometric data such as heart rate and skin temperature and transmits the data to a terminal. The terminal analyzes this data using a toilet prediction algorithm (e.g., a machine learning model implemented in Python) to determine when the care recipient needs to go to the toilet.

[0094] Details of control measures

[0095] The anxiety state data and excretion prediction data sent from the device are received by a server (e.g., AWS (registered trademark) EC2, Google (registered trademark) Cloud VM). The server integrates this data, analyzes the condition of the care recipient, and if the care recipient needs to use the toilet, the server sends a notification to the caregiver's smartphone (e.g., iPhone (registered trademark) or Android (registered trademark) device) saying, "Toilet assistance required." Notifications are sent using services such as AWS SNS.

[0096] Presentation details

[0097] When no toileting is required, the device displays an avatar on a display device (e.g., tablet, smart display) that is familiar to the care recipient. The avatar speaks to the care recipient about topics of interest (e.g., family updates, topics related to hobbies) to divert the care recipient's attention from toileting.

[0098] Specific examples

[0099] When the care recipient becomes anxious on the sofa

[0100] 1. Camera Means Operation

[0101] The user, a care recipient, is watching TV on the sofa. A camera (e.g., Logitech C920) monitors the user's behavior, and the device (NVIDIA Jetson Nano) detects that the user is looking around anxiously.

[0102] 2. Operation of the prediction means

[0103] The toileting prediction device (e.g., Fitbit) detects an increase in the care recipient's heart rate and changes in skin temperature and sends the data to a terminal. The terminal uses this data to predict the need for toileting using a machine learning model and sends the results to a server (AWS EC2).

[0104] 3. Operation of the control means

[0105] The server analyzes the need for excretion and sends a notification to the caregiver's smartphone if it determines that the person needs to go to the toilet. On the other hand, if it determines that the person does not need to go to the toilet, it sends a command to the device to display an avatar.

[0106] 4. Presentation Method Operation

[0107] The device displays an avatar on a tablet and speaks to the care recipient in a friendly manner, for example, "Would you like to see some photos you recently took with your family?", helping to ease their anxiety.

[0108] Prompt Sentence Examples

[0109] "Tell me how to raise the alarm when a care recipient is exhibiting anxious behavior on the couch."

[0110] This system can efficiently manage the toileting needs of care recipients, reduce the occurrence of unnecessary toileting assistance, and alleviate the anxiety of care recipients, improving their quality of life and significantly reducing the physical and mental burden on caregivers.

[0111] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0112] Program processing steps

[0113] Camera measures steps

[0114] Step 1:

[0115] To monitor the behavior and facial expressions of the care recipient, a camera acquires video data of the care recipient. Specifically, a camera (e.g., Logitech C920) captures video at 30 frames per second. The input is real-time video of the care recipient, and the output is the video data acquired by the camera.

[0116] Step 2:

[0117] The device receives and analyzes the video data from the camera. The device (e.g., NVIDIA Jetson Nano) analyzes the video data using OpenCV and performs facial recognition and facial expression analysis. The input is the video data acquired in step 1, and the output is the facial expression data resulting from the analysis.

[0118] Step 3:

[0119] The device detects the anxiety state based on the analysis results. Specifically, it determines whether the user frequently looks around or shows an anxious expression. The input is the facial expression data obtained in step 2, and the output is the anxiety state determination result.

[0120] Prediction steps

[0121] Step 4:

[0122] The excretion prediction device worn by the user measures biological data (heart rate and skin temperature). Specifically, the Fitbit measures heart rate every minute and skin temperature every five minutes. The input is the care recipient's real-time biological data, and the output is the measurement data.

[0123] Step 5:

[0124] The measured biometric data is sent to the device. The Fitbit sends the data to the NVIDIA Jetson Nano via Bluetooth. The input is the measurement data obtained in step 4, and the output is the biometric data sent to the device.

[0125] Step 6:

[0126] The device analyzes the collected biometric data and performs excretion prediction. The device uses a machine learning model implemented in Python to predict the need for excretion from the biometric data. The input is the biometric data received in step 5, and the output is a prediction result indicating the need for excretion.

[0127] Control Measure Steps

[0128] Step 7:

[0129] The server receives the anxiety state data and excretion prediction data sent from the device. The server (e.g., AWS EC2) integrates and analyzes this data. The input is the anxiety state data from step 3 and the excretion prediction data from step 6, and the output is the integrated analysis results.

[0130] Step 8:

[0131] The server decides how to respond based on the integrated data. If the user needs to use the toilet, the server determines that "toileting assistance is required," and if not, issues an instruction to display an avatar. The input is the analysis result from step 7, and the output is the decision on how to respond.

[0132] Step 9:

[0133] The server sends a notification to the caregiver's smartphone. Specifically, it uses AWS SNS to send a message such as "Toileting assistance required." The input is the result of the decision on how to respond in step 8, and the output is the notification sent to the caregiver's smartphone.

[0134] Presentation steps

[0135] Step 10:

[0136] The terminal displays a friendly avatar for the care recipient. The terminal displays the avatar on a display device (e.g., a tablet or smart display). The input is an instruction to display the avatar from the server, and the output is the avatar displayed on the display device.

[0137] Step 11:

[0138] The avatar speaks to the care recipient on topics that match their interests. For example, the avatar might say, "Would you like to see some photos you recently took with your family?" to ease their anxiety. The input is a registered conversation script, and the output is a voice message to the care recipient.

[0139] (Application example 1)

[0140] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0141] Conventional nursing care systems lack the functionality to adequately manage anxiety and stress, let alone assist with toileting. Furthermore, in businesses like food delivery, there was a lack of a system for monitoring the health and work efficiency of delivery personnel in real time, making effective labor management and safety measures difficult. Given these circumstances, there is a need for a system that can appropriately manage and monitor the condition of both care recipients and delivery personnel, and respond efficiently based on that information.

[0142] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0143] In this invention, the server includes a monitoring means for monitoring the behavior and facial expression of the care recipient and detecting an anxiety state, a prediction means for collecting biological information of the care recipient and predicting the need for excretion, a control means for determining how to respond to the care recipient based on the anxiety state and the need for excretion, a display means for displaying information that is friendly to the care recipient and diverting the care recipient's attention from excretion, and a means for monitoring the health state of the delivery person and collecting data to improve delivery efficiency. This not only enables efficient management of the care recipient's anxiety and excretion, but also enables safe and efficient labor management by monitoring the health state of the delivery person in real time.

[0144] A "monitoring means" is a tool used to monitor a subject's behavior, facial expressions, and biometric information in real time and detect specific conditions.

[0145] A "prediction tool" is a tool that analyzes collected biological data and predicts specific needs or conditions (e.g., the need to defecate or stress conditions).

[0146] "Control means" are tools that determine the optimal response to the target person based on information obtained from the monitoring means and prediction means, and provide instructions and notifications.

[0147] A "display means" is an interface that presents information to a subject and encourages or diverts attention to a particular action.

[0148] "Means for collecting data" refers to tools for continuously collecting biometric and behavioral data of subjects and transmitting them to the system.

[0149] "Notification means" is a tool for notifying the target person or administrator in real time and encouraging them to take the necessary action.

[0150] A specific system for implementing the present invention mainly includes the following hardware and software components:

[0151] 1. System Configuration

[0152] Monitoring method: Camera that captures user behavior and facial expressions in real time.

[0153] Examples: high resolution cameras, infrared cameras

[0154] Prediction means: A biosensor that acquires the user's biological information.

[0155] Examples: Heart rate sensor, skin temperature sensor

[0156] Control means: A control program executed on the server.

[0157] Examples: Linux (registered trademark) server, AWS (Amazon Web Services)

[0158] Display means: A display that displays information and an avatar display program.

[0159] Examples: tablets, smartphones, smart glasses

[0160] 2. Program Processing

[0161] The program performs complex data analysis and processing using data obtained from each means. The main processing contents are shown below.

[0162] monitoring means

[0163] The camera captures the user's behavior and facial expressions in real time and transmits the video data to a server, which then analyzes the video data using a generative AI model to detect anxiety states.

[0164] Hardware: High-resolution camera

[0165] Software: Image analysis software (e.g. OpenCV)

[0166] Prediction methods

[0167] Biosensors measure the user's heart rate and skin temperature and transmit this vital information to a server, which uses predictive algorithms to analyze the need for urination.

[0168] Hardware: Heart rate sensor, skin temperature sensor

[0169] Software: Data analysis software (e.g., Python, NumPy)

[0170] Control means

[0171] The server determines an appropriate response to the user based on the data obtained from the monitoring means and the prediction means, and notifies the caregiver and the user by the notification means.

[0172] Hardware: Server

[0173] Software: Communication software (e.g. HTTP API)

[0174] Display means

[0175] The server generates an avatar based on the user's state and displays it on the screen. The avatar provides the user with friendly information and topics to divert their attention.

[0176] Hardware: Smart displays, tablets

[0177] Software: Avatar generation software (e.g. Unity, Blender)

[0178] 3. Specific Examples

[0179] Caregiving scenario

[0180] If the user is anxious on the sofa, the camera monitors the user's behavior and sends the video data to the server in real time. The server uses a generative AI model to detect the state of anxiety and also analyzes the need for toileting based on data from the heart rate sensor and skin temperature sensor. If it is determined that toileting is necessary, a notification is sent to the caregiver.

[0181] Food Delivery Scenario

[0182] A camera installed on the doorbell monitors the customer's situation and detects if the customer is absent or unable to answer. Along with the camera, biosensors on the delivery person measure the heart rate and body temperature and send the data to a server. The server analyzes this data, monitors the delivery person's health, and plans a safe and efficient delivery.

[0183] Prompt Sentence Examples

[0184] "Please submit your idea for developing an application for a food delivery system that monitors the delivery person's biodata (heart rate, body temperature) and the customer's situation (doorbell camera footage) and suggests the optimal delivery schedule."

[0185] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0186] Step 1:

[0187] The camera captures the user's actions and facial expressions in real time, capturing video data.

[0188] Input: Real-time video data

[0189] Output: Captured video data

[0190] Specific operation: The camera captures the user's actions and facial expressions, and the data is immediately sent to the server.

[0191] Step 2:

[0192] The server receives the video data and analyzes it in real time using a generative AI model to detect anxiety states.

[0193] Input: Captured video data

[0194] Output: Anxiety state judgment result (e.g., anxious, normal)

[0195] Specific behavior: The generative AI model analyzes video data and detects anxiety from the user's facial expressions and behaviors. For example, it determines whether the user frequently looks around or shows anxious facial expressions.

[0196] Step 3:

[0197] The biosensor measures the user's heart rate and skin temperature and transmits the biometric data to a server.

[0198] Input: Biometric information such as heart rate and skin temperature

[0199] Output: Acquired biometric data

[0200] Specific operation: A biosensor worn by the user continuously measures data such as heart rate and skin temperature and sends it to a server.

[0201] Step 4:

[0202] The server receives the biometric data and predicts the need for toileting using an excretion prediction algorithm.

[0203] Input: Acquired biometric data

[0204] Output: Judgment result of necessity of excretion (e.g., necessary, unnecessary)

[0205] How it works: The server analyzes biometric data and predicts the need to urinate based on certain patterns (e.g., increased heart rate or changes in skin temperature).

[0206] Step 5:

[0207] The server determines how to respond to the user based on the results of the judgment of the anxiety state and the need for excretion.

[0208] Input: Anxiety state judgment result, judgment result of need for excretion

[0209] Output: Decide how to respond (e.g., notify caregiver, display avatar)

[0210] Specific operation: The server comprehensively analyzes the anxiety state and toileting need data and decides whether to send a notification to the caregiver or display an avatar to the user.

[0211] Step 6:

[0212] The server sends a notification to the caregiver or displays an avatar on the display based on the response decision.

[0213] Input: Decision result of response method

[0214] Output: Sending a notification or displaying an avatar

[0215] Specific behavior: When the server sends a notification to the caregiver's smartphone, the caregiver receives a message such as "Toileting assistance required." Alternatively, an avatar appears on the user's display and speaks a voice message such as "Would you like to see some recent photos taken with your family?"

[0216] Step 7:

[0217] The doorbell camera monitors the customer's situation and detects if they are absent or unable to answer the door, sending the information to the server.

[0218] Input: Doorbell camera video data

[0219] Output: Customer status determination result (e.g., at home, away)

[0220] Specific operation: The doorbell camera monitors the customer's status, and if the customer is not present, the data is sent to the server and the absence of the customer is detected.

[0221] Step 8:

[0222] The delivery person's biosensor measures their heart rate and body temperature and sends the data to a server.

[0223] Input: Delivery person's heart rate, body temperature

[0224] Output: Retrieved biodata

[0225] Specific operation: A biosensor attached to the delivery person continuously measures heart rate and body temperature data and transmits it to a server.

[0226] Step 9:

[0227] The server analyzes the delivery person's biodata and creates an optimal delivery plan to improve delivery efficiency.

[0228] Input: Delivery person's biodata

[0229] Output: Optimal delivery schedule

[0230] Specific operation: The server analyzes the delivery person's heart rate and body temperature data, and if the delivery person is tired or stressed, it suggests taking a break or recalculates the optimal delivery route.

[0231] Step 10:

[0232] The server sends a notification to the customer and delivery person, prompting them to take appropriate action.

[0233] Input: Delivery schedule determination result

[0234] Output: Send notification

[0235] Specific operation: The server sends a notification to the customer such as "The delivery person will arrive soon" and a notification to the delivery person such as "Please take a break."

[0236] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0237] The present invention relates to a system for reducing the burden of assisting with excretion. In particular, it aims to efficiently manage the toilet needs of care recipients with physical disabilities and short-term memory impairments, thereby reducing the physical and mental burden on caregivers. To this end, in addition to camera monitoring means, excretion prediction means, control means, and display means, it uses an emotion engine that recognizes the user's emotions. The system of the present invention is configured as follows:

[0238] Program processing

[0239] Operation of camera monitoring means

[0240] A camera installed in the living space of the user (care recipient) captures the behavior and facial expressions of the care recipient in real time. This video data is analyzed in real time by the device, and if the care recipient appears anxious (e.g., looking around restlessly, showing anxious expressions), that state of anxiety is detected.

[0241] Operation of anxiety state detection and analysis means

[0242] The device analyzes the acquired video data in real time to detect anxiety, and the emotion engine analyzes the emotions of the care recipient based on the video data and transmits the emotion data to the control means.

[0243] Operation of the excretion prediction means

[0244] The device, worn by the care recipient, continuously measures biomarkers such as heart rate and skin temperature and transmits the data to a terminal, which then analyzes the data with a predictive algorithm to determine whether the care recipient needs to go to the toilet.

[0245] Operation of the control means

[0246] The control means provided by the server determines how to deal with the care recipient based on the anxiety state data, emotion data, and biodata sent from the terminal. Specifically, when the care recipient needs to go to the toilet, the server sends a notification to the caregiver's terminal informing them that toilet assistance is required.

[0247] Display operation

[0248] When there is no need to use the toilet, the device displays a friendly avatar on the screen for the care recipient. Based on the emotional data obtained from the emotion engine, the displayed avatar will talk to the care recipient about topics that interest them (e.g., recent family news, topics related to hobbies) to distract them.

[0249] Specific examples

[0250] When the care recipient becomes anxious on the sofa

[0251] 1. Operation of camera monitoring means

[0252] The user, who is being cared for, is watching TV on the sofa. The camera monitors the situation, and the device detects that the user is looking around anxiously.

[0253] 2. Operation of anxiety state detection and analysis method

[0254] The device detects the anxiety state of the care recipient and sends the data to the server. In addition, the emotion engine analyzes the care recipient's emotions (e.g., fear, relief, etc.) from the video data and sends the emotion data to the control means.

[0255] 3. Operation of the excretion prediction means

[0256] The excretion prediction device detects increases in the care recipient's heart rate and changes in skin temperature and sends this data to a terminal, which then uses this data to predict when the care recipient needs to excrete and sends the prediction to a server.

[0257] 4. Operation of the control means

[0258] The server analyzes the need for excretion and emotional data, and if it determines that the user needs to use the toilet, it sends a notification to the caregiver's smartphone saying that toilet assistance is required. On the other hand, if it determines that the user does not need to use the toilet, it sends a command to the device to display an avatar.

[0259] 5. Operation of the display means

[0260] The device displays an avatar on the screen, and the avatar speaks to the person being cared for in a friendly manner based on emotional data obtained from the emotion engine.

[0261] 6. Selecting and executing conversation content

[0262] The device selects appropriate conversation topics based on the care recipient's interests and emotions and sends instructions to the avatar, which then starts a conversation with the care recipient to ease their anxiety.

[0263] System Effects

[0264] This system can efficiently manage the toileting needs of care recipients and reduce the occurrence of unnecessary toileting assistance. It also alleviates the anxiety of care recipients, improving their quality of life and significantly reducing the physical and mental burden on caregivers. Utilizing an emotion engine enables more personalized responses, further improving the sense of security of care recipients.

[0265] The processing flow will be explained below.

[0266] Step 1: Obtain camera footage (device)

[0267] The device acquires real-time video data from a camera installed in the living space of the care recipient, including the care recipient's behavior and facial expressions.

[0268] Step 2: Anxiety state detection (device)

[0269] The device analyzes the captured video data in real time and detects anxiety based on the care recipient's behavior and facial expressions, such as restlessness and anxious facial expressions.

[0270] Step 3: Sentiment Analysis (Device)

[0271] The device uses an emotion engine to analyze the facial expressions of the care recipient to obtain emotional data, which includes the care recipient's emotional state (e.g., joy, sadness, fear, etc.).

[0272] Step 4: Biodata collection (terminal)

[0273] The excretion prediction device measures biomarkers such as the care recipient's heart rate and skin temperature in real time, and the terminal receives the measured biodata.

[0274] Step 5: Transfer data (device)

[0275] The terminal transmits the anxiety state data, emotion data, and biodata to the server, which prepares the data for analysis.

[0276] Step 6: Excretion prediction calculation (server)

[0277] The server analyzes the transmitted data and applies a toileting prediction algorithm to determine whether the care recipient needs to go to the toilet. The server generates a result of "needs toilet" or "no toilet."

[0278] Step 7: Restroom Need Notification (Server)

[0279] The server sends the results of the excretion prediction to the device. If the user "needs to use the toilet," the server sends notification data to the device saying "needs to use the toilet."

[0280] Step 8: Caregiver notification (device)

[0281] When the device receives a notification that the user needs to go to the toilet, it sends a notification to the caregiver's smartphone saying, "Toilet assistance required." The caregiver receives the notification and takes the necessary measures.

[0282] Step 9: Avatar Appearance and Response (Device)

[0283] If it is determined that the user does not need to use the toilet, the device will instruct an avatar to appear on the display, which will then provide friendly conversation and encourage the user to ease their anxiety.

[0284] Step 10: Select and execute conversation content (terminal)

[0285] The device selects appropriate conversation content based on the care recipient's emotional data and past data, and sends instructions to the avatar, which then starts a conversation with the care recipient and makes them forget about the toilet.

[0286] In this way, the system uses a camera monitoring means, an excretion prediction means, a control means, a display means, and an emotion engine to reduce anxiety in the care recipient and efficiently manage the need for excretion, thereby significantly reducing the burden on the caregiver.

[0287] Example 2

[0288] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0289] Conventional care systems have had difficulty quickly and accurately detecting the toilet needs and anxiety states of care recipients and providing appropriate responses. This has resulted in increased physical and mental burdens on caregivers and a decline in the quality of life of care recipients. Specifically, the accuracy of detecting anxiety states and predicting excretion needs has been low, making it difficult to respond in real time.

[0290] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0291] In this invention, the server includes a monitoring means for monitoring the behavior and facial expression of the care recipient and detecting an anxiety state, a prediction means for collecting physiological data of the care recipient and predicting the need for excretion, a control means for determining how to respond to the care recipient based on the anxiety state and the need for excretion, a display means for displaying a character that is friendly to the care recipient on the screen to distract the care recipient, and an emotion analysis means for recognizing the user's emotions. This makes it possible to monitor and analyze the condition of the care recipient in real time and take prompt and appropriate action.

[0292] The "monitoring method" is a means for photographing and recording the behavior and facial expressions of the care recipient in real time and analyzing the data.

[0293] The "prediction means" is a means for collecting physiological data such as the heart rate and skin temperature of the care recipient, and analyzing this data to determine the need for excretion.

[0294] The "control means" is a means for determining the condition of the care recipient based on data obtained from the monitoring means and prediction means, and for instructing appropriate responses.

[0295] The "display means" is a means for displaying a character that is friendly to the care recipient on the screen and serving to divert the care recipient's attention.

[0296] The "emotion analysis means" is a means for analyzing the behavior and facial expressions of the care recipient and recognizing their emotions.

[0297] The "notification means" is a means for notifying the caregiver when it is determined that the care recipient needs to go to the toilet based on the data acquired from the prediction means.

[0298] This invention is a system that aims to efficiently manage the toilet needs of a user (i.e., a care recipient) and reduce the physical and mental burden on the caregiver. This system includes a monitoring means, a prediction means, a control means, a display means, and an emotion analysis means.

[0299] Specific system configuration

[0300] 1. Monitoring measures

[0301] The monitoring method is a camera installed in the living space of the care recipient. This camera captures the care recipient's behavior and facial expressions in real time and collects the data. For example, a scene where a care recipient is watching TV on the sofa can be filmed, and the care recipient's behavior (e.g., looking around restlessly, anxious expressions, etc.) can be recorded.

[0302] 2. Prediction methods

[0303] The prediction method is to use an excretion prediction device attached to the care recipient. This device continuously measures physiological data such as heart rate and skin temperature and sends the data to a terminal. The terminal analyzes this data using an excretion prediction algorithm (machine learning model) to determine whether the care recipient needs to go to the toilet.

[0304] 3. Control Measures

[0305] The control means is a central server. The server determines the optimal way to respond to the care recipient based on anxiety state data, emotional data, and physiological data sent from the device. For example, if the server determines that the care recipient needs to go to the toilet, it sends a notification to the caregiver's device informing them that toilet assistance is required.

[0306] 4. Display means

[0307] The display is placed in a position where the care recipient can see it, and a friendly avatar is displayed on the display. The avatar distracts the care recipient by talking to them about topics that interest them based on data obtained from the emotion analysis method.

[0308] 5. Emotion analysis method

[0309] The emotion analysis means uses an emotion engine built into the device. The emotion engine analyzes the video data and recognizes the emotions of the care recipient. This emotion data is sent to the control means and used to determine the optimal response.

[0310] Specific examples

[0311] When the care recipient becomes anxious on the sofa

[0312] 1. When a user is sitting on a sofa, a camera monitors the user's behavior and detects the user looking around anxiously.

[0313] 2. The device acquires the video data, analyzes it in real time, detects the anxiety state, and sends the data to the server.

[0314] 3. Furthermore, the emotion engine analyzes the user's emotions (e.g., fear, relief, etc.) from the video data and sends that data to the control means.

[0315] 4. The excretion prediction device measures the user's heart rate and skin temperature and sends the data to the device, which then uses this data to determine the need for excretion.

[0316] 5. The server analyzes all of this data and decides on the appropriate course of action. If it determines that the person needs to go to the toilet, it sends a notification to the caregiver's device saying, "Toileting assistance required."

[0317] 6. If it is determined that the user does not need to use the toilet, the device will display an avatar on the screen and the avatar will speak to the user in a friendly manner.

[0318] Example prompts for generative AI models

[0319] "If your care recipient is anxiously looking around on the couch, instruct your avatar on how to assist them. Also, choose topics that will help the avatar to reassure the care recipient."

[0320] In this way, the invention provides a system that can efficiently manage the toilet needs of the care recipient and significantly reduce the burden on the caregiver by linking monitoring means, prediction means, control means, display means, and emotion analysis means.

[0321] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0322] Step 1:

[0323] A user sits on the sofa and starts watching TV. The camera starts operating and captures the user's actions and expressions in real time as a means of monitoring. The input is the video data captured by the camera, and the output is the data sent to the terminal.

[0324] Step 2:

[0325] The device receives video data sent from the camera. The input is video data, and the device analyzes this data in real time to detect patterns of the user's behavior and facial expressions. The output is the detected anxiety state.

[0326] Step 3:

[0327] The device sends the detected anxiety state data to the server. The input is the anxiety state data, and the output is the anxiety state data sent to the server.

[0328] Step 4:

[0329] At the same time, the device uses an emotion engine as an emotion analysis means to analyze the user's emotions from the video data. The input is the video data, which is analyzed by the emotion engine. The output is the analyzed emotion data.

[0330] Step 5:

[0331] The emotion data generated by the emotion engine is received by the terminal and sent to the server, which is the control means. The input is emotion data, and the output is the emotion data sent to the server.

[0332] Step 6:

[0333] The excretion prediction device worn by the user measures heart rate and skin temperature. The input is the user's physiological data, which is sent to a terminal via Bluetooth or other means. The output is physiological data.

[0334] Step 7:

[0335] The device uses the received physiological data to execute an excretion prediction algorithm. The input is heart rate and skin temperature data, and the device analyzes this data to predict the need for excretion. The output is the determination of the need for excretion.

[0336] Step 8:

[0337] The device determines whether the user needs to defecate and sends the result to the server. The input is the result of the determination, and the output is the result sent to the server.

[0338] Step 9:

[0339] The server analyzes the collected anxiety state data, emotional data, and toileting need data to determine how to respond to the care recipient. The input is the above data, and the appropriate response method is determined based on the analysis results. The output is a response method decision.

[0340] Step 10:

[0341] If the server determines that the user needs to go to the toilet, it sends a notification to the caregiver's device. The input is the decision on how to respond, and the output is the notification to the caregiver's device.

[0342] Step 11:

[0343] If it is determined that the user does not need to use the toilet, the terminal displays an avatar on the display. The input is instructions from the server, and the output is the avatar displayed on the display.

[0344] Step 12:

[0345] The avatar speaks to the user in a friendly manner based on the emotion data acquired by the emotion engine. The input is the emotion data, and the output is the conversation content of the avatar displayed to the user.

[0346] This system aims to improve the efficiency and quality of care by having the user, server, and terminal work together at each step to monitor and manage the toilet needs and psychological state of the care recipient in real time.

[0347] (Application example 2)

[0348] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0349] Conventional health management systems face the challenge of detecting workers' anxiety and stress levels in real time and providing appropriate responses quickly. Furthermore, health prediction systems based on workers' biodata are often not highly accurate, resulting in ineffective health management. Furthermore, health management information and reminders are not provided in a user-friendly format, preventing workers from maintaining their health and improving their productivity.

[0350] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0351] In this invention, the server includes a camera monitor that monitors the behavior and facial expressions of workers and detects their stress levels, a health prediction unit that collects biometric data from workers and predicts their health status, a control unit that determines how to respond to workers based on their anxiety and stress levels, and a display unit that provides workers with friendly health management information and reminders. This allows for highly accurate real-time monitoring and analysis of workers' health status and stress levels, enabling prompt and appropriate responses. Furthermore, providing workers with health management information and reminders through friendly avatars helps maintain their health and improve productivity.

[0352] "Care recipients" refers to people such as elderly people or people with disabilities who require support and care.

[0353] "Worker" refers to a worker engaged in work in a particular workplace.

[0354] "Camera monitoring means" refers to a device or system that has the function of monitoring the behavior and facial expressions of a subject using a camera.

[0355] The term "excretion prediction means" refers to a device or system that has the function of predicting the need for excretion based on the biodata of the care recipient.

[0356] "Health prediction means" refers to a device or system that has the function of predicting the health status of a worker based on the worker's biodata.

[0357] "Control means" refers to a device or system that has the ability to determine appropriate responses or actions based on collected data.

[0358] "Display means" refers to a device or system that has the function of visualizing and displaying information.

[0359] "Anxiety state" refers to a state in which the care recipient or worker feels mental anxiety or stress.

[0360] "Biodata" refers to data about living organisms, such as heart rate and skin temperature.

[0361] An "avatar" is a computer-generated character that appears as a user's alter ego or stand-in.

[0362] "Notification means" refers to a device or system that has the function of sending specific information or alerts to a target person.

[0363] "Analysis means" refers to a device or system that has the function of analyzing collected data and extracting useful information.

[0364] The present invention relates to a health management system for workers using factory robots. This system is composed of a camera monitor, a health prediction unit, a control unit, and a display unit.

[0365] Overall system configuration

[0366] Camera monitoring means

[0367] The server uses a camera monitor to monitor the behavior and facial expressions of workers. The camera can be, for example, a smartphone camera or a fixed camera installed in the factory. The video data acquired from this camera is analyzed in real time. For the video analysis, a common video processing library (e.g., OpenCV) and an AI model are used.

[0368] Health prediction tools

[0369] The server uses health prediction methods to predict the health status of workers. For example, wearable devices such as smart watches are used. These devices collect biodata such as heart rate and skin temperature and send it to the server in real time. Health management software (e.g., HealthKit) is used to analyze the data.

[0370] Control means

[0371] The server has a control means for determining the appropriate response based on the collected anxiety and stress levels and biodata. Based on the analysis results, it determines the appropriate response for the worker and sends a notification to the worker and their supervisor. The notification means uses the notification function of a typical smartphone.

[0372] Display means

[0373] The server has a display means for providing workers with friendly health management information and reminders. For example, it uses an avatar to display instructions for relaxation and deep breathing to workers. For this, an avatar generation application such as Unity is used.

[0374] Specific examples

[0375] Example 1: When a worker shows signs of instability while working on a production line

[0376] The camera monitoring means monitors the worker's behavior in real time and detects anxiety. The emotion engine analyzes the anxiety state and sends the data to the server. Meanwhile, the health prediction means collects heart rate and skin temperature from the smartwatch and predicts the worker's health state. The server performs sequential analysis based on this data and determines the necessary response. For example, it sends a notification to the worker to take a break, and an avatar teaches them how to relax.

[0377] Prompt Sentence Examples

[0378] This assistant is designed to monitor the health and stress levels of industrial workers using advanced AI and wearable technology. It can detect workers' anxiety through facial expression analysis and provide real-time health predictions based on biometric data from wearable devices. When necessary, the system alerts the worker or supervisor to suggest a break or convey other health messages through an interactive avatar.

[0379] Features:

[0380] Real-time facial expression analysis using OpenCV

[0381] Emotion recognition using Azure® Cognitive Services

[0382] Health prediction using data from Apple Watch via HealthKit

[0383] Actionable alerts and notifications from cloud server

[0384] Friendly avatar interactions using Unity

[0385] Please design a machine learning model that can detect stress from facial expression data and integrate it with the existing health prediction and notification system.

[0386] effect

[0387] This system makes it possible to monitor and analyze workers' health and stress levels in real time with high accuracy, and to take appropriate measures quickly, which not only improves work efficiency and safety but also helps maintain worker health and reduce stress.

[0388] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0389] Step 1:

[0390] Acquisition of video data by camera monitor means

[0391] The server acquires video data of the workers' behavior and facial expressions in real time from cameras in the factory. The input is the video from the camera, and the output is the input video data. This video data is used to analyze anxiety and stress levels in subsequent processing steps. Specifically, the camera monitors the workers 24 hours a day and transmits the video data to the server at a constant frame rate.

[0392] Step 2:

[0393] Analysis of anxiety states using an emotion engine

[0394] The server inputs the acquired video data into the emotion engine for analysis. The input is the video data obtained in step 1, and the output is the analysis result indicating the worker's anxiety state. Specifically, the emotion engine (e.g., Azure Cognitive Services) is used to analyze the worker's facial expressions and movements from the video data, and evaluate the worker's anxiety state numerically.

[0395] Step 3:

[0396] Acquisition and analysis of biodata with health prediction tools

[0397] The server acquires biodata such as heart rate and skin temperature from the worker's wearable device. The input is the biodata from the wearable device, and the output is health status assessment data analyzed by a health prediction algorithm. Specifically, the server receives data from a smartwatch (e.g., Apple Watch) via Bluetooth and analyzes it using health management software such as HealthKit.

[0398] Step 4:

[0399] Deciding how to respond using control measures

[0400] The server determines how to respond based on the anxiety state data from the emotion engine and the health state evaluation data from the health prediction means. The input is the anxiety state data and the health state evaluation data, and the output is the decision on how to respond (e.g., instructing a break, notifying a supervisor). Specifically, a program is run that integrates the data and generates a notification instructing a break if the anxiety state is high or an abnormality in the health state is detected.

[0401] Step 5:

[0402] Sending notifications via notification methods

[0403] The server sends notifications to workers and their supervisors based on the response method determined by the control means. The input is response method data, and the output is a notification message to the recipient. Specifically, the program uses the smartphone's push notification function to send break instructions and health status alerts.

[0404] Step 6:

[0405] Providing information through display means

[0406] The server displays health management information and relaxation method reminders on the worker's smartphone or dedicated device. The input is response method data, and the output is a friendly message using an avatar. Specifically, the avatar is displayed using Unity, and a program runs that suggests ways to relax and take a break to anxious workers.

[0407] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0408] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0409] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0410] [Second embodiment]

[0411] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0412] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0413] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0414] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0415] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0416] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0417] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0418] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0419] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0420] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0421] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0422] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0423] This invention is a system for efficiently managing excretion assistance in the living space of a care recipient and reducing the burden on caregivers. The system includes a "camera monitoring means" that monitors the behavior and facial expression of the care recipient, an "excretion prediction means" that collects biodata of the care recipient and predicts the need for excretion, a "control means" that determines how to respond to the care recipient based on the care recipient's anxiety state and the need for excretion, and a "display means" that displays a friendly avatar to the care recipient to distract the care recipient from excretion.

[0424] The program processing in this system is as follows.

[0425] Program processing

[0426] Operation of camera monitoring means

[0427] A camera installed in the living space of the user (care recipient) captures the behavior and facial expressions of the care recipient in real time. This video data is analyzed in real time by the device, and if the care recipient appears anxious (e.g., looking around restlessly, showing anxious expressions), that state of anxiety is detected.

[0428] Operation of the excretion prediction means

[0429] The excretion prediction device attached to the care recipient continuously measures biodata such as heart rate and skin temperature and transmits the data to a terminal, which then analyzes the data with an excretion prediction algorithm to determine whether the care recipient needs to go to the toilet.

[0430] Operation of the control means

[0431] The control means provided by the server determines how to respond to the care recipient based on the anxiety state data and excretion prediction data sent from the terminal. Specifically, when the care recipient needs to go to the toilet, the server sends a notification to the caregiver's terminal informing them that toilet assistance is required.

[0432] Display operation

[0433] When there is no need to use the toilet, the device displays a friendly avatar on the screen, which diverts the care recipient's attention by talking about topics that interest them (e.g., recent family news, hobbies, etc.).

[0434] Specific examples

[0435] When the care recipient becomes anxious on the sofa

[0436] 1. Operation of camera monitoring means

[0437] The user, who is being cared for, is watching TV on the sofa. The camera monitors the situation, and the device detects that the user is looking around anxiously.

[0438] 2. Operation of the excretion prediction means

[0439] The excretion prediction device detects increases in the care recipient's heart rate and changes in skin temperature and sends this data to a terminal, which then uses this data to predict when the care recipient needs to excrete and sends the prediction to a server.

[0440] 3. Operation of the control means

[0441] The server analyzes the need for excretion, and if it determines that the user needs a toilet, it sends a notification to the caregiver's smartphone saying that toilet assistance is required. On the other hand, if it determines that the user does not need a toilet, it sends a command to the device to display an avatar.

[0442] 4. Display operation

[0443] The device displays an avatar on the screen and speaks to the care recipient in a friendly manner, for example, "Would you like to see some photos you took recently with your family?", helping to ease their anxiety.

[0444] System Effects

[0445] This system can efficiently manage the toileting needs of care recipients, reduce the occurrence of unnecessary toileting assistance, and alleviate the anxiety of care recipients, improving their quality of life and significantly reducing the physical and mental burden on caregivers.

[0446] The processing flow will be explained below.

[0447] Step 1: Obtain camera footage (device)

[0448] The device acquires real-time video data from a camera installed in the living space of the care recipient, including the care recipient's behavior and facial expressions.

[0449] Step 2: Anxiety state detection (device)

[0450] The device analyzes the captured video data in real time and detects anxiety based on the care recipient's behavior and facial expressions, such as restlessness and anxious facial expressions.

[0451] Step 3: Biodata collection (terminal)

[0452] The excretion prediction device measures biomarkers such as the care recipient's heart rate and skin temperature in real time, and the terminal receives the measured biodata.

[0453] Step 4: Transfer data (device)

[0454] The device transmits anxiety state data and biodata to the server, which prepares the data for analysis.

[0455] Step 5: Excretion prediction calculation (server)

[0456] The server analyzes the transmitted data and applies a toileting prediction algorithm to determine whether the care recipient needs to go to the toilet. The server generates a result of "needs toilet" or "no toilet."

[0457] Step 6: Restroom Need Notification (Server)

[0458] The server sends the results of the excretion prediction to the device. If the user "needs to use the toilet," the server sends notification data to the device saying "needs to use the toilet."

[0459] Step 7: Caregiver notification (device)

[0460] When the device receives a notification that the user needs to go to the toilet, it sends a notification to the caregiver's smartphone saying, "Toilet assistance required." The caregiver receives the notification and takes the necessary measures.

[0461] Step 8: Avatar Appearance and Response (Device)

[0462] If it is determined that the user does not need to use the toilet, the device will instruct an avatar to appear on the display, which will then provide friendly conversation and encourage the user to ease their anxiety.

[0463] Step 9: Select and execute conversation content (terminal)

[0464] The device selects appropriate conversation topics based on the care recipient's interests and past data, and sends instructions to the avatar, which then starts a conversation with the care recipient and makes them forget about the toilet.

[0465] In this way, the system uses a camera monitoring means, an excretion prediction means, a control means, and a display means to reduce the anxiety of the care recipient and efficiently manage the need for excretion, thereby significantly reducing the burden on the caregiver.

[0466] Example 1

[0467] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0468] In the daily lives of elderly people and physically disabled care recipients, they often lose track of when to go to the toilet, which can lead to increased anxiety and stress. It has also been pointed out that frequent toilet assistance is required of caregivers, placing a heavy physical and mental burden on them. There is a need for a system that can solve these problems, improve the quality of life of care recipients, and reduce the burden on caregivers.

[0469] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0470] In this invention, the server includes a camera means for monitoring the behavior and facial expression of the care recipient and detecting their state of anxiety, a prediction means for collecting biometric data of the care recipient and predicting their need for excretion, and a control means for determining how to respond to the care recipient based on their state of anxiety and their need for excretion. This makes it possible to efficiently manage the toilet needs of the care recipient, reduce the need for unnecessary toilet assistance, and alleviate their anxiety.

[0471] "Care recipients" refers to elderly people who require care or those who are physically unable to live independently.

[0472] "Camera means" refers to a device that captures the actions and expressions of the care recipient in real time and acquires that data.

[0473] "Biometric data" refers to data that indicates the physical condition of the care recipient, and includes information such as heart rate and skin temperature.

[0474] "Prediction means" refers to a device or program that analyzes biological data and predicts the need for excretion of the care recipient.

[0475] The "analysis means" refers to a device or program that analyzes the video data obtained from the camera means in real time and detects an anxiety state.

[0476] "Controlling means" refers to a device or program that determines how to respond to a care recipient based on their anxiety state and need to eliminate.

[0477] "Notification means" refers to a device or program that notifies a caregiver when it is determined that an individual needs to toilet.

[0478] "Display device" refers to a device for providing visual or audio information to a care recipient.

[0479] "Presentation means" refers to a device or program that displays an avatar or the like to divert the care recipient's attention from toileting.

[0480] "Anxious state" refers to a mentally unsettled state of the care recipient, such as frequently looking around or showing an anxious expression.

[0481] The present invention is a system for efficiently managing excretion assistance in the living space of a care recipient and reducing the burden on caregivers. The system includes a "camera means" that monitors the behavior and facial expression of the care recipient, a "prediction means" that collects biometric data of the care recipient and predicts the need for excretion, a "control means" that determines how to respond to the care recipient based on the care recipient's anxiety state and the need for excretion, and a "presentation means" that uses a display device that is familiar to the care recipient to divert the care recipient's attention from excretion.

[0482] Camera Means Details

[0483] A camera installed in the living space of the care recipient captures the care recipient's behavior and facial expressions in real time. This video data is analyzed by a device (e.g., Raspberry Pi or NVIDIA Jetson Nano). The device uses an open-source computer vision library (e.g., OpenCV) to perform facial recognition and facial expression analysis, and detects anxiety states (e.g., frequently looking around, showing anxious expressions, etc.).

[0484] Details of the forecasting method

[0485] A toilet prediction device (e.g., Fitbit or Apple Watch) worn by the care recipient continuously measures biometric data such as heart rate and skin temperature and transmits the data to a terminal. The terminal analyzes this data using a toilet prediction algorithm (e.g., a machine learning model implemented in Python) to determine when the care recipient needs to go to the toilet.

[0486] Details of control measures

[0487] The anxiety state data and excretion prediction data sent from the device are received by a server (e.g., AWS EC2, Google Cloud VM). The server integrates this data, analyzes the care recipient's situation, and if the user needs to use the toilet, the server sends a notification to the caregiver's smartphone (e.g., iPhone or Android device) saying, "Toilet assistance required." Notifications are sent using services such as AWS SNS.

[0488] Presentation details

[0489] When no toileting is required, the device displays an avatar on a display device (e.g., tablet, smart display) that is familiar to the care recipient. The avatar speaks to the care recipient about topics of interest (e.g., family updates, topics related to hobbies) to divert the care recipient's attention from toileting.

[0490] Specific examples

[0491] When the care recipient becomes anxious on the sofa

[0492] 1. Camera Means Operation

[0493] The user, a care recipient, is watching TV on the sofa. A camera (e.g., Logitech C920) monitors the user's behavior, and the device (NVIDIA Jetson Nano) detects that the user is looking around anxiously.

[0494] 2. Operation of the prediction means

[0495] The toileting prediction device (e.g., Fitbit) detects an increase in the care recipient's heart rate and changes in skin temperature and sends the data to a terminal. The terminal uses this data to predict the need for toileting using a machine learning model and sends the results to a server (AWS EC2).

[0496] 3. Operation of the control means

[0497] The server analyzes the need for excretion and sends a notification to the caregiver's smartphone if it determines that the person needs to go to the toilet. On the other hand, if it determines that the person does not need to go to the toilet, it sends a command to the device to display an avatar.

[0498] 4. Presentation Method Operation

[0499] The device displays an avatar on a tablet and speaks to the care recipient in a friendly manner, for example, "Would you like to see some photos you recently took with your family?", helping to ease their anxiety.

[0500] Prompt Sentence Examples

[0501] "Tell me how to raise the alarm when a care recipient is exhibiting anxious behavior on the couch."

[0502] This system can efficiently manage the toileting needs of care recipients, reduce the occurrence of unnecessary toileting assistance, and alleviate the anxiety of care recipients, improving their quality of life and significantly reducing the physical and mental burden on caregivers.

[0503] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0504] Program processing steps

[0505] Camera measures steps

[0506] Step 1:

[0507] To monitor the behavior and facial expressions of the care recipient, a camera acquires video data of the care recipient. Specifically, a camera (e.g., Logitech C920) captures video at 30 frames per second. The input is real-time video of the care recipient, and the output is the video data acquired by the camera.

[0508] Step 2:

[0509] The device receives and analyzes the video data from the camera. The device (e.g., NVIDIA Jetson Nano) analyzes the video data using OpenCV and performs facial recognition and facial expression analysis. The input is the video data acquired in step 1, and the output is the facial expression data resulting from the analysis.

[0510] Step 3:

[0511] The device detects the anxiety state based on the analysis results. Specifically, it determines whether the user frequently looks around or shows an anxious expression. The input is the facial expression data obtained in step 2, and the output is the anxiety state determination result.

[0512] Prediction steps

[0513] Step 4:

[0514] The excretion prediction device worn by the user measures biological data (heart rate and skin temperature). Specifically, the Fitbit measures heart rate every minute and skin temperature every five minutes. The input is the care recipient's real-time biological data, and the output is the measurement data.

[0515] Step 5:

[0516] The measured biometric data is sent to the device. The Fitbit sends the data to the NVIDIA Jetson Nano via Bluetooth. The input is the measurement data obtained in step 4, and the output is the biometric data sent to the device.

[0517] Step 6:

[0518] The device analyzes the collected biometric data and performs excretion prediction. The device uses a machine learning model implemented in Python to predict the need for excretion from the biometric data. The input is the biometric data received in step 5, and the output is a prediction result indicating the need for excretion.

[0519] Control Measure Steps

[0520] Step 7:

[0521] The server receives the anxiety state data and excretion prediction data sent from the device. The server (e.g., AWS EC2) integrates and analyzes this data. The input is the anxiety state data from step 3 and the excretion prediction data from step 6, and the output is the integrated analysis results.

[0522] Step 8:

[0523] The server decides how to respond based on the integrated data. If the user needs to use the toilet, the server determines that "toileting assistance is required," and if not, issues an instruction to display an avatar. The input is the analysis result from step 7, and the output is the decision on how to respond.

[0524] Step 9:

[0525] The server sends a notification to the caregiver's smartphone. Specifically, it uses AWS SNS to send a message such as "Toileting assistance required." The input is the result of the decision on how to respond in step 8, and the output is the notification sent to the caregiver's smartphone.

[0526] Presentation steps

[0527] Step 10:

[0528] The terminal displays a friendly avatar for the care recipient. The terminal displays the avatar on a display device (e.g., a tablet or smart display). The input is an instruction to display the avatar from the server, and the output is the avatar displayed on the display device.

[0529] Step 11:

[0530] The avatar speaks to the care recipient on topics that match their interests. For example, the avatar might say, "Would you like to see some photos you recently took with your family?" to ease their anxiety. The input is a registered conversation script, and the output is a voice message to the care recipient.

[0531] (Application example 1)

[0532] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0533] Conventional nursing care systems lack the functionality to adequately manage anxiety and stress, let alone assist with toileting. Furthermore, in businesses like food delivery, there was a lack of a system for monitoring the health and work efficiency of delivery personnel in real time, making effective labor management and safety measures difficult. Given these circumstances, there is a need for a system that can appropriately manage and monitor the condition of both care recipients and delivery personnel, and respond efficiently based on that information.

[0534] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0535] In this invention, the server includes a monitoring means for monitoring the behavior and facial expression of the care recipient and detecting an anxiety state, a prediction means for collecting biological information of the care recipient and predicting the need for excretion, a control means for determining how to respond to the care recipient based on the anxiety state and the need for excretion, a display means for displaying information that is friendly to the care recipient and diverting the care recipient's attention from excretion, and a means for monitoring the health state of the delivery person and collecting data to improve delivery efficiency. This not only enables efficient management of the care recipient's anxiety and excretion, but also enables safe and efficient labor management by monitoring the health state of the delivery person in real time.

[0536] A "monitoring means" is a tool used to monitor a subject's behavior, facial expressions, and biometric information in real time and detect specific conditions.

[0537] A "prediction tool" is a tool that analyzes collected biological data and predicts specific needs or conditions (e.g., the need to defecate or stress conditions).

[0538] "Control means" are tools that determine the optimal response to the target person based on information obtained from the monitoring means and prediction means, and provide instructions and notifications.

[0539] A "display means" is an interface that presents information to a subject and encourages or diverts attention to a particular action.

[0540] "Means for collecting data" refers to tools for continuously collecting biometric and behavioral data of subjects and transmitting them to the system.

[0541] "Notification means" is a tool for notifying the target person or administrator in real time and encouraging them to take the necessary action.

[0542] A specific system for implementing the present invention mainly includes the following hardware and software components:

[0543] 1. System Configuration

[0544] Monitoring method: Camera that captures user behavior and facial expressions in real time.

[0545] Examples: high resolution cameras, infrared cameras

[0546] Prediction means: A biosensor that acquires the user's biological information.

[0547] Examples: Heart rate sensor, skin temperature sensor

[0548] Control means: A control program executed on the server.

[0549] Example: Linux server, AWS (Amazon Web Services)

[0550] Display means: A display that displays information and an avatar display program.

[0551] Examples: tablets, smartphones, smart glasses

[0552] 2. Program Processing

[0553] The program performs complex data analysis and processing using data obtained from each means. The main processing contents are shown below.

[0554] monitoring means

[0555] The camera captures the user's behavior and facial expressions in real time and transmits the video data to a server, which then analyzes the video data using a generative AI model to detect anxiety states.

[0556] Hardware: High-resolution camera

[0557] Software: Image analysis software (e.g. OpenCV)

[0558] Prediction methods

[0559] Biosensors measure the user's heart rate and skin temperature and transmit this vital information to a server, which uses predictive algorithms to analyze the need for urination.

[0560] Hardware: Heart rate sensor, skin temperature sensor

[0561] Software: Data analysis software (e.g., Python, NumPy)

[0562] Control means

[0563] The server determines an appropriate response to the user based on the data obtained from the monitoring means and the prediction means, and notifies the caregiver and the user by the notification means.

[0564] Hardware: Server

[0565] Software: Communication software (e.g. HTTP API)

[0566] Display means

[0567] The server generates an avatar based on the user's state and displays it on the screen. The avatar provides the user with friendly information and topics to divert their attention.

[0568] Hardware: Smart displays, tablets

[0569] Software: Avatar generation software (e.g. Unity, Blender)

[0570] 3. Specific Examples

[0571] Caregiving scenario

[0572] If the user is anxious on the sofa, the camera monitors the user's behavior and sends the video data to the server in real time. The server uses a generative AI model to detect the state of anxiety and also analyzes the need for toileting based on data from the heart rate sensor and skin temperature sensor. If it is determined that toileting is necessary, a notification is sent to the caregiver.

[0573] Food Delivery Scenario

[0574] A camera installed on the doorbell monitors the customer's situation and detects if the customer is absent or unable to answer. Along with the camera, biosensors on the delivery person measure the heart rate and body temperature and send the data to a server. The server analyzes this data, monitors the delivery person's health, and plans a safe and efficient delivery.

[0575] Prompt Sentence Examples

[0576] "Please submit your idea for developing an application for a food delivery system that monitors the delivery person's biodata (heart rate, body temperature) and the customer's situation (doorbell camera footage) and suggests the optimal delivery schedule."

[0577] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0578] Step 1:

[0579] The camera captures the user's actions and facial expressions in real time, capturing video data.

[0580] Input: Real-time video data

[0581] Output: Captured video data

[0582] Specific operation: The camera captures the user's actions and facial expressions, and the data is immediately sent to the server.

[0583] Step 2:

[0584] The server receives the video data and analyzes it in real time using a generative AI model to detect anxiety states.

[0585] Input: Captured video data

[0586] Output: Anxiety state judgment result (e.g., anxious, normal)

[0587] Specific behavior: The generative AI model analyzes video data and detects anxiety from the user's facial expressions and behaviors. For example, it determines whether the user frequently looks around or shows anxious facial expressions.

[0588] Step 3:

[0589] The biosensor measures the user's heart rate and skin temperature and transmits the biometric data to a server.

[0590] Input: Biometric information such as heart rate and skin temperature

[0591] Output: Acquired biometric data

[0592] Specific operation: A biosensor worn by the user continuously measures data such as heart rate and skin temperature and sends it to a server.

[0593] Step 4:

[0594] The server receives the biometric data and predicts the need for toileting using an excretion prediction algorithm.

[0595] Input: Acquired biometric data

[0596] Output: Judgment result of necessity of excretion (e.g., necessary, unnecessary)

[0597] How it works: The server analyzes biometric data and predicts the need to urinate based on certain patterns (e.g., increased heart rate or changes in skin temperature).

[0598] Step 5:

[0599] The server determines how to respond to the user based on the results of the judgment of the anxiety state and the need for excretion.

[0600] Input: Anxiety state judgment result, judgment result of need for excretion

[0601] Output: Decide how to respond (e.g., notify caregiver, display avatar)

[0602] Specific operation: The server comprehensively analyzes the anxiety state and toileting need data and decides whether to send a notification to the caregiver or display an avatar to the user.

[0603] Step 6:

[0604] The server sends a notification to the caregiver or displays an avatar on the display based on the response decision.

[0605] Input: Decision result of response method

[0606] Output: Sending a notification or displaying an avatar

[0607] Specific behavior: When the server sends a notification to the caregiver's smartphone, the caregiver receives a message such as "Toileting assistance required." Alternatively, an avatar appears on the user's display and speaks a voice message such as "Would you like to see some recent photos taken with your family?"

[0608] Step 7:

[0609] The doorbell camera monitors the customer's situation and detects if they are absent or unable to answer the door, sending the information to the server.

[0610] Input: Doorbell camera video data

[0611] Output: Customer status determination result (e.g., at home, away)

[0612] Specific operation: The doorbell camera monitors the customer's status, and if the customer is not present, the data is sent to the server and the absence of the customer is detected.

[0613] Step 8:

[0614] The delivery person's biosensor measures their heart rate and body temperature and sends the data to a server.

[0615] Input: Delivery person's heart rate, body temperature

[0616] Output: Retrieved biodata

[0617] Specific operation: A biosensor attached to the delivery person continuously measures heart rate and body temperature data and transmits it to a server.

[0618] Step 9:

[0619] The server analyzes the delivery person's biodata and creates an optimal delivery plan to improve delivery efficiency.

[0620] Input: Delivery person's biodata

[0621] Output: Optimal delivery schedule

[0622] Specific operation: The server analyzes the delivery person's heart rate and body temperature data, and if the delivery person is tired or stressed, it suggests taking a break or recalculates the optimal delivery route.

[0623] Step 10:

[0624] The server sends a notification to the customer and delivery person, prompting them to take appropriate action.

[0625] Input: Delivery schedule determination result

[0626] Output: Send notification

[0627] Specific operation: The server sends a notification to the customer such as "The delivery person will arrive soon" and a notification to the delivery person such as "Please take a break."

[0628] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0629] The present invention relates to a system for reducing the burden of assisting with excretion. In particular, it aims to efficiently manage the toilet needs of care recipients with physical disabilities and short-term memory impairments, thereby reducing the physical and mental burden on caregivers. To this end, in addition to camera monitoring means, excretion prediction means, control means, and display means, it uses an emotion engine that recognizes the user's emotions. The system of the present invention is configured as follows:

[0630] Program processing

[0631] Operation of camera monitoring means

[0632] A camera installed in the living space of the user (care recipient) captures the behavior and facial expressions of the care recipient in real time. This video data is analyzed in real time by the device, and if the care recipient appears anxious (e.g., looking around restlessly, showing anxious expressions), that state of anxiety is detected.

[0633] Operation of anxiety state detection and analysis means

[0634] The device analyzes the acquired video data in real time to detect anxiety, and the emotion engine analyzes the emotions of the care recipient based on the video data and transmits the emotion data to the control means.

[0635] Operation of the excretion prediction means

[0636] The device, worn by the care recipient, continuously measures biomarkers such as heart rate and skin temperature and transmits the data to a terminal, which then analyzes the data with a predictive algorithm to determine whether the care recipient needs to go to the toilet.

[0637] Operation of the control means

[0638] The control means provided by the server determines how to deal with the care recipient based on the anxiety state data, emotion data, and biodata sent from the terminal. Specifically, when the care recipient needs to go to the toilet, the server sends a notification to the caregiver's terminal informing them that toilet assistance is required.

[0639] Display operation

[0640] When there is no need to use the toilet, the device displays a friendly avatar on the screen for the care recipient. Based on the emotional data obtained from the emotion engine, the displayed avatar will talk to the care recipient about topics that interest them (e.g., recent family news, topics related to hobbies) to distract them.

[0641] Specific examples

[0642] When the care recipient becomes anxious on the sofa

[0643] 1. Operation of camera monitoring means

[0644] The user, who is being cared for, is watching TV on the sofa. The camera monitors the situation, and the device detects that the user is looking around anxiously.

[0645] 2. Operation of anxiety state detection and analysis method

[0646] The device detects the anxiety state of the care recipient and sends the data to the server. In addition, the emotion engine analyzes the care recipient's emotions (e.g., fear, relief, etc.) from the video data and sends the emotion data to the control means.

[0647] 3. Operation of the excretion prediction means

[0648] The excretion prediction device detects increases in the care recipient's heart rate and changes in skin temperature and sends this data to a terminal, which then uses this data to predict when the care recipient needs to excrete and sends the prediction to a server.

[0649] 4. Operation of the control means

[0650] The server analyzes the need for excretion and emotional data, and if it determines that the user needs to use the toilet, it sends a notification to the caregiver's smartphone saying that toilet assistance is required. On the other hand, if it determines that the user does not need to use the toilet, it sends a command to the device to display an avatar.

[0651] 5. Operation of the display means

[0652] The device displays an avatar on the screen, and the avatar speaks to the person being cared for in a friendly manner based on emotional data obtained from the emotion engine.

[0653] 6. Selecting and executing conversation content

[0654] The device selects appropriate conversation topics based on the care recipient's interests and emotions and sends instructions to the avatar, which then starts a conversation with the care recipient to ease their anxiety.

[0655] System Effects

[0656] This system can efficiently manage the toileting needs of care recipients and reduce the occurrence of unnecessary toileting assistance. It also alleviates the anxiety of care recipients, improving their quality of life and significantly reducing the physical and mental burden on caregivers. Utilizing an emotion engine enables more personalized responses, further improving the sense of security of care recipients.

[0657] The processing flow will be explained below.

[0658] Step 1: Obtain camera footage (device)

[0659] The device acquires real-time video data from a camera installed in the living space of the care recipient, including the care recipient's behavior and facial expressions.

[0660] Step 2: Anxiety state detection (device)

[0661] The device analyzes the captured video data in real time and detects anxiety based on the care recipient's behavior and facial expressions, such as restlessness and anxious facial expressions.

[0662] Step 3: Sentiment Analysis (Device)

[0663] The device uses an emotion engine to analyze the facial expressions of the care recipient to obtain emotional data, which includes the care recipient's emotional state (e.g., joy, sadness, fear, etc.).

[0664] Step 4: Biodata collection (terminal)

[0665] The excretion prediction device measures biomarkers such as the care recipient's heart rate and skin temperature in real time, and the terminal receives the measured biodata.

[0666] Step 5: Transfer data (device)

[0667] The terminal transmits the anxiety state data, emotion data, and biodata to the server, which prepares the data for analysis.

[0668] Step 6: Excretion prediction calculation (server)

[0669] The server analyzes the transmitted data and applies a toileting prediction algorithm to determine whether the care recipient needs to go to the toilet. The server generates a result of "needs toilet" or "no toilet."

[0670] Step 7: Restroom Need Notification (Server)

[0671] The server sends the results of the excretion prediction to the device. If the user "needs to use the toilet," the server sends notification data to the device saying "needs to use the toilet."

[0672] Step 8: Caregiver notification (device)

[0673] When the device receives a notification that the user needs to go to the toilet, it sends a notification to the caregiver's smartphone saying, "Toilet assistance required." The caregiver receives the notification and takes the necessary measures.

[0674] Step 9: Avatar Appearance and Response (Device)

[0675] If it is determined that the user does not need to use the toilet, the device will instruct an avatar to appear on the display, which will then provide friendly conversation and encourage the user to ease their anxiety.

[0676] Step 10: Select and execute conversation content (terminal)

[0677] The device selects appropriate conversation content based on the care recipient's emotional data and past data, and sends instructions to the avatar, which then starts a conversation with the care recipient and makes them forget about the toilet.

[0678] In this way, the system uses a camera monitoring means, an excretion prediction means, a control means, a display means, and an emotion engine to reduce anxiety in the care recipient and efficiently manage the need for excretion, thereby significantly reducing the burden on the caregiver.

[0679] Example 2

[0680] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0681] Conventional care systems have had difficulty quickly and accurately detecting the toilet needs and anxiety states of care recipients and providing appropriate responses. This has resulted in increased physical and mental burdens on caregivers and a decline in the quality of life of care recipients. Specifically, the accuracy of detecting anxiety states and predicting excretion needs has been low, making it difficult to respond in real time.

[0682] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0683] In this invention, the server includes a monitoring means for monitoring the behavior and facial expression of the care recipient and detecting an anxiety state, a prediction means for collecting physiological data of the care recipient and predicting the need for excretion, a control means for determining how to respond to the care recipient based on the anxiety state and the need for excretion, a display means for displaying a character that is friendly to the care recipient on the screen to distract the care recipient, and an emotion analysis means for recognizing the user's emotions. This makes it possible to monitor and analyze the condition of the care recipient in real time and take prompt and appropriate action.

[0684] The "monitoring method" is a means for photographing and recording the behavior and facial expressions of the care recipient in real time and analyzing the data.

[0685] The "prediction means" is a means for collecting physiological data such as the heart rate and skin temperature of the care recipient, and analyzing this data to determine the need for excretion.

[0686] The "control means" is a means for determining the condition of the care recipient based on data obtained from the monitoring means and prediction means, and for instructing appropriate responses.

[0687] The "display means" is a means for displaying a character that is friendly to the care recipient on the screen and serving to divert the care recipient's attention.

[0688] The "emotion analysis means" is a means for analyzing the behavior and facial expressions of the care recipient and recognizing their emotions.

[0689] The "notification means" is a means for notifying the caregiver when it is determined that the care recipient needs to go to the toilet based on the data acquired from the prediction means.

[0690] This invention is a system that aims to efficiently manage the toilet needs of a user (i.e., a care recipient) and reduce the physical and mental burden on the caregiver. This system includes a monitoring means, a prediction means, a control means, a display means, and an emotion analysis means.

[0691] Specific system configuration

[0692] 1. Monitoring measures

[0693] The monitoring method is a camera installed in the living space of the care recipient. This camera captures the care recipient's behavior and facial expressions in real time and collects the data. For example, a scene where a care recipient is watching TV on the sofa can be filmed, and the care recipient's behavior (e.g., looking around restlessly, anxious expressions, etc.) can be recorded.

[0694] 2. Prediction methods

[0695] The prediction method is to use an excretion prediction device attached to the care recipient. This device continuously measures physiological data such as heart rate and skin temperature and sends the data to a terminal. The terminal analyzes this data using an excretion prediction algorithm (machine learning model) to determine whether the care recipient needs to go to the toilet.

[0696] 3. Control Measures

[0697] The control means is a central server. The server determines the optimal way to respond to the care recipient based on anxiety state data, emotional data, and physiological data sent from the device. For example, if the server determines that the care recipient needs to go to the toilet, it sends a notification to the caregiver's device informing them that toilet assistance is required.

[0698] 4. Display means

[0699] The display is placed in a position where the care recipient can see it, and a friendly avatar is displayed on the display. The avatar distracts the care recipient by talking to them about topics that interest them based on data obtained from the emotion analysis method.

[0700] 5. Emotion analysis method

[0701] The emotion analysis means uses an emotion engine built into the device. The emotion engine analyzes the video data and recognizes the emotions of the care recipient. This emotion data is sent to the control means and used to determine the optimal response.

[0702] Specific examples

[0703] When the care recipient becomes anxious on the sofa

[0704] 1. When a user is sitting on a sofa, a camera monitors the user's behavior and detects the user looking around anxiously.

[0705] 2. The device acquires the video data, analyzes it in real time, detects the anxiety state, and sends the data to the server.

[0706] 3. Furthermore, the emotion engine analyzes the user's emotions (e.g., fear, relief, etc.) from the video data and sends that data to the control means.

[0707] 4. The excretion prediction device measures the user's heart rate and skin temperature and sends the data to the device, which then uses this data to determine the need for excretion.

[0708] 5. The server analyzes all of this data and decides on the appropriate course of action. If it determines that the person needs to go to the toilet, it sends a notification to the caregiver's device saying, "Toileting assistance required."

[0709] 6. If it is determined that the user does not need to use the toilet, the device will display an avatar on the screen and the avatar will speak to the user in a friendly manner.

[0710] Example prompts for generative AI models

[0711] "If your care recipient is anxiously looking around on the couch, instruct your avatar on how to assist them. Also, choose topics that will help the avatar to reassure the care recipient."

[0712] In this way, the invention provides a system that can efficiently manage the toilet needs of the care recipient and significantly reduce the burden on the caregiver by linking monitoring means, prediction means, control means, display means, and emotion analysis means.

[0713] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0714] Step 1:

[0715] A user sits on the sofa and starts watching TV. The camera starts operating and captures the user's actions and expressions in real time as a means of monitoring. The input is the video data captured by the camera, and the output is the data sent to the terminal.

[0716] Step 2:

[0717] The device receives video data sent from the camera. The input is video data, and the device analyzes this data in real time to detect patterns of the user's behavior and facial expressions. The output is the detected anxiety state.

[0718] Step 3:

[0719] The device sends the detected anxiety state data to the server. The input is the anxiety state data, and the output is the anxiety state data sent to the server.

[0720] Step 4:

[0721] At the same time, the device uses an emotion engine as an emotion analysis means to analyze the user's emotions from the video data. The input is the video data, which is analyzed by the emotion engine. The output is the analyzed emotion data.

[0722] Step 5:

[0723] The emotion data generated by the emotion engine is received by the terminal and sent to the server, which is the control means. The input is emotion data, and the output is the emotion data sent to the server.

[0724] Step 6:

[0725] The excretion prediction device worn by the user measures heart rate and skin temperature. The input is the user's physiological data, which is sent to a terminal via Bluetooth or other means. The output is physiological data.

[0726] Step 7:

[0727] The device uses the received physiological data to execute an excretion prediction algorithm. The input is heart rate and skin temperature data, and the device analyzes this data to predict the need for excretion. The output is the determination of the need for excretion.

[0728] Step 8:

[0729] The device determines whether the user needs to defecate and sends the result to the server. The input is the result of the determination, and the output is the result sent to the server.

[0730] Step 9:

[0731] The server analyzes the collected anxiety state data, emotional data, and toileting need data to determine how to respond to the care recipient. The input is the above data, and the appropriate response method is determined based on the analysis results. The output is a response method decision.

[0732] Step 10:

[0733] If the server determines that the user needs to go to the toilet, it sends a notification to the caregiver's device. The input is the decision on how to respond, and the output is the notification to the caregiver's device.

[0734] Step 11:

[0735] If it is determined that the user does not need to use the toilet, the terminal displays an avatar on the display. The input is instructions from the server, and the output is the avatar displayed on the display.

[0736] Step 12:

[0737] The avatar speaks to the user in a friendly manner based on the emotion data acquired by the emotion engine. The input is the emotion data, and the output is the conversation content of the avatar displayed to the user.

[0738] This system aims to improve the efficiency and quality of care by having the user, server, and terminal work together at each step to monitor and manage the toilet needs and psychological state of the care recipient in real time.

[0739] (Application example 2)

[0740] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0741] Conventional health management systems face the challenge of detecting workers' anxiety and stress levels in real time and providing appropriate responses quickly. Furthermore, health prediction systems based on workers' biodata are often not highly accurate, resulting in ineffective health management. Furthermore, health management information and reminders are not provided in a user-friendly format, preventing workers from maintaining their health and improving their productivity.

[0742] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0743] In this invention, the server includes a camera monitor that monitors the behavior and facial expressions of workers and detects their stress levels, a health prediction unit that collects biometric data from workers and predicts their health status, a control unit that determines how to respond to workers based on their anxiety and stress levels, and a display unit that provides workers with friendly health management information and reminders. This allows for highly accurate real-time monitoring and analysis of workers' health status and stress levels, enabling prompt and appropriate responses. Furthermore, providing workers with health management information and reminders through friendly avatars helps maintain their health and improve productivity.

[0744] "Care recipients" refers to people such as elderly people or people with disabilities who require support and care.

[0745] "Worker" refers to a worker engaged in work in a particular workplace.

[0746] "Camera monitoring means" refers to a device or system that has the function of monitoring the behavior and facial expressions of a subject using a camera.

[0747] The term "excretion prediction means" refers to a device or system that has the function of predicting the need for excretion based on the biodata of the care recipient.

[0748] "Health prediction means" refers to a device or system that has the function of predicting the health status of a worker based on the worker's biodata.

[0749] "Control means" refers to a device or system that has the ability to determine appropriate responses or actions based on collected data.

[0750] "Display means" refers to a device or system that has the function of visualizing and displaying information.

[0751] "Anxiety state" refers to a state in which the care recipient or worker feels mental anxiety or stress.

[0752] "Biodata" refers to data about living organisms, such as heart rate and skin temperature.

[0753] An "avatar" is a computer-generated character that appears as a user's alter ego or stand-in.

[0754] "Notification means" refers to a device or system that has the function of sending specific information or alerts to a target person.

[0755] "Analysis means" refers to a device or system that has the function of analyzing collected data and extracting useful information.

[0756] The present invention relates to a health management system for workers using factory robots. This system is composed of a camera monitor, a health prediction unit, a control unit, and a display unit.

[0757] Overall system configuration

[0758] Camera monitoring means

[0759] The server uses a camera monitor to monitor the behavior and facial expressions of workers. The camera can be, for example, a smartphone camera or a fixed camera installed in the factory. The video data acquired from this camera is analyzed in real time. For the video analysis, a common video processing library (e.g., OpenCV) and an AI model are used.

[0760] Health prediction tools

[0761] The server uses health prediction methods to predict the health status of workers. For example, wearable devices such as smart watches are used. These devices collect biodata such as heart rate and skin temperature and send it to the server in real time. Health management software (e.g., HealthKit) is used to analyze the data.

[0762] Control means

[0763] The server has a control means for determining the appropriate response based on the collected anxiety and stress levels and biodata. Based on the analysis results, it determines the appropriate response for the worker and sends a notification to the worker and their supervisor. The notification means uses the notification function of a typical smartphone.

[0764] Display means

[0765] The server has a display means for providing workers with friendly health management information and reminders. For example, it uses an avatar to display instructions for relaxation and deep breathing to workers. For this, an avatar generation application such as Unity is used.

[0766] Specific examples

[0767] Example 1: When a worker shows signs of instability while working on a production line

[0768] The camera monitoring means monitors the worker's behavior in real time and detects anxiety. The emotion engine analyzes the anxiety state and sends the data to the server. Meanwhile, the health prediction means collects heart rate and skin temperature from the smartwatch and predicts the worker's health state. The server performs sequential analysis based on this data and determines the necessary response. For example, it sends a notification to the worker to take a break, and an avatar teaches them how to relax.

[0769] Prompt Sentence Examples

[0770] This assistant is designed to monitor the health and stress levels of industrial workers using advanced AI and wearable technology. It can detect workers' anxiety through facial expression analysis and provide real-time health predictions based on biometric data from wearable devices. When necessary, the system alerts the worker or supervisor to suggest a break or convey other health messages through an interactive avatar.

[0771] Features:

[0772] Real-time facial expression analysis using OpenCV

[0773] Emotion recognition using Azure Cognitive Services

[0774] Health prediction using data from Apple Watch via HealthKit

[0775] Actionable alerts and notifications from cloud server

[0776] Friendly avatar interactions using Unity

[0777] Please design a machine learning model that can detect stress from facial expression data and integrate it with the existing health prediction and notification system.

[0778] effect

[0779] This system makes it possible to monitor and analyze workers' health and stress levels in real time with high accuracy, and to take appropriate measures quickly, which not only improves work efficiency and safety but also helps maintain worker health and reduce stress.

[0780] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0781] Step 1:

[0782] Acquisition of video data by camera monitor means

[0783] The server acquires video data of the workers' behavior and facial expressions in real time from cameras in the factory. The input is the video from the camera, and the output is the input video data. This video data is used to analyze anxiety and stress levels in subsequent processing steps. Specifically, the camera monitors the workers 24 hours a day and transmits the video data to the server at a constant frame rate.

[0784] Step 2:

[0785] Analysis of anxiety states using an emotion engine

[0786] The server inputs the acquired video data into the emotion engine for analysis. The input is the video data obtained in step 1, and the output is the analysis result indicating the worker's anxiety state. Specifically, the emotion engine (e.g., Azure Cognitive Services) is used to analyze the worker's facial expressions and movements from the video data, and evaluate the worker's anxiety state numerically.

[0787] Step 3:

[0788] Acquisition and analysis of biodata with health prediction tools

[0789] The server acquires biodata such as heart rate and skin temperature from the worker's wearable device. The input is the biodata from the wearable device, and the output is health status assessment data analyzed by a health prediction algorithm. Specifically, the server receives data from a smartwatch (e.g., Apple Watch) via Bluetooth and analyzes it using health management software such as HealthKit.

[0790] Step 4:

[0791] Deciding how to respond using control measures

[0792] The server determines how to respond based on the anxiety state data from the emotion engine and the health state evaluation data from the health prediction means. The input is the anxiety state data and the health state evaluation data, and the output is the decision on how to respond (e.g., instructing a break, notifying a supervisor). Specifically, a program is run that integrates the data and generates a notification instructing a break if the anxiety state is high or an abnormality in the health state is detected.

[0793] Step 5:

[0794] Sending notifications via notification methods

[0795] The server sends notifications to workers and their supervisors based on the response method determined by the control means. The input is response method data, and the output is a notification message to the recipient. Specifically, the program uses the smartphone's push notification function to send break instructions and health status alerts.

[0796] Step 6:

[0797] Providing information through display means

[0798] The server displays health management information and relaxation method reminders on the worker's smartphone or dedicated device. The input is response method data, and the output is a friendly message using an avatar. Specifically, the avatar is displayed using Unity, and a program runs that suggests ways to relax and take a break to anxious workers.

[0799] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0800] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0801] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0802] [Third embodiment]

[0803] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0804] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0805] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0806] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0807] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0808] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0809] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0810] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0811] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0812] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0813] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0814] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0815] This invention is a system for efficiently managing excretion assistance in the living space of a care recipient and reducing the burden on caregivers. The system includes a "camera monitoring means" that monitors the behavior and facial expression of the care recipient, an "excretion prediction means" that collects biodata of the care recipient and predicts the need for excretion, a "control means" that determines how to respond to the care recipient based on the care recipient's anxiety state and the need for excretion, and a "display means" that displays a friendly avatar to the care recipient to distract the care recipient from excretion.

[0816] The program processing in this system is as follows.

[0817] Program processing

[0818] Operation of camera monitoring means

[0819] A camera installed in the living space of the user (care recipient) captures the behavior and facial expressions of the care recipient in real time. This video data is analyzed in real time by the device, and if the care recipient appears anxious (e.g., looking around restlessly, showing anxious expressions), that state of anxiety is detected.

[0820] Operation of the excretion prediction means

[0821] The excretion prediction device attached to the care recipient continuously measures biodata such as heart rate and skin temperature and transmits the data to a terminal, which then analyzes the data with an excretion prediction algorithm to determine whether the care recipient needs to go to the toilet.

[0822] Operation of the control means

[0823] The control means provided by the server determines how to respond to the care recipient based on the anxiety state data and excretion prediction data sent from the terminal. Specifically, when the care recipient needs to go to the toilet, the server sends a notification to the caregiver's terminal informing them that toilet assistance is required.

[0824] Display operation

[0825] When there is no need to use the toilet, the device displays a friendly avatar on the screen, which diverts the care recipient's attention by talking about topics that interest them (e.g., recent family news, hobbies, etc.).

[0826] Specific examples

[0827] When the care recipient becomes anxious on the sofa

[0828] 1. Operation of camera monitoring means

[0829] The user, who is being cared for, is watching TV on the sofa. The camera monitors the situation, and the device detects that the user is looking around anxiously.

[0830] 2. Operation of the excretion prediction means

[0831] The excretion prediction device detects increases in the care recipient's heart rate and changes in skin temperature and sends this data to a terminal, which then uses this data to predict when the care recipient needs to excrete and sends the prediction to a server.

[0832] 3. Operation of the control means

[0833] The server analyzes the need for excretion, and if it determines that the user needs a toilet, it sends a notification to the caregiver's smartphone saying that toilet assistance is required. On the other hand, if it determines that the user does not need a toilet, it sends a command to the device to display an avatar.

[0834] 4. Display operation

[0835] The device displays an avatar on the screen and speaks to the care recipient in a friendly manner, for example, "Would you like to see some photos you took recently with your family?", helping to ease their anxiety.

[0836] System Effects

[0837] This system can efficiently manage the toileting needs of care recipients, reduce the occurrence of unnecessary toileting assistance, and alleviate the anxiety of care recipients, improving their quality of life and significantly reducing the physical and mental burden on caregivers.

[0838] The processing flow will be explained below.

[0839] Step 1: Obtain camera footage (device)

[0840] The device acquires real-time video data from a camera installed in the living space of the care recipient, including the care recipient's behavior and facial expressions.

[0841] Step 2: Anxiety state detection (device)

[0842] The device analyzes the captured video data in real time and detects anxiety based on the care recipient's behavior and facial expressions, such as restlessness and anxious facial expressions.

[0843] Step 3: Biodata collection (terminal)

[0844] The excretion prediction device measures biomarkers such as the care recipient's heart rate and skin temperature in real time, and the terminal receives the measured biodata.

[0845] Step 4: Transfer data (device)

[0846] The device transmits anxiety state data and biodata to the server, which prepares the data for analysis.

[0847] Step 5: Excretion prediction calculation (server)

[0848] The server analyzes the transmitted data and applies a toileting prediction algorithm to determine whether the care recipient needs to go to the toilet. The server generates a result of "needs toilet" or "no toilet."

[0849] Step 6: Restroom Need Notification (Server)

[0850] The server sends the results of the excretion prediction to the device. If the user "needs to use the toilet," the server sends notification data to the device saying "needs to use the toilet."

[0851] Step 7: Caregiver notification (device)

[0852] When the device receives a notification that the user needs to go to the toilet, it sends a notification to the caregiver's smartphone saying, "Toilet assistance required." The caregiver receives the notification and takes the necessary measures.

[0853] Step 8: Avatar Appearance and Response (Device)

[0854] If it is determined that the user does not need to use the toilet, the device will instruct an avatar to appear on the display, which will then provide friendly conversation and encourage the user to ease their anxiety.

[0855] Step 9: Select and execute conversation content (terminal)

[0856] The device selects appropriate conversation topics based on the care recipient's interests and past data, and sends instructions to the avatar, which then starts a conversation with the care recipient and makes them forget about the toilet.

[0857] In this way, the system uses a camera monitoring means, an excretion prediction means, a control means, and a display means to reduce the anxiety of the care recipient and efficiently manage the need for excretion, thereby significantly reducing the burden on the caregiver.

[0858] Example 1

[0859] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0860] In the daily lives of elderly people and physically disabled care recipients, they often lose track of when to go to the toilet, which can lead to increased anxiety and stress. It has also been pointed out that frequent toilet assistance is required of caregivers, placing a heavy physical and mental burden on them. There is a need for a system that can solve these problems, improve the quality of life of care recipients, and reduce the burden on caregivers.

[0861] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0862] In this invention, the server includes a camera means for monitoring the behavior and facial expression of the care recipient and detecting their state of anxiety, a prediction means for collecting biometric data of the care recipient and predicting their need for excretion, and a control means for determining how to respond to the care recipient based on their state of anxiety and their need for excretion. This makes it possible to efficiently manage the toilet needs of the care recipient, reduce the need for unnecessary toilet assistance, and alleviate their anxiety.

[0863] "Care recipients" refers to elderly people who require care or those who are physically unable to live independently.

[0864] "Camera means" refers to a device that captures the actions and expressions of the care recipient in real time and acquires that data.

[0865] "Biometric data" refers to data that indicates the physical condition of the care recipient, and includes information such as heart rate and skin temperature.

[0866] "Prediction means" refers to a device or program that analyzes biological data and predicts the need for excretion of the care recipient.

[0867] The "analysis means" refers to a device or program that analyzes the video data obtained from the camera means in real time and detects an anxiety state.

[0868] "Controlling means" refers to a device or program that determines how to respond to a care recipient based on their anxiety state and need to eliminate.

[0869] "Notification means" refers to a device or program that notifies a caregiver when it is determined that an individual needs to toilet.

[0870] "Display device" refers to a device for providing visual or audio information to a care recipient.

[0871] "Presentation means" refers to a device or program that displays an avatar or the like to divert the care recipient's attention from toileting.

[0872] "Anxious state" refers to a mentally unsettled state of the care recipient, such as frequently looking around or showing an anxious expression.

[0873] The present invention is a system for efficiently managing excretion assistance in the living space of a care recipient and reducing the burden on caregivers. The system includes a "camera means" that monitors the behavior and facial expression of the care recipient, a "prediction means" that collects biometric data of the care recipient and predicts the need for excretion, a "control means" that determines how to respond to the care recipient based on the care recipient's anxiety state and the need for excretion, and a "presentation means" that uses a display device that is familiar to the care recipient to divert the care recipient's attention from excretion.

[0874] Camera Means Details

[0875] A camera installed in the living space of the care recipient captures the care recipient's behavior and facial expressions in real time. This video data is analyzed by a device (e.g., Raspberry Pi or NVIDIA Jetson Nano). The device uses an open-source computer vision library (e.g., OpenCV) to perform facial recognition and facial expression analysis, and detects anxiety states (e.g., frequently looking around, showing anxious expressions, etc.).

[0876] Details of the forecasting method

[0877] A toilet prediction device (e.g., Fitbit or Apple Watch) worn by the care recipient continuously measures biometric data such as heart rate and skin temperature and transmits the data to a terminal. The terminal analyzes this data using a toilet prediction algorithm (e.g., a machine learning model implemented in Python) to determine when the care recipient needs to go to the toilet.

[0878] Details of control measures

[0879] The anxiety state data and excretion prediction data sent from the device are received by a server (e.g., AWS EC2, Google Cloud VM). The server integrates this data, analyzes the care recipient's situation, and if the user needs to use the toilet, the server sends a notification to the caregiver's smartphone (e.g., iPhone or Android device) saying, "Toilet assistance required." Notifications are sent using services such as AWS SNS.

[0880] Presentation details

[0881] When no toileting is required, the device displays an avatar on a display device (e.g., tablet, smart display) that is familiar to the care recipient. The avatar speaks to the care recipient about topics of interest (e.g., family updates, topics related to hobbies) to divert the care recipient's attention from toileting.

[0882] Specific examples

[0883] When the care recipient becomes anxious on the sofa

[0884] 1. Camera Means Operation

[0885] The user, a care recipient, is watching TV on the sofa. A camera (e.g., Logitech C920) monitors the user's behavior, and the device (NVIDIA Jetson Nano) detects that the user is looking around anxiously.

[0886] 2. Operation of the prediction means

[0887] The toileting prediction device (e.g., Fitbit) detects an increase in the care recipient's heart rate and changes in skin temperature and sends the data to a terminal. The terminal uses this data to predict the need for toileting using a machine learning model and sends the results to a server (AWS EC2).

[0888] 3. Operation of the control means

[0889] The server analyzes the need for excretion and sends a notification to the caregiver's smartphone if it determines that the person needs to go to the toilet. On the other hand, if it determines that the person does not need to go to the toilet, it sends a command to the device to display an avatar.

[0890] 4. Presentation Method Operation

[0891] The device displays an avatar on a tablet and speaks to the care recipient in a friendly manner, for example, "Would you like to see some photos you recently took with your family?", helping to ease their anxiety.

[0892] Prompt Sentence Examples

[0893] "Tell me how to raise the alarm when a care recipient is exhibiting anxious behavior on the couch."

[0894] This system can efficiently manage the toileting needs of care recipients, reduce the occurrence of unnecessary toileting assistance, and alleviate the anxiety of care recipients, improving their quality of life and significantly reducing the physical and mental burden on caregivers.

[0895] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0896] Program processing steps

[0897] Camera measures steps

[0898] Step 1:

[0899] To monitor the behavior and facial expressions of the care recipient, a camera acquires video data of the care recipient. Specifically, a camera (e.g., Logitech C920) captures video at 30 frames per second. The input is real-time video of the care recipient, and the output is the video data acquired by the camera.

[0900] Step 2:

[0901] The device receives and analyzes the video data from the camera. The device (e.g., NVIDIA Jetson Nano) analyzes the video data using OpenCV and performs facial recognition and facial expression analysis. The input is the video data acquired in step 1, and the output is the facial expression data resulting from the analysis.

[0902] Step 3:

[0903] The device detects the anxiety state based on the analysis results. Specifically, it determines whether the user frequently looks around or shows an anxious expression. The input is the facial expression data obtained in step 2, and the output is the anxiety state determination result.

[0904] Prediction steps

[0905] Step 4:

[0906] The excretion prediction device worn by the user measures biological data (heart rate and skin temperature). Specifically, the Fitbit measures heart rate every minute and skin temperature every five minutes. The input is the care recipient's real-time biological data, and the output is the measurement data.

[0907] Step 5:

[0908] The measured biometric data is sent to the device. The Fitbit sends the data to the NVIDIA Jetson Nano via Bluetooth. The input is the measurement data obtained in step 4, and the output is the biometric data sent to the device.

[0909] Step 6:

[0910] The device analyzes the collected biometric data and performs excretion prediction. The device uses a machine learning model implemented in Python to predict the need for excretion from the biometric data. The input is the biometric data received in step 5, and the output is a prediction result indicating the need for excretion.

[0911] Control Measure Steps

[0912] Step 7:

[0913] The server receives the anxiety state data and excretion prediction data sent from the device. The server (e.g., AWS EC2) integrates and analyzes this data. The input is the anxiety state data from step 3 and the excretion prediction data from step 6, and the output is the integrated analysis results.

[0914] Step 8:

[0915] The server decides how to respond based on the integrated data. If the user needs to use the toilet, the server determines that "toileting assistance is required," and if not, issues an instruction to display an avatar. The input is the analysis result from step 7, and the output is the decision on how to respond.

[0916] Step 9:

[0917] The server sends a notification to the caregiver's smartphone. Specifically, it uses AWS SNS to send a message such as "Toileting assistance required." The input is the result of the decision on how to respond in step 8, and the output is the notification sent to the caregiver's smartphone.

[0918] Presentation steps

[0919] Step 10:

[0920] The terminal displays a friendly avatar for the care recipient. The terminal displays the avatar on a display device (e.g., a tablet or smart display). The input is an instruction to display the avatar from the server, and the output is the avatar displayed on the display device.

[0921] Step 11:

[0922] The avatar speaks to the care recipient on topics that match their interests. For example, the avatar might say, "Would you like to see some photos you recently took with your family?" to ease their anxiety. The input is a registered conversation script, and the output is a voice message to the care recipient.

[0923] (Application example 1)

[0924] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0925] Conventional nursing care systems lack the functionality to adequately manage anxiety and stress, let alone assist with toileting. Furthermore, in businesses like food delivery, there was a lack of a system for monitoring the health and work efficiency of delivery personnel in real time, making effective labor management and safety measures difficult. Given these circumstances, there is a need for a system that can appropriately manage and monitor the condition of both care recipients and delivery personnel, and respond efficiently based on that information.

[0926] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0927] In this invention, the server includes a monitoring means for monitoring the behavior and facial expression of the care recipient and detecting an anxiety state, a prediction means for collecting biological information of the care recipient and predicting the need for excretion, a control means for determining how to respond to the care recipient based on the anxiety state and the need for excretion, a display means for displaying information that is friendly to the care recipient and diverting the care recipient's attention from excretion, and a means for monitoring the health state of the delivery person and collecting data to improve delivery efficiency. This not only enables efficient management of the care recipient's anxiety and excretion, but also enables safe and efficient labor management by monitoring the health state of the delivery person in real time.

[0928] A "monitoring means" is a tool used to monitor a subject's behavior, facial expressions, and biometric information in real time and detect specific conditions.

[0929] A "prediction tool" is a tool that analyzes collected biological data and predicts specific needs or conditions (e.g., the need to defecate or stress conditions).

[0930] "Control means" are tools that determine the optimal response to the target person based on information obtained from the monitoring means and prediction means, and provide instructions and notifications.

[0931] A "display means" is an interface that presents information to a subject and encourages or diverts attention to a particular action.

[0932] "Means for collecting data" refers to tools for continuously collecting biometric and behavioral data of subjects and transmitting them to the system.

[0933] "Notification means" is a tool for notifying the target person or administrator in real time and encouraging them to take the necessary action.

[0934] A specific system for implementing the present invention mainly includes the following hardware and software components:

[0935] 1. System Configuration

[0936] Monitoring method: Camera that captures user behavior and facial expressions in real time.

[0937] Examples: high resolution cameras, infrared cameras

[0938] Prediction means: A biosensor that acquires the user's biological information.

[0939] Examples: Heart rate sensor, skin temperature sensor

[0940] Control means: A control program executed on the server.

[0941] Example: Linux server, AWS (Amazon Web Services)

[0942] Display means: A display that displays information and an avatar display program.

[0943] Examples: tablets, smartphones, smart glasses

[0944] 2. Program Processing

[0945] The program performs complex data analysis and processing using data obtained from each means. The main processing contents are shown below.

[0946] monitoring means

[0947] The camera captures the user's behavior and facial expressions in real time and transmits the video data to a server, which then analyzes the video data using a generative AI model to detect anxiety states.

[0948] Hardware: High-resolution camera

[0949] Software: Image analysis software (e.g. OpenCV)

[0950] Prediction methods

[0951] Biosensors measure the user's heart rate and skin temperature and transmit this vital information to a server, which uses predictive algorithms to analyze the need for urination.

[0952] Hardware: Heart rate sensor, skin temperature sensor

[0953] Software: Data analysis software (e.g., Python, NumPy)

[0954] Control means

[0955] The server determines an appropriate response to the user based on the data obtained from the monitoring means and the prediction means, and notifies the caregiver and the user by the notification means.

[0956] Hardware: Server

[0957] Software: Communication software (e.g. HTTP API)

[0958] Display means

[0959] The server generates an avatar based on the user's state and displays it on the screen. The avatar provides the user with friendly information and topics to divert their attention.

[0960] Hardware: Smart displays, tablets

[0961] Software: Avatar generation software (e.g. Unity, Blender)

[0962] 3. Specific Examples

[0963] Caregiving scenario

[0964] If the user is anxious on the sofa, the camera monitors the user's behavior and sends the video data to the server in real time. The server uses a generative AI model to detect the state of anxiety and also analyzes the need for toileting based on data from the heart rate sensor and skin temperature sensor. If it is determined that toileting is necessary, a notification is sent to the caregiver.

[0965] Food Delivery Scenario

[0966] A camera installed on the doorbell monitors the customer's situation and detects if the customer is absent or unable to answer. Along with the camera, biosensors on the delivery person measure the heart rate and body temperature and send the data to a server. The server analyzes this data, monitors the delivery person's health, and plans a safe and efficient delivery.

[0967] Prompt Sentence Examples

[0968] "Please submit your idea for developing an application for a food delivery system that monitors the delivery person's biodata (heart rate, body temperature) and the customer's situation (doorbell camera footage) and suggests the optimal delivery schedule."

[0969] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0970] Step 1:

[0971] The camera captures the user's actions and facial expressions in real time, capturing video data.

[0972] Input: Real-time video data

[0973] Output: Captured video data

[0974] Specific operation: The camera captures the user's actions and facial expressions, and the data is immediately sent to the server.

[0975] Step 2:

[0976] The server receives the video data and analyzes it in real time using a generative AI model to detect anxiety states.

[0977] Input: Captured video data

[0978] Output: Anxiety state judgment result (e.g., anxious, normal)

[0979] Specific behavior: The generative AI model analyzes video data and detects anxiety from the user's facial expressions and behaviors. For example, it determines whether the user frequently looks around or shows anxious facial expressions.

[0980] Step 3:

[0981] The biosensor measures the user's heart rate and skin temperature and transmits the biometric data to a server.

[0982] Input: Biometric information such as heart rate and skin temperature

[0983] Output: Acquired biometric data

[0984] Specific operation: A biosensor worn by the user continuously measures data such as heart rate and skin temperature and sends it to a server.

[0985] Step 4:

[0986] The server receives the biometric data and predicts the need for toileting using an excretion prediction algorithm.

[0987] Input: Acquired biometric data

[0988] Output: Judgment result of necessity of excretion (e.g., necessary, unnecessary)

[0989] How it works: The server analyzes biometric data and predicts the need to urinate based on certain patterns (e.g., increased heart rate or changes in skin temperature).

[0990] Step 5:

[0991] The server determines how to respond to the user based on the results of the judgment of the anxiety state and the need for excretion.

[0992] Input: Anxiety state judgment result, judgment result of need for excretion

[0993] Output: Decide how to respond (e.g., notify caregiver, display avatar)

[0994] Specific operation: The server comprehensively analyzes the anxiety state and toileting need data and decides whether to send a notification to the caregiver or display an avatar to the user.

[0995] Step 6:

[0996] The server sends a notification to the caregiver or displays an avatar on the display based on the response decision.

[0997] Input: Decision result of response method

[0998] Output: Sending a notification or displaying an avatar

[0999] Specific behavior: When the server sends a notification to the caregiver's smartphone, the caregiver receives a message such as "Toileting assistance required." Alternatively, an avatar appears on the user's display and speaks a voice message such as "Would you like to see some recent photos taken with your family?"

[1000] Step 7:

[1001] The doorbell camera monitors the customer's situation and detects if they are absent or unable to answer the door, sending the information to the server.

[1002] Input: Doorbell camera video data

[1003] Output: Customer status determination result (e.g., at home, away)

[1004] Specific operation: The doorbell camera monitors the customer's status, and if the customer is not present, the data is sent to the server and the absence of the customer is detected.

[1005] Step 8:

[1006] The delivery person's biosensor measures their heart rate and body temperature and sends the data to a server.

[1007] Input: Delivery person's heart rate, body temperature

[1008] Output: Retrieved biodata

[1009] Specific operation: A biosensor attached to the delivery person continuously measures heart rate and body temperature data and transmits it to a server.

[1010] Step 9:

[1011] The server analyzes the delivery person's biodata and creates an optimal delivery plan to improve delivery efficiency.

[1012] Input: Delivery person's biodata

[1013] Output: Optimal delivery schedule

[1014] Specific operation: The server analyzes the delivery person's heart rate and body temperature data, and if the delivery person is tired or stressed, it suggests taking a break or recalculates the optimal delivery route.

[1015] Step 10:

[1016] The server sends a notification to the customer and delivery person, prompting them to take appropriate action.

[1017] Input: Delivery schedule determination result

[1018] Output: Send notification

[1019] Specific operation: The server sends a notification to the customer such as "The delivery person will arrive soon" and a notification to the delivery person such as "Please take a break."

[1020] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1021] The present invention relates to a system for reducing the burden of assisting with excretion. In particular, it aims to efficiently manage the toilet needs of care recipients with physical disabilities and short-term memory impairments, thereby reducing the physical and mental burden on caregivers. To this end, in addition to camera monitoring means, excretion prediction means, control means, and display means, it uses an emotion engine that recognizes the user's emotions. The system of the present invention is configured as follows:

[1022] Program processing

[1023] Operation of camera monitoring means

[1024] A camera installed in the living space of the user (care recipient) captures the behavior and facial expressions of the care recipient in real time. This video data is analyzed in real time by the device, and if the care recipient appears anxious (e.g., looking around restlessly, showing anxious expressions), that state of anxiety is detected.

[1025] Operation of anxiety state detection and analysis means

[1026] The device analyzes the acquired video data in real time to detect anxiety, and the emotion engine analyzes the emotions of the care recipient based on the video data and transmits the emotion data to the control means.

[1027] Operation of the excretion prediction means

[1028] The device, worn by the care recipient, continuously measures biomarkers such as heart rate and skin temperature and transmits the data to a terminal, which then analyzes the data with a predictive algorithm to determine whether the care recipient needs to go to the toilet.

[1029] Operation of the control means

[1030] The control means provided by the server determines how to deal with the care recipient based on the anxiety state data, emotion data, and biodata sent from the terminal. Specifically, when the care recipient needs to go to the toilet, the server sends a notification to the caregiver's terminal informing them that toilet assistance is required.

[1031] Display operation

[1032] When there is no need to use the toilet, the device displays a friendly avatar on the screen for the care recipient. Based on the emotional data obtained from the emotion engine, the displayed avatar will talk to the care recipient about topics that interest them (e.g., recent family news, topics related to hobbies) to distract them.

[1033] Specific examples

[1034] When the care recipient becomes anxious on the sofa

[1035] 1. Operation of camera monitoring means

[1036] The user, who is being cared for, is watching TV on the sofa. The camera monitors the situation, and the device detects that the user is looking around anxiously.

[1037] 2. Operation of anxiety state detection and analysis method

[1038] The device detects the anxiety state of the care recipient and sends the data to the server. In addition, the emotion engine analyzes the care recipient's emotions (e.g., fear, relief, etc.) from the video data and sends the emotion data to the control means.

[1039] 3. Operation of the excretion prediction means

[1040] The excretion prediction device detects increases in the care recipient's heart rate and changes in skin temperature and sends this data to a terminal, which then uses this data to predict when the care recipient needs to excrete and sends the prediction to a server.

[1041] 4. Operation of the control means

[1042] The server analyzes the need for excretion and emotional data, and if it determines that the user needs to use the toilet, it sends a notification to the caregiver's smartphone saying that toilet assistance is required. On the other hand, if it determines that the user does not need to use the toilet, it sends a command to the device to display an avatar.

[1043] 5. Operation of the display means

[1044] The device displays an avatar on the screen, and the avatar speaks to the person being cared for in a friendly manner based on emotional data obtained from the emotion engine.

[1045] 6. Selecting and executing conversation content

[1046] The device selects appropriate conversation topics based on the care recipient's interests and emotions and sends instructions to the avatar, which then starts a conversation with the care recipient to ease their anxiety.

[1047] System Effects

[1048] This system can efficiently manage the toileting needs of care recipients and reduce the occurrence of unnecessary toileting assistance. It also alleviates the anxiety of care recipients, improving their quality of life and significantly reducing the physical and mental burden on caregivers. Utilizing an emotion engine enables more personalized responses, further improving the sense of security of care recipients.

[1049] The processing flow will be explained below.

[1050] Step 1: Obtain camera footage (device)

[1051] The device acquires real-time video data from a camera installed in the living space of the care recipient, including the care recipient's behavior and facial expressions.

[1052] Step 2: Anxiety state detection (device)

[1053] The device analyzes the captured video data in real time and detects anxiety based on the care recipient's behavior and facial expressions, such as restlessness and anxious facial expressions.

[1054] Step 3: Sentiment Analysis (Device)

[1055] The device uses an emotion engine to analyze the facial expressions of the care recipient to obtain emotional data, which includes the care recipient's emotional state (e.g., joy, sadness, fear, etc.).

[1056] Step 4: Biodata collection (terminal)

[1057] The excretion prediction device measures biomarkers such as the care recipient's heart rate and skin temperature in real time, and the terminal receives the measured biodata.

[1058] Step 5: Transfer data (device)

[1059] The terminal transmits the anxiety state data, emotion data, and biodata to the server, which prepares the data for analysis.

[1060] Step 6: Excretion prediction calculation (server)

[1061] The server analyzes the transmitted data and applies a toileting prediction algorithm to determine whether the care recipient needs to go to the toilet. The server generates a result of "needs toilet" or "no toilet."

[1062] Step 7: Restroom Need Notification (Server)

[1063] The server sends the results of the excretion prediction to the device. If the user "needs to use the toilet," the server sends notification data to the device saying "needs to use the toilet."

[1064] Step 8: Caregiver notification (device)

[1065] When the device receives a notification that the user needs to go to the toilet, it sends a notification to the caregiver's smartphone saying, "Toilet assistance required." The caregiver receives the notification and takes the necessary measures.

[1066] Step 9: Avatar Appearance and Response (Device)

[1067] If it is determined that the user does not need to use the toilet, the device will instruct an avatar to appear on the display, which will then provide friendly conversation and encourage the user to ease their anxiety.

[1068] Step 10: Select and execute conversation content (terminal)

[1069] The device selects appropriate conversation content based on the care recipient's emotional data and past data, and sends instructions to the avatar, which then starts a conversation with the care recipient and makes them forget about the toilet.

[1070] In this way, the system uses a camera monitoring means, an excretion prediction means, a control means, a display means, and an emotion engine to reduce anxiety in the care recipient and efficiently manage the need for excretion, thereby significantly reducing the burden on the caregiver.

[1071] Example 2

[1072] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1073] Conventional care systems have had difficulty quickly and accurately detecting the toilet needs and anxiety states of care recipients and providing appropriate responses. This has resulted in increased physical and mental burdens on caregivers and a decline in the quality of life of care recipients. Specifically, the accuracy of detecting anxiety states and predicting excretion needs has been low, making it difficult to respond in real time.

[1074] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1075] In this invention, the server includes a monitoring means for monitoring the behavior and facial expression of the care recipient and detecting an anxiety state, a prediction means for collecting physiological data of the care recipient and predicting the need for excretion, a control means for determining how to respond to the care recipient based on the anxiety state and the need for excretion, a display means for displaying a character that is friendly to the care recipient on the screen to distract the care recipient, and an emotion analysis means for recognizing the user's emotions. This makes it possible to monitor and analyze the condition of the care recipient in real time and take prompt and appropriate action.

[1076] The "monitoring method" is a means for photographing and recording the behavior and facial expressions of the care recipient in real time and analyzing the data.

[1077] The "prediction means" is a means for collecting physiological data such as the heart rate and skin temperature of the care recipient, and analyzing this data to determine the need for excretion.

[1078] The "control means" is a means for determining the condition of the care recipient based on data obtained from the monitoring means and prediction means, and for instructing appropriate responses.

[1079] The "display means" is a means for displaying a character that is friendly to the care recipient on the screen and serving to divert the care recipient's attention.

[1080] The "emotion analysis means" is a means for analyzing the behavior and facial expressions of the care recipient and recognizing their emotions.

[1081] The "notification means" is a means for notifying the caregiver when it is determined that the care recipient needs to go to the toilet based on the data acquired from the prediction means.

[1082] This invention is a system that aims to efficiently manage the toilet needs of a user (i.e., a care recipient) and reduce the physical and mental burden on the caregiver. This system includes a monitoring means, a prediction means, a control means, a display means, and an emotion analysis means.

[1083] Specific system configuration

[1084] 1. Monitoring measures

[1085] The monitoring method is a camera installed in the living space of the care recipient. This camera captures the care recipient's behavior and facial expressions in real time and collects the data. For example, a scene where a care recipient is watching TV on the sofa can be filmed, and the care recipient's behavior (e.g., looking around restlessly, anxious expressions, etc.) can be recorded.

[1086] 2. Prediction methods

[1087] The prediction method is to use an excretion prediction device attached to the care recipient. This device continuously measures physiological data such as heart rate and skin temperature and sends the data to a terminal. The terminal analyzes this data using an excretion prediction algorithm (machine learning model) to determine whether the care recipient needs to go to the toilet.

[1088] 3. Control Measures

[1089] The control means is a central server. The server determines the optimal way to respond to the care recipient based on anxiety state data, emotional data, and physiological data sent from the device. For example, if the server determines that the care recipient needs to go to the toilet, it sends a notification to the caregiver's device informing them that toilet assistance is required.

[1090] 4. Display means

[1091] The display is placed in a position where the care recipient can see it, and a friendly avatar is displayed on the display. The avatar distracts the care recipient by talking to them about topics that interest them based on data obtained from the emotion analysis method.

[1092] 5. Emotion analysis method

[1093] The emotion analysis means uses an emotion engine built into the device. The emotion engine analyzes the video data and recognizes the emotions of the care recipient. This emotion data is sent to the control means and used to determine the optimal response.

[1094] Specific examples

[1095] When the care recipient becomes anxious on the sofa

[1096] 1. When a user is sitting on a sofa, a camera monitors the user's behavior and detects the user looking around anxiously.

[1097] 2. The device acquires the video data, analyzes it in real time, detects the anxiety state, and sends the data to the server.

[1098] 3. Furthermore, the emotion engine analyzes the user's emotions (e.g., fear, relief, etc.) from the video data and sends that data to the control means.

[1099] 4. The excretion prediction device measures the user's heart rate and skin temperature and sends the data to the device, which then uses this data to determine the need for excretion.

[1100] 5. The server analyzes all of this data and decides on the appropriate course of action. If it determines that the person needs to go to the toilet, it sends a notification to the caregiver's device saying, "Toileting assistance required."

[1101] 6. If it is determined that the user does not need to use the toilet, the device will display an avatar on the screen and the avatar will speak to the user in a friendly manner.

[1102] Example prompts for generative AI models

[1103] "If your care recipient is anxiously looking around on the couch, instruct your avatar on how to assist them. Also, choose topics that will help the avatar to reassure the care recipient."

[1104] In this way, the invention provides a system that can efficiently manage the toilet needs of the care recipient and significantly reduce the burden on the caregiver by linking monitoring means, prediction means, control means, display means, and emotion analysis means.

[1105] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1106] Step 1:

[1107] A user sits on the sofa and starts watching TV. The camera starts operating and captures the user's actions and expressions in real time as a means of monitoring. The input is the video data captured by the camera, and the output is the data sent to the terminal.

[1108] Step 2:

[1109] The device receives video data sent from the camera. The input is video data, and the device analyzes this data in real time to detect patterns of the user's behavior and facial expressions. The output is the detected anxiety state.

[1110] Step 3:

[1111] The device sends the detected anxiety state data to the server. The input is the anxiety state data, and the output is the anxiety state data sent to the server.

[1112] Step 4:

[1113] At the same time, the device uses an emotion engine as an emotion analysis means to analyze the user's emotions from the video data. The input is the video data, which is analyzed by the emotion engine. The output is the analyzed emotion data.

[1114] Step 5:

[1115] The emotion data generated by the emotion engine is received by the terminal and sent to the server, which is the control means. The input is emotion data, and the output is the emotion data sent to the server.

[1116] Step 6:

[1117] The excretion prediction device worn by the user measures heart rate and skin temperature. The input is the user's physiological data, which is sent to a terminal via Bluetooth or other means. The output is physiological data.

[1118] Step 7:

[1119] The device uses the received physiological data to execute an excretion prediction algorithm. The input is heart rate and skin temperature data, and the device analyzes this data to predict the need for excretion. The output is the determination of the need for excretion.

[1120] Step 8:

[1121] The device determines whether the user needs to defecate and sends the result to the server. The input is the result of the determination, and the output is the result sent to the server.

[1122] Step 9:

[1123] The server analyzes the collected anxiety state data, emotional data, and toileting need data to determine how to respond to the care recipient. The input is the above data, and the appropriate response method is determined based on the analysis results. The output is a response method decision.

[1124] Step 10:

[1125] If the server determines that the user needs to go to the toilet, it sends a notification to the caregiver's device. The input is the decision on how to respond, and the output is the notification to the caregiver's device.

[1126] Step 11:

[1127] If it is determined that the user does not need to use the toilet, the terminal displays an avatar on the display. The input is instructions from the server, and the output is the avatar displayed on the display.

[1128] Step 12:

[1129] The avatar speaks to the user in a friendly manner based on the emotion data acquired by the emotion engine. The input is the emotion data, and the output is the conversation content of the avatar displayed to the user.

[1130] This system aims to improve the efficiency and quality of care by having the user, server, and terminal work together at each step to monitor and manage the toilet needs and psychological state of the care recipient in real time.

[1131] (Application example 2)

[1132] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1133] Conventional health management systems face the challenge of detecting workers' anxiety and stress levels in real time and providing appropriate responses quickly. Furthermore, health prediction systems based on workers' biodata are often not highly accurate, resulting in ineffective health management. Furthermore, health management information and reminders are not provided in a user-friendly format, preventing workers from maintaining their health and improving their productivity.

[1134] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1135] In this invention, the server includes a camera monitor that monitors the behavior and facial expressions of workers and detects their stress levels, a health prediction unit that collects biometric data from workers and predicts their health status, a control unit that determines how to respond to workers based on their anxiety and stress levels, and a display unit that provides workers with friendly health management information and reminders. This allows for highly accurate real-time monitoring and analysis of workers' health status and stress levels, enabling prompt and appropriate responses. Furthermore, providing workers with health management information and reminders through friendly avatars helps maintain their health and improve productivity.

[1136] "Care recipients" refers to people such as elderly people or people with disabilities who require support and care.

[1137] "Worker" refers to a worker engaged in work in a particular workplace.

[1138] "Camera monitoring means" refers to a device or system that has the function of monitoring the behavior and facial expressions of a subject using a camera.

[1139] The term "excretion prediction means" refers to a device or system that has the function of predicting the need for excretion based on the biodata of the care recipient.

[1140] "Health prediction means" refers to a device or system that has the function of predicting the health status of a worker based on the worker's biodata.

[1141] "Control means" refers to a device or system that has the ability to determine appropriate responses or actions based on collected data.

[1142] "Display means" refers to a device or system that has the function of visualizing and displaying information.

[1143] "Anxiety state" refers to a state in which the care recipient or worker feels mental anxiety or stress.

[1144] "Biodata" refers to data about living organisms, such as heart rate and skin temperature.

[1145] An "avatar" is a computer-generated character that appears as a user's alter ego or stand-in.

[1146] "Notification means" refers to a device or system that has the function of sending specific information or alerts to a target person.

[1147] "Analysis means" refers to a device or system that has the function of analyzing collected data and extracting useful information.

[1148] The present invention relates to a health management system for workers using factory robots. This system is composed of a camera monitor, a health prediction unit, a control unit, and a display unit.

[1149] Overall system configuration

[1150] Camera monitoring means

[1151] The server uses a camera monitor to monitor the behavior and facial expressions of workers. The camera can be, for example, a smartphone camera or a fixed camera installed in the factory. The video data acquired from this camera is analyzed in real time. For the video analysis, a common video processing library (e.g., OpenCV) and an AI model are used.

[1152] Health prediction tools

[1153] The server uses health prediction methods to predict the health status of workers. For example, wearable devices such as smart watches are used. These devices collect biodata such as heart rate and skin temperature and send it to the server in real time. Health management software (e.g., HealthKit) is used to analyze the data.

[1154] Control means

[1155] The server has a control means for determining the appropriate response based on the collected anxiety and stress levels and biodata. Based on the analysis results, it determines the appropriate response for the worker and sends a notification to the worker and their supervisor. The notification means uses the notification function of a typical smartphone.

[1156] Display means

[1157] The server has a display means for providing workers with friendly health management information and reminders. For example, it uses an avatar to display instructions for relaxation and deep breathing to workers. For this, an avatar generation application such as Unity is used.

[1158] Specific examples

[1159] Example 1: When a worker shows signs of instability while working on a production line

[1160] The camera monitoring means monitors the worker's behavior in real time and detects anxiety. The emotion engine analyzes the anxiety state and sends the data to the server. Meanwhile, the health prediction means collects heart rate and skin temperature from the smartwatch and predicts the worker's health state. The server performs sequential analysis based on this data and determines the necessary response. For example, it sends a notification to the worker to take a break, and an avatar teaches them how to relax.

[1161] Prompt Sentence Examples

[1162] This assistant is designed to monitor the health and stress levels of industrial workers using advanced AI and wearable technology. It can detect workers' anxiety through facial expression analysis and provide real-time health predictions based on biometric data from wearable devices. When necessary, the system alerts the worker or supervisor to suggest a break or convey other health messages through an interactive avatar.

[1163] Features:

[1164] Real-time facial expression analysis using OpenCV

[1165] Emotion recognition using Azure Cognitive Services

[1166] Health prediction using data from Apple Watch via HealthKit

[1167] Actionable alerts and notifications from cloud server

[1168] Friendly avatar interactions using Unity

[1169] Please design a machine learning model that can detect stress from facial expression data and integrate it with the existing health prediction and notification system.

[1170] effect

[1171] This system makes it possible to monitor and analyze workers' health and stress levels in real time with high accuracy, and to take appropriate measures quickly, which not only improves work efficiency and safety but also helps maintain worker health and reduce stress.

[1172] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1173] Step 1:

[1174] Acquisition of video data by camera monitor means

[1175] The server acquires video data of the workers' behavior and facial expressions in real time from cameras in the factory. The input is the video from the camera, and the output is the input video data. This video data is used to analyze anxiety and stress levels in subsequent processing steps. Specifically, the camera monitors the workers 24 hours a day and transmits the video data to the server at a constant frame rate.

[1176] Step 2:

[1177] Analysis of anxiety states using an emotion engine

[1178] The server inputs the acquired video data into the emotion engine for analysis. The input is the video data obtained in step 1, and the output is the analysis result indicating the worker's anxiety state. Specifically, the emotion engine (e.g., Azure Cognitive Services) is used to analyze the worker's facial expressions and movements from the video data, and evaluate the worker's anxiety state numerically.

[1179] Step 3:

[1180] Acquisition and analysis of biodata with health prediction tools

[1181] The server acquires biodata such as heart rate and skin temperature from the worker's wearable device. The input is the biodata from the wearable device, and the output is health status assessment data analyzed by a health prediction algorithm. Specifically, the server receives data from a smartwatch (e.g., Apple Watch) via Bluetooth and analyzes it using health management software such as HealthKit.

[1182] Step 4:

[1183] Deciding how to respond using control measures

[1184] The server determines how to respond based on the anxiety state data from the emotion engine and the health state evaluation data from the health prediction means. The input is the anxiety state data and the health state evaluation data, and the output is the decision on how to respond (e.g., instructing a break, notifying a supervisor). Specifically, a program is run that integrates the data and generates a notification instructing a break if the anxiety state is high or an abnormality in the health state is detected.

[1185] Step 5:

[1186] Sending notifications via notification methods

[1187] The server sends notifications to workers and their supervisors based on the response method determined by the control means. The input is response method data, and the output is a notification message to the recipient. Specifically, the program uses the smartphone's push notification function to send break instructions and health status alerts.

[1188] Step 6:

[1189] Providing information through display means

[1190] The server displays health management information and relaxation method reminders on the worker's smartphone or dedicated device. The input is response method data, and the output is a friendly message using an avatar. Specifically, the avatar is displayed using Unity, and a program runs that suggests ways to relax and take a break to anxious workers.

[1191] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1192] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1193] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1194] [Fourth embodiment]

[1195] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1196] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1197] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1198] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1199] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1200] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1201] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1202] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1203] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1204] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1205] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1206] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1207] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1208] This invention is a system for efficiently managing excretion assistance in the living space of a care recipient and reducing the burden on caregivers. The system includes a "camera monitoring means" that monitors the behavior and facial expression of the care recipient, an "excretion prediction means" that collects biodata of the care recipient and predicts the need for excretion, a "control means" that determines how to respond to the care recipient based on the care recipient's anxiety state and the need for excretion, and a "display means" that displays a friendly avatar to the care recipient to distract the care recipient from excretion.

[1209] The program processing in this system is as follows.

[1210] Program processing

[1211] Operation of camera monitoring means

[1212] A camera installed in the living space of the user (care recipient) captures the behavior and facial expressions of the care recipient in real time. This video data is analyzed in real time by the device, and if the care recipient appears anxious (e.g., looking around restlessly, showing anxious expressions), that state of anxiety is detected.

[1213] Operation of the excretion prediction means

[1214] The excretion prediction device attached to the care recipient continuously measures biodata such as heart rate and skin temperature and transmits the data to a terminal, which then analyzes the data with an excretion prediction algorithm to determine whether the care recipient needs to go to the toilet.

[1215] Operation of the control means

[1216] The control means provided by the server determines how to respond to the care recipient based on the anxiety state data and excretion prediction data sent from the terminal. Specifically, when the care recipient needs to go to the toilet, the server sends a notification to the caregiver's terminal informing them that toilet assistance is required.

[1217] Display operation

[1218] When there is no need to use the toilet, the device displays a friendly avatar on the screen, which diverts the care recipient's attention by talking about topics that interest them (e.g., recent family news, hobbies, etc.).

[1219] Specific examples

[1220] When the care recipient becomes anxious on the sofa

[1221] 1. Operation of camera monitoring means

[1222] The user, who is being cared for, is watching TV on the sofa. The camera monitors the situation, and the device detects that the user is looking around anxiously.

[1223] 2. Operation of the excretion prediction means

[1224] The excretion prediction device detects increases in the care recipient's heart rate and changes in skin temperature and sends this data to a terminal, which then uses this data to predict when the care recipient needs to excrete and sends the prediction to a server.

[1225] 3. Operation of the control means

[1226] The server analyzes the need for excretion, and if it determines that the user needs a toilet, it sends a notification to the caregiver's smartphone saying that toilet assistance is required. On the other hand, if it determines that the user does not need a toilet, it sends a command to the device to display an avatar.

[1227] 4. Display operation

[1228] The device displays an avatar on the screen and speaks to the care recipient in a friendly manner, for example, "Would you like to see some photos you took recently with your family?", helping to ease their anxiety.

[1229] System Effects

[1230] This system can efficiently manage the toileting needs of care recipients, reduce the occurrence of unnecessary toileting assistance, and alleviate the anxiety of care recipients, improving their quality of life and significantly reducing the physical and mental burden on caregivers.

[1231] The processing flow will be explained below.

[1232] Step 1: Obtain camera footage (device)

[1233] The device acquires real-time video data from a camera installed in the living space of the care recipient, including the care recipient's behavior and facial expressions.

[1234] Step 2: Anxiety state detection (device)

[1235] The device analyzes the captured video data in real time and detects anxiety based on the care recipient's behavior and facial expressions, such as restlessness and anxious facial expressions.

[1236] Step 3: Biodata collection (terminal)

[1237] The excretion prediction device measures biomarkers such as the care recipient's heart rate and skin temperature in real time, and the terminal receives the measured biodata.

[1238] Step 4: Transfer data (device)

[1239] The device transmits anxiety state data and biodata to the server, which prepares the data for analysis.

[1240] Step 5: Excretion prediction calculation (server)

[1241] The server analyzes the transmitted data and applies a toileting prediction algorithm to determine whether the care recipient needs to go to the toilet. The server generates a result of "needs toilet" or "no toilet."

[1242] Step 6: Restroom Need Notification (Server)

[1243] The server sends the results of the excretion prediction to the device. If the user "needs to use the toilet," the server sends notification data to the device saying "needs to use the toilet."

[1244] Step 7: Caregiver notification (device)

[1245] When the device receives a notification that the user needs to go to the toilet, it sends a notification to the caregiver's smartphone saying, "Toilet assistance required." The caregiver receives the notification and takes the necessary measures.

[1246] Step 8: Avatar Appearance and Response (Device)

[1247] If it is determined that the user does not need to use the toilet, the device will instruct an avatar to appear on the display, which will then provide friendly conversation and encourage the user to ease their anxiety.

[1248] Step 9: Select and execute conversation content (terminal)

[1249] The device selects appropriate conversation topics based on the care recipient's interests and past data, and sends instructions to the avatar, which then starts a conversation with the care recipient and makes them forget about the toilet.

[1250] In this way, the system uses a camera monitoring means, an excretion prediction means, a control means, and a display means to reduce the anxiety of the care recipient and efficiently manage the need for excretion, thereby significantly reducing the burden on the caregiver.

[1251] Example 1

[1252] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1253] In the daily lives of elderly people and physically disabled care recipients, they often lose track of when to go to the toilet, which can lead to increased anxiety and stress. It has also been pointed out that frequent toilet assistance is required of caregivers, placing a heavy physical and mental burden on them. There is a need for a system that can solve these problems, improve the quality of life of care recipients, and reduce the burden on caregivers.

[1254] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1255] In this invention, the server includes a camera means for monitoring the behavior and facial expression of the care recipient and detecting their state of anxiety, a prediction means for collecting biometric data of the care recipient and predicting their need for excretion, and a control means for determining how to respond to the care recipient based on their state of anxiety and their need for excretion. This makes it possible to efficiently manage the toilet needs of the care recipient, reduce the need for unnecessary toilet assistance, and alleviate their anxiety.

[1256] "Care recipients" refers to elderly people who require care or those who are physically unable to live independently.

[1257] "Camera means" refers to a device that captures the actions and expressions of the care recipient in real time and acquires that data.

[1258] "Biometric data" refers to data that indicates the physical condition of the care recipient, and includes information such as heart rate and skin temperature.

[1259] "Prediction means" refers to a device or program that analyzes biological data and predicts the need for excretion of the care recipient.

[1260] The "analysis means" refers to a device or program that analyzes the video data obtained from the camera means in real time and detects an anxiety state.

[1261] "Controlling means" refers to a device or program that determines how to respond to a care recipient based on their anxiety state and need to eliminate.

[1262] "Notification means" refers to a device or program that notifies a caregiver when it is determined that an individual needs to toilet.

[1263] "Display device" refers to a device for providing visual or audio information to a care recipient.

[1264] "Presentation means" refers to a device or program that displays an avatar or the like to divert the care recipient's attention from toileting.

[1265] "Anxious state" refers to a mentally unsettled state of the care recipient, such as frequently looking around or showing an anxious expression.

[1266] The present invention is a system for efficiently managing excretion assistance in the living space of a care recipient and reducing the burden on caregivers. The system includes a "camera means" that monitors the behavior and facial expression of the care recipient, a "prediction means" that collects biometric data of the care recipient and predicts the need for excretion, a "control means" that determines how to respond to the care recipient based on the care recipient's anxiety state and the need for excretion, and a "presentation means" that uses a display device that is familiar to the care recipient to divert the care recipient's attention from excretion.

[1267] Camera Means Details

[1268] A camera installed in the living space of the care recipient captures the care recipient's behavior and facial expressions in real time. This video data is analyzed by a device (e.g., Raspberry Pi or NVIDIA Jetson Nano). The device uses an open-source computer vision library (e.g., OpenCV) to perform facial recognition and facial expression analysis, and detects anxiety states (e.g., frequently looking around, showing anxious expressions, etc.).

[1269] Details of the forecasting method

[1270] A toilet prediction device (e.g., Fitbit or Apple Watch) worn by the care recipient continuously measures biometric data such as heart rate and skin temperature and transmits the data to a terminal. The terminal analyzes this data using a toilet prediction algorithm (e.g., a machine learning model implemented in Python) to determine when the care recipient needs to go to the toilet.

[1271] Details of control measures

[1272] The anxiety state data and excretion prediction data sent from the device are received by a server (e.g., AWS EC2, Google Cloud VM). The server integrates this data, analyzes the care recipient's situation, and if the user needs to use the toilet, the server sends a notification to the caregiver's smartphone (e.g., iPhone or Android device) saying, "Toilet assistance required." Notifications are sent using services such as AWS SNS.

[1273] Presentation details

[1274] When no toileting is required, the device displays an avatar on a display device (e.g., tablet, smart display) that is familiar to the care recipient. The avatar speaks to the care recipient about topics of interest (e.g., family updates, topics related to hobbies) to divert the care recipient's attention from toileting.

[1275] Specific examples

[1276] When the care recipient becomes anxious on the sofa

[1277] 1. Camera Means Operation

[1278] The user, a care recipient, is watching TV on the sofa. A camera (e.g., Logitech C920) monitors the user's behavior, and the device (NVIDIA Jetson Nano) detects that the user is looking around anxiously.

[1279] 2. Operation of the prediction means

[1280] The toileting prediction device (e.g., Fitbit) detects an increase in the care recipient's heart rate and changes in skin temperature and sends the data to a terminal. The terminal uses this data to predict the need for toileting using a machine learning model and sends the results to a server (AWS EC2).

[1281] 3. Operation of the control means

[1282] The server analyzes the need for excretion and sends a notification to the caregiver's smartphone if it determines that the person needs to go to the toilet. On the other hand, if it determines that the person does not need to go to the toilet, it sends a command to the device to display an avatar.

[1283] 4. Presentation Method Operation

[1284] The device displays an avatar on a tablet and speaks to the care recipient in a friendly manner, for example, "Would you like to see some photos you recently took with your family?", helping to ease their anxiety.

[1285] Prompt Sentence Examples

[1286] "Tell me how to raise the alarm when a care recipient is exhibiting anxious behavior on the couch."

[1287] This system can efficiently manage the toileting needs of care recipients, reduce the occurrence of unnecessary toileting assistance, and alleviate the anxiety of care recipients, improving their quality of life and significantly reducing the physical and mental burden on caregivers.

[1288] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1289] Program processing steps

[1290] Camera measures steps

[1291] Step 1:

[1292] To monitor the behavior and facial expressions of the care recipient, a camera acquires video data of the care recipient. Specifically, a camera (e.g., Logitech C920) captures video at 30 frames per second. The input is real-time video of the care recipient, and the output is the video data acquired by the camera.

[1293] Step 2:

[1294] The device receives and analyzes the video data from the camera. The device (e.g., NVIDIA Jetson Nano) analyzes the video data using OpenCV and performs facial recognition and facial expression analysis. The input is the video data acquired in step 1, and the output is the facial expression data resulting from the analysis.

[1295] Step 3:

[1296] The device detects the anxiety state based on the analysis results. Specifically, it determines whether the user frequently looks around or shows an anxious expression. The input is the facial expression data obtained in step 2, and the output is the anxiety state determination result.

[1297] Prediction steps

[1298] Step 4:

[1299] The excretion prediction device worn by the user measures biological data (heart rate and skin temperature). Specifically, the Fitbit measures heart rate every minute and skin temperature every five minutes. The input is the care recipient's real-time biological data, and the output is the measurement data.

[1300] Step 5:

[1301] The measured biometric data is sent to the device. The Fitbit sends the data to the NVIDIA Jetson Nano via Bluetooth. The input is the measurement data obtained in step 4, and the output is the biometric data sent to the device.

[1302] Step 6:

[1303] The device analyzes the collected biometric data and performs excretion prediction. The device uses a machine learning model implemented in Python to predict the need for excretion from the biometric data. The input is the biometric data received in step 5, and the output is a prediction result indicating the need for excretion.

[1304] Control Measure Steps

[1305] Step 7:

[1306] The server receives the anxiety state data and excretion prediction data sent from the device. The server (e.g., AWS EC2) integrates and analyzes this data. The input is the anxiety state data from step 3 and the excretion prediction data from step 6, and the output is the integrated analysis results.

[1307] Step 8:

[1308] The server decides how to respond based on the integrated data. If the user needs to use the toilet, the server determines that "toileting assistance is required," and if not, issues an instruction to display an avatar. The input is the analysis result from step 7, and the output is the decision on how to respond.

[1309] Step 9:

[1310] The server sends a notification to the caregiver's smartphone. Specifically, it uses AWS SNS to send a message such as "Toileting assistance required." The input is the result of the decision on how to respond in step 8, and the output is the notification sent to the caregiver's smartphone.

[1311] Presentation steps

[1312] Step 10:

[1313] The terminal displays a friendly avatar for the care recipient. The terminal displays the avatar on a display device (e.g., a tablet or smart display). The input is an instruction to display the avatar from the server, and the output is the avatar displayed on the display device.

[1314] Step 11:

[1315] The avatar speaks to the care recipient on topics that match their interests. For example, the avatar might say, "Would you like to see some photos you recently took with your family?" to ease their anxiety. The input is a registered conversation script, and the output is a voice message to the care recipient.

[1316] (Application example 1)

[1317] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1318] Conventional nursing care systems lack the functionality to adequately manage anxiety and stress, let alone assist with toileting. Furthermore, in businesses like food delivery, there was a lack of a system for monitoring the health and work efficiency of delivery personnel in real time, making effective labor management and safety measures difficult. Given these circumstances, there is a need for a system that can appropriately manage and monitor the condition of both care recipients and delivery personnel, and respond efficiently based on that information.

[1319] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1320] In this invention, the server includes a monitoring means for monitoring the behavior and facial expression of the care recipient and detecting an anxiety state, a prediction means for collecting biological information of the care recipient and predicting the need for excretion, a control means for determining how to respond to the care recipient based on the anxiety state and the need for excretion, a display means for displaying information that is friendly to the care recipient and diverting the care recipient's attention from excretion, and a means for monitoring the health state of the delivery person and collecting data to improve delivery efficiency. This not only enables efficient management of the care recipient's anxiety and excretion, but also enables safe and efficient labor management by monitoring the health state of the delivery person in real time.

[1321] A "monitoring means" is a tool used to monitor a subject's behavior, facial expressions, and biometric information in real time and detect specific conditions.

[1322] A "prediction tool" is a tool that analyzes collected biological data and predicts specific needs or conditions (e.g., the need to defecate or stress conditions).

[1323] "Control means" are tools that determine the optimal response to the target person based on information obtained from the monitoring means and prediction means, and provide instructions and notifications.

[1324] A "display means" is an interface that presents information to a subject and encourages or diverts attention to a particular action.

[1325] "Means for collecting data" refers to tools for continuously collecting biometric and behavioral data of subjects and transmitting them to the system.

[1326] "Notification means" is a tool for notifying the target person or administrator in real time and encouraging them to take the necessary action.

[1327] A specific system for implementing the present invention mainly includes the following hardware and software components:

[1328] 1. System Configuration

[1329] Monitoring method: Camera that captures user behavior and facial expressions in real time.

[1330] Examples: high resolution cameras, infrared cameras

[1331] Prediction means: A biosensor that acquires the user's biological information.

[1332] Examples: Heart rate sensor, skin temperature sensor

[1333] Control means: A control program executed on the server.

[1334] Example: Linux server, AWS (Amazon Web Services)

[1335] Display means: A display that displays information and an avatar display program.

[1336] Examples: tablets, smartphones, smart glasses

[1337] 2. Program Processing

[1338] The program performs complex data analysis and processing using data obtained from each means. The main processing contents are shown below.

[1339] monitoring means

[1340] The camera captures the user's behavior and facial expressions in real time and transmits the video data to a server, which then analyzes the video data using a generative AI model to detect anxiety states.

[1341] Hardware: High-resolution camera

[1342] Software: Image analysis software (e.g. OpenCV)

[1343] Prediction methods

[1344] Biosensors measure the user's heart rate and skin temperature and transmit this vital information to a server, which uses predictive algorithms to analyze the need for urination.

[1345] Hardware: Heart rate sensor, skin temperature sensor

[1346] Software: Data analysis software (e.g., Python, NumPy)

[1347] Control means

[1348] The server determines an appropriate response to the user based on the data obtained from the monitoring means and the prediction means, and notifies the caregiver and the user by the notification means.

[1349] Hardware: Server

[1350] Software: Communication software (e.g. HTTP API)

[1351] Display means

[1352] The server generates an avatar based on the user's state and displays it on the screen. The avatar provides the user with friendly information and topics to divert their attention.

[1353] Hardware: Smart displays, tablets

[1354] Software: Avatar generation software (e.g. Unity, Blender)

[1355] 3. Specific Examples

[1356] Caregiving scenario

[1357] If the user is anxious on the sofa, the camera monitors the user's behavior and sends the video data to the server in real time. The server uses a generative AI model to detect the state of anxiety and also analyzes the need for toileting based on data from the heart rate sensor and skin temperature sensor. If it is determined that toileting is necessary, a notification is sent to the caregiver.

[1358] Food Delivery Scenario

[1359] A camera installed on the doorbell monitors the customer's situation and detects if the customer is absent or unable to answer. Along with the camera, biosensors on the delivery person measure the heart rate and body temperature and send the data to a server. The server analyzes this data, monitors the delivery person's health, and plans a safe and efficient delivery.

[1360] Prompt Sentence Examples

[1361] "Please submit your idea for developing an application for a food delivery system that monitors the delivery person's biodata (heart rate, body temperature) and the customer's situation (doorbell camera footage) and suggests the optimal delivery schedule."

[1362] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1363] Step 1:

[1364] The camera captures the user's actions and facial expressions in real time, capturing video data.

[1365] Input: Real-time video data

[1366] Output: Captured video data

[1367] Specific operation: The camera captures the user's actions and facial expressions, and the data is immediately sent to the server.

[1368] Step 2:

[1369] The server receives the video data and analyzes it in real time using a generative AI model to detect anxiety states.

[1370] Input: Captured video data

[1371] Output: Anxiety state judgment result (e.g., anxious, normal)

[1372] Specific behavior: The generative AI model analyzes video data and detects anxiety from the user's facial expressions and behaviors. For example, it determines whether the user frequently looks around or shows anxious facial expressions.

[1373] Step 3:

[1374] The biosensor measures the user's heart rate and skin temperature and transmits the biometric data to a server.

[1375] Input: Biometric information such as heart rate and skin temperature

[1376] Output: Acquired biometric data

[1377] Specific operation: A biosensor worn by the user continuously measures data such as heart rate and skin temperature and sends it to a server.

[1378] Step 4:

[1379] The server receives the biometric data and predicts the need for toileting using an excretion prediction algorithm.

[1380] Input: Acquired biometric data

[1381] Output: Judgment result of necessity of excretion (e.g., necessary, unnecessary)

[1382] How it works: The server analyzes biometric data and predicts the need to urinate based on certain patterns (e.g., increased heart rate or changes in skin temperature).

[1383] Step 5:

[1384] The server determines how to respond to the user based on the results of the judgment of the anxiety state and the need for excretion.

[1385] Input: Anxiety state judgment result, judgment result of need for excretion

[1386] Output: Decide how to respond (e.g., notify caregiver, display avatar)

[1387] Specific operation: The server comprehensively analyzes the anxiety state and toileting need data and decides whether to send a notification to the caregiver or display an avatar to the user.

[1388] Step 6:

[1389] The server sends a notification to the caregiver or displays an avatar on the display based on the response decision.

[1390] Input: Decision result of response method

[1391] Output: Sending a notification or displaying an avatar

[1392] Specific behavior: When the server sends a notification to the caregiver's smartphone, the caregiver receives a message such as "Toileting assistance required." Alternatively, an avatar appears on the user's display and speaks a voice message such as "Would you like to see some recent photos taken with your family?"

[1393] Step 7:

[1394] The doorbell camera monitors the customer's situation and detects if they are absent or unable to answer the door, sending the information to the server.

[1395] Input: Doorbell camera video data

[1396] Output: Customer status determination result (e.g., at home, away)

[1397] Specific operation: The doorbell camera monitors the customer's status, and if the customer is not present, the data is sent to the server and the absence of the customer is detected.

[1398] Step 8:

[1399] The delivery person's biosensor measures their heart rate and body temperature and sends the data to a server.

[1400] Input: Delivery person's heart rate, body temperature

[1401] Output: Retrieved biodata

[1402] Specific operation: A biosensor attached to the delivery person continuously measures heart rate and body temperature data and transmits it to a server.

[1403] Step 9:

[1404] The server analyzes the delivery person's biodata and creates an optimal delivery plan to improve delivery efficiency.

[1405] Input: Delivery person's biodata

[1406] Output: Optimal delivery schedule

[1407] Specific operation: The server analyzes the delivery person's heart rate and body temperature data, and if the delivery person is tired or stressed, it suggests taking a break or recalculates the optimal delivery route.

[1408] Step 10:

[1409] The server sends a notification to the customer and delivery person, prompting them to take appropriate action.

[1410] Input: Delivery schedule determination result

[1411] Output: Send notification

[1412] Specific operation: The server sends a notification to the customer such as "The delivery person will arrive soon" and a notification to the delivery person such as "Please take a break."

[1413] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1414] The present invention relates to a system for reducing the burden of assisting with excretion. In particular, it aims to efficiently manage the toilet needs of care recipients with physical disabilities and short-term memory impairments, thereby reducing the physical and mental burden on caregivers. To this end, in addition to camera monitoring means, excretion prediction means, control means, and display means, it uses an emotion engine that recognizes the user's emotions. The system of the present invention is configured as follows:

[1415] Program processing

[1416] Operation of camera monitoring means

[1417] A camera installed in the living space of the user (care recipient) captures the behavior and facial expressions of the care recipient in real time. This video data is analyzed in real time by the device, and if the care recipient appears anxious (e.g., looking around restlessly, showing anxious expressions), that state of anxiety is detected.

[1418] Operation of anxiety state detection and analysis means

[1419] The device analyzes the acquired video data in real time to detect anxiety, and the emotion engine analyzes the emotions of the care recipient based on the video data and transmits the emotion data to the control means.

[1420] Operation of the excretion prediction means

[1421] The device, worn by the care recipient, continuously measures biomarkers such as heart rate and skin temperature and transmits the data to a terminal, which then analyzes the data with a predictive algorithm to determine whether the care recipient needs to go to the toilet.

[1422] Operation of the control means

[1423] The control means provided by the server determines how to deal with the care recipient based on the anxiety state data, emotion data, and biodata sent from the terminal. Specifically, when the care recipient needs to go to the toilet, the server sends a notification to the caregiver's terminal informing them that toilet assistance is required.

[1424] Display operation

[1425] When there is no need to use the toilet, the device displays a friendly avatar on the screen for the care recipient. Based on the emotional data obtained from the emotion engine, the displayed avatar will talk to the care recipient about topics that interest them (e.g., recent family news, topics related to hobbies) to distract them.

[1426] Specific examples

[1427] When the care recipient becomes anxious on the sofa

[1428] 1. Operation of camera monitoring means

[1429] The user, who is being cared for, is watching TV on the sofa. The camera monitors the situation, and the device detects that the user is looking around anxiously.

[1430] 2. Operation of anxiety state detection and analysis method

[1431] The device detects the anxiety state of the care recipient and sends the data to the server. In addition, the emotion engine analyzes the care recipient's emotions (e.g., fear, relief, etc.) from the video data and sends the emotion data to the control means.

[1432] 3. Operation of the excretion prediction means

[1433] The excretion prediction device detects increases in the care recipient's heart rate and changes in skin temperature and sends this data to a terminal, which then uses this data to predict when the care recipient needs to excrete and sends the prediction to a server.

[1434] 4. Operation of the control means

[1435] The server analyzes the need for excretion and emotional data, and if it determines that the user needs to use the toilet, it sends a notification to the caregiver's smartphone saying that toilet assistance is required. On the other hand, if it determines that the user does not need to use the toilet, it sends a command to the device to display an avatar.

[1436] 5. Operation of the display means

[1437] The device displays an avatar on the screen, and the avatar speaks to the person being cared for in a friendly manner based on emotional data obtained from the emotion engine.

[1438] 6. Selecting and executing conversation content

[1439] The device selects appropriate conversation topics based on the care recipient's interests and emotions and sends instructions to the avatar, which then starts a conversation with the care recipient to ease their anxiety.

[1440] System Effects

[1441] This system can efficiently manage the toileting needs of care recipients and reduce the occurrence of unnecessary toileting assistance. It also alleviates the anxiety of care recipients, improving their quality of life and significantly reducing the physical and mental burden on caregivers. Utilizing an emotion engine enables more personalized responses, further improving the sense of security of care recipients.

[1442] The processing flow will be explained below.

[1443] Step 1: Obtain camera footage (device)

[1444] The device acquires real-time video data from a camera installed in the living space of the care recipient, including the care recipient's behavior and facial expressions.

[1445] Step 2: Anxiety state detection (device)

[1446] The device analyzes the captured video data in real time and detects anxiety based on the care recipient's behavior and facial expressions, such as restlessness and anxious facial expressions.

[1447] Step 3: Sentiment Analysis (Device)

[1448] The device uses an emotion engine to analyze the facial expressions of the care recipient to obtain emotional data, which includes the care recipient's emotional state (e.g., joy, sadness, fear, etc.).

[1449] Step 4: Biodata collection (terminal)

[1450] The excretion prediction device measures biomarkers such as the care recipient's heart rate and skin temperature in real time, and the terminal receives the measured biodata.

[1451] Step 5: Transfer data (device)

[1452] The terminal transmits the anxiety state data, emotion data, and biodata to the server, which prepares the data for analysis.

[1453] Step 6: Excretion prediction calculation (server)

[1454] The server analyzes the transmitted data and applies a toileting prediction algorithm to determine whether the care recipient needs to go to the toilet. The server generates a result of "needs toilet" or "no toilet."

[1455] Step 7: Restroom Need Notification (Server)

[1456] The server sends the results of the excretion prediction to the device. If the user "needs to use the toilet," the server sends notification data to the device saying "needs to use the toilet."

[1457] Step 8: Caregiver notification (device)

[1458] When the device receives a notification that the user needs to go to the toilet, it sends a notification to the caregiver's smartphone saying, "Toilet assistance required." The caregiver receives the notification and takes the necessary measures.

[1459] Step 9: Avatar Appearance and Response (Device)

[1460] If it is determined that the user does not need to use the toilet, the device will instruct an avatar to appear on the display, which will then provide friendly conversation and encourage the user to ease their anxiety.

[1461] Step 10: Select and execute conversation content (terminal)

[1462] The device selects appropriate conversation content based on the care recipient's emotional data and past data, and sends instructions to the avatar, which then starts a conversation with the care recipient and makes them forget about the toilet.

[1463] In this way, the system uses a camera monitoring means, an excretion prediction means, a control means, a display means, and an emotion engine to reduce anxiety in the care recipient and efficiently manage the need for excretion, thereby significantly reducing the burden on the caregiver.

[1464] Example 2

[1465] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1466] Conventional care systems have had difficulty quickly and accurately detecting the toilet needs and anxiety states of care recipients and providing appropriate responses. This has resulted in increased physical and mental burdens on caregivers and a decline in the quality of life of care recipients. Specifically, the accuracy of detecting anxiety states and predicting excretion needs has been low, making it difficult to respond in real time.

[1467] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1468] In this invention, the server includes a monitoring means for monitoring the behavior and facial expression of the care recipient and detecting an anxiety state, a prediction means for collecting physiological data of the care recipient and predicting the need for excretion, a control means for determining how to respond to the care recipient based on the anxiety state and the need for excretion, a display means for displaying a character that is friendly to the care recipient on the screen to distract the care recipient, and an emotion analysis means for recognizing the user's emotions. This makes it possible to monitor and analyze the condition of the care recipient in real time and take prompt and appropriate action.

[1469] The "monitoring method" is a means for photographing and recording the behavior and facial expressions of the care recipient in real time and analyzing the data.

[1470] The "prediction means" is a means for collecting physiological data such as the heart rate and skin temperature of the care recipient, and analyzing this data to determine the need for excretion.

[1471] The "control means" is a means for determining the condition of the care recipient based on data obtained from the monitoring means and prediction means, and for instructing appropriate responses.

[1472] The "display means" is a means for displaying a character that is friendly to the care recipient on the screen and serving to divert the care recipient's attention.

[1473] The "emotion analysis means" is a means for analyzing the behavior and facial expressions of the care recipient and recognizing their emotions.

[1474] The "notification means" is a means for notifying the caregiver when it is determined that the care recipient needs to go to the toilet based on the data acquired from the prediction means.

[1475] This invention is a system that aims to efficiently manage the toilet needs of a user (i.e., a care recipient) and reduce the physical and mental burden on the caregiver. This system includes a monitoring means, a prediction means, a control means, a display means, and an emotion analysis means.

[1476] Specific system configuration

[1477] 1. Monitoring measures

[1478] The monitoring method is a camera installed in the living space of the care recipient. This camera captures the care recipient's behavior and facial expressions in real time and collects the data. For example, a scene where a care recipient is watching TV on the sofa can be filmed, and the care recipient's behavior (e.g., looking around restlessly, anxious expressions, etc.) can be recorded.

[1479] 2. Prediction methods

[1480] The prediction method is to use an excretion prediction device attached to the care recipient. This device continuously measures physiological data such as heart rate and skin temperature and sends the data to a terminal. The terminal analyzes this data using an excretion prediction algorithm (machine learning model) to determine whether the care recipient needs to go to the toilet.

[1481] 3. Control Measures

[1482] The control means is a central server. The server determines the optimal way to respond to the care recipient based on anxiety state data, emotional data, and physiological data sent from the device. For example, if the server determines that the care recipient needs to go to the toilet, it sends a notification to the caregiver's device informing them that toilet assistance is required.

[1483] 4. Display means

[1484] The display is placed in a position where the care recipient can see it, and a friendly avatar is displayed on the display. The avatar distracts the care recipient by talking to them about topics that interest them based on data obtained from the emotion analysis method.

[1485] 5. Emotion analysis method

[1486] The emotion analysis means uses an emotion engine built into the device. The emotion engine analyzes the video data and recognizes the emotions of the care recipient. This emotion data is sent to the control means and used to determine the optimal response.

[1487] Specific examples

[1488] When the care recipient becomes anxious on the sofa

[1489] 1. When a user is sitting on a sofa, a camera monitors the user's behavior and detects the user looking around anxiously.

[1490] 2. The device acquires the video data, analyzes it in real time, detects the anxiety state, and sends the data to the server.

[1491] 3. Furthermore, the emotion engine analyzes the user's emotions (e.g., fear, relief, etc.) from the video data and sends that data to the control means.

[1492] 4. The excretion prediction device measures the user's heart rate and skin temperature and sends the data to the device, which then uses this data to determine the need for excretion.

[1493] 5. The server analyzes all of this data and decides on the appropriate course of action. If it determines that the person needs to go to the toilet, it sends a notification to the caregiver's device saying, "Toileting assistance required."

[1494] 6. If it is determined that the user does not need to use the toilet, the device will display an avatar on the screen and the avatar will speak to the user in a friendly manner.

[1495] Example prompts for generative AI models

[1496] "If your care recipient is anxiously looking around on the couch, instruct your avatar on how to assist them. Also, choose topics that will help the avatar to reassure the care recipient."

[1497] In this way, the invention provides a system that can efficiently manage the toilet needs of the care recipient and significantly reduce the burden on the caregiver by linking monitoring means, prediction means, control means, display means, and emotion analysis means.

[1498] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1499] Step 1:

[1500] A user sits on the sofa and starts watching TV. The camera starts operating and captures the user's actions and expressions in real time as a means of monitoring. The input is the video data captured by the camera, and the output is the data sent to the terminal.

[1501] Step 2:

[1502] The device receives video data sent from the camera. The input is video data, and the device analyzes this data in real time to detect patterns of the user's behavior and facial expressions. The output is the detected anxiety state.

[1503] Step 3:

[1504] The device sends the detected anxiety state data to the server. The input is the anxiety state data, and the output is the anxiety state data sent to the server.

[1505] Step 4:

[1506] At the same time, the device uses an emotion engine as an emotion analysis means to analyze the user's emotions from the video data. The input is the video data, which is analyzed by the emotion engine. The output is the analyzed emotion data.

[1507] Step 5:

[1508] The emotion data generated by the emotion engine is received by the terminal and sent to the server, which is the control means. The input is emotion data, and the output is the emotion data sent to the server.

[1509] Step 6:

[1510] The excretion prediction device worn by the user measures heart rate and skin temperature. The input is the user's physiological data, which is sent to a terminal via Bluetooth or other means. The output is physiological data.

[1511] Step 7:

[1512] The device uses the received physiological data to execute an excretion prediction algorithm. The input is heart rate and skin temperature data, and the device analyzes this data to predict the need for excretion. The output is the determination of the need for excretion.

[1513] Step 8:

[1514] The device determines whether the user needs to defecate and sends the result to the server. The input is the result of the determination, and the output is the result sent to the server.

[1515] Step 9:

[1516] The server analyzes the collected anxiety state data, emotional data, and toileting need data to determine how to respond to the care recipient. The input is the above data, and the appropriate response method is determined based on the analysis results. The output is a response method decision.

[1517] Step 10:

[1518] If the server determines that the user needs to go to the toilet, it sends a notification to the caregiver's device. The input is the decision on how to respond, and the output is the notification to the caregiver's device.

[1519] Step 11:

[1520] If it is determined that the user does not need to use the toilet, the terminal displays an avatar on the display. The input is instructions from the server, and the output is the avatar displayed on the display.

[1521] Step 12:

[1522] The avatar speaks to the user in a friendly manner based on the emotion data acquired by the emotion engine. The input is the emotion data, and the output is the conversation content of the avatar displayed to the user.

[1523] This system aims to improve the efficiency and quality of care by having the user, server, and terminal work together at each step to monitor and manage the toilet needs and psychological state of the care recipient in real time.

[1524] (Application example 2)

[1525] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1526] Conventional health management systems face the challenge of detecting workers' anxiety and stress levels in real time and providing appropriate responses quickly. Furthermore, health prediction systems based on workers' biodata are often not highly accurate, resulting in ineffective health management. Furthermore, health management information and reminders are not provided in a user-friendly format, preventing workers from maintaining their health and improving their productivity.

[1527] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1528] In this invention, the server includes a camera monitor that monitors the behavior and facial expressions of workers and detects their stress levels, a health prediction unit that collects biometric data from workers and predicts their health status, a control unit that determines how to respond to workers based on their anxiety and stress levels, and a display unit that provides workers with friendly health management information and reminders. This allows for highly accurate real-time monitoring and analysis of workers' health status and stress levels, enabling prompt and appropriate responses. Furthermore, providing workers with health management information and reminders through friendly avatars helps maintain their health and improve productivity.

[1529] "Care recipients" refers to people such as elderly people or people with disabilities who require support and care.

[1530] "Worker" refers to a worker engaged in work in a particular workplace.

[1531] "Camera monitoring means" refers to a device or system that has the function of monitoring the behavior and facial expressions of a subject using a camera.

[1532] The term "excretion prediction means" refers to a device or system that has the function of predicting the need for excretion based on the biodata of the care recipient.

[1533] "Health prediction means" refers to a device or system that has the function of predicting the health status of a worker based on the worker's biodata.

[1534] "Control means" refers to a device or system that has the ability to determine appropriate responses or actions based on collected data.

[1535] "Display means" refers to a device or system that has the function of visualizing and displaying information.

[1536] "Anxiety state" refers to a state in which the care recipient or worker feels mental anxiety or stress.

[1537] "Biodata" refers to data about living organisms, such as heart rate and skin temperature.

[1538] An "avatar" is a computer-generated character that appears as a user's alter ego or stand-in.

[1539] "Notification means" refers to a device or system that has the function of sending specific information or alerts to a target person.

[1540] "Analysis means" refers to a device or system that has the function of analyzing collected data and extracting useful information.

[1541] The present invention relates to a health management system for workers using factory robots. This system is composed of a camera monitor, a health prediction unit, a control unit, and a display unit.

[1542] Overall system configuration

[1543] Camera monitoring means

[1544] The server uses a camera monitor to monitor the behavior and facial expressions of workers. The camera can be, for example, a smartphone camera or a fixed camera installed in the factory. The video data acquired from this camera is analyzed in real time. For the video analysis, a common video processing library (e.g., OpenCV) and an AI model are used.

[1545] Health prediction tools

[1546] The server uses health prediction methods to predict the health status of workers. For example, wearable devices such as smart watches are used. These devices collect biodata such as heart rate and skin temperature and send it to the server in real time. Health management software (e.g., HealthKit) is used to analyze the data.

[1547] Control means

[1548] The server has a control means for determining the appropriate response based on the collected anxiety and stress levels and biodata. Based on the analysis results, it determines the appropriate response for the worker and sends a notification to the worker and their supervisor. The notification means uses the notification function of a typical smartphone.

[1549] Display means

[1550] The server has a display means for providing workers with friendly health management information and reminders. For example, it uses an avatar to display instructions for relaxation and deep breathing to workers. For this, an avatar generation application such as Unity is used.

[1551] Specific examples

[1552] Example 1: When a worker shows signs of instability while working on a production line

[1553] The camera monitoring means monitors the worker's behavior in real time and detects anxiety. The emotion engine analyzes the anxiety state and sends the data to the server. Meanwhile, the health prediction means collects heart rate and skin temperature from the smartwatch and predicts the worker's health state. The server performs sequential analysis based on this data and determines the necessary response. For example, it sends a notification to the worker to take a break, and an avatar teaches them how to relax.

[1554] Prompt Sentence Examples

[1555] This assistant is designed to monitor the health and stress levels of industrial workers using advanced AI and wearable technology. It can detect workers' anxiety through facial expression analysis and provide real-time health predictions based on biometric data from wearable devices. When necessary, the system alerts the worker or supervisor to suggest a break or convey other health messages through an interactive avatar.

[1556] Features:

[1557] Real-time facial expression analysis using OpenCV

[1558] Emotion recognition using Azure Cognitive Services

[1559] Health prediction using data from Apple Watch via HealthKit

[1560] Actionable alerts and notifications from cloud server

[1561] Friendly avatar interactions using Unity

[1562] Please design a machine learning model that can detect stress from facial expression data and integrate it with the existing health prediction and notification system.

[1563] effect

[1564] This system makes it possible to monitor and analyze workers' health and stress levels in real time with high accuracy, and to take appropriate measures quickly, which not only improves work efficiency and safety but also helps maintain worker health and reduce stress.

[1565] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1566] Step 1:

[1567] Acquisition of video data by camera monitor means

[1568] The server acquires video data of the workers' behavior and facial expressions in real time from cameras in the factory. The input is the video from the camera, and the output is the input video data. This video data is used to analyze anxiety and stress levels in subsequent processing steps. Specifically, the camera monitors the workers 24 hours a day and transmits the video data to the server at a constant frame rate.

[1569] Step 2:

[1570] Analysis of anxiety states using an emotion engine

[1571] The server inputs the acquired video data into the emotion engine for analysis. The input is the video data obtained in step 1, and the output is the analysis result indicating the worker's anxiety state. Specifically, the emotion engine (e.g., Azure Cognitive Services) is used to analyze the worker's facial expressions and movements from the video data, and evaluate the worker's anxiety state numerically.

[1572] Step 3:

[1573] Acquisition and analysis of biodata with health prediction tools

[1574] The server acquires biodata such as heart rate and skin temperature from the worker's wearable device. The input is the biodata from the wearable device, and the output is health status assessment data analyzed by a health prediction algorithm. Specifically, the server receives data from a smartwatch (e.g., Apple Watch) via Bluetooth and analyzes it using health management software such as HealthKit.

[1575] Step 4:

[1576] Deciding how to respond using control measures

[1577] The server determines how to respond based on the anxiety state data from the emotion engine and the health state evaluation data from the health prediction means. The input is the anxiety state data and the health state evaluation data, and the output is the decision on how to respond (e.g., instructing a break, notifying a supervisor). Specifically, a program is run that integrates the data and generates a notification instructing a break if the anxiety state is high or an abnormality in the health state is detected.

[1578] Step 5:

[1579] Sending notifications via notification methods

[1580] The server sends notifications to workers and their supervisors based on the response method determined by the control means. The input is response method data, and the output is a notification message to the recipient. Specifically, the program uses the smartphone's push notification function to send break instructions and health status alerts.

[1581] Step 6:

[1582] Providing information through display means

[1583] The server displays health management information and relaxation method reminders on the worker's smartphone or dedicated device. The input is response method data, and the output is a friendly message using an avatar. Specifically, the avatar is displayed using Unity, and a program runs that suggests ways to relax and take a break to anxious workers.

[1584] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1585] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1586] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1587] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1588] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1589] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1590] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1591] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1592] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1593] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1594] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1595] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1596] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1597] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1598] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1599] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1600] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1601] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1602] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1603] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1604] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1605] The following is further disclosed regarding the above embodiment.

[1606] (Claim 1)

[1607] a camera monitor means for monitoring the behavior and facial expression of the care recipient and detecting an anxiety state;

[1608] an excretion prediction means for collecting biodata of the care recipient and predicting the need for excretion;

[1609] a control measure that determines how to respond to the care recipient based on their anxiety state and need for elimination;

[1610] a display means for displaying a friendly avatar to the care recipient to divert the care recipient's attention from excretion;

[1611] A system including:

[1612] (Claim 2)

[1613] 2. The system according to claim 1, further comprising an analysis means for analyzing the video data acquired by the camera monitor means in real time to detect an anxiety state.

[1614] (Claim 3)

[1615] 2. The system according to claim 1, further comprising a notification means for receiving data from the excretion prediction means and notifying a caregiver when it is determined that the care recipient needs to go to the toilet.

[1616] "Example 1"

[1617] (Claim 1)

[1618] a camera means for monitoring the behavior and facial expression of the care recipient and detecting an anxiety state;

[1619] a predicting means for collecting biological data of the care recipient and predicting the need for excretion;

[1620] a control measure that determines how to respond to the care recipient based on their anxiety state and need for elimination;

[1621] a presentation means for diverting the care recipient's attention from toileting using a display device that is friendly to the care recipient;

[1622] A system including:

[1623] (Claim 2)

[1624] 2. The system according to claim 1, further comprising an analysis means for analyzing the video data acquired by the camera means in real time to detect an anxiety state.

[1625] (Claim 3)

[1626] 2. The system of claim 1, further comprising a notification means for receiving data from the prediction means and notifying a caregiver when it is determined that the care recipient needs to go to the toilet.

[1627] "Application Example 1"

[1628] (Claim 1)

[1629] a monitoring means for monitoring the behavior and facial expression of the care recipient to detect an anxiety state;

[1630] A prediction means for collecting biological information of the care recipient and predicting the need for excretion;

[1631] a control measure that determines how to respond to the care recipient based on their anxiety state and need for elimination;

[1632] a display means for displaying a display that is friendly to the care recipient and diverting the care recipient's attention from excretion;

[1633] A means of collecting data to monitor the health of delivery personnel and improve delivery efficiency;

[1634] A system including:

[1635] (Claim 2)

[1636] 2. The system according to claim 1, further comprising an analysis means for analyzing the video data acquired by the camera monitor means in real time to detect an anxiety state.

[1637] (Claim 3)

[1638] a notification means for receiving data from the excretion prediction means and notifying a caregiver when it is determined that the care recipient needs to go to the toilet;

[1639] 2. The system according to claim 1, further comprising a notification means for analyzing the biometric information of the delivery person and taking measures to ensure that the delivery person can perform their work safely.

[1640] "Example 2: Combining Emotion Engines"

[1641] (Claim 1)

[1642] a monitoring means for monitoring the behavior and facial expression of the care recipient to detect an anxiety state;

[1643] a predictive means for collecting physiological data of the care recipient and predicting the need for excretion;

[1644] a control measure that determines how to respond to the care recipient based on their anxiety state and need for elimination;

[1645] a display means for displaying a display character that is friendly to the care recipient on a screen to divert the care recipient's attention;

[1646] emotion analysis means for recognizing the emotion of a user;

[1647] A system including:

[1648] (Claim 2)

[1649] 2. The system according to claim 1, further comprising an analysis means for analyzing the video data acquired by the monitoring means in real time to detect an anxiety state.

[1650] (Claim 3)

[1651] 2. The system of claim 1, further comprising a notification means for receiving data from the prediction means and notifying a caregiver when it is determined that the care recipient needs to go to the toilet.

[1652] "Application example 2 when combining emotion engines"

[1653] (Claim 1)

[1654] a camera monitor means for monitoring the behavior and facial expression of the care recipient and detecting an anxiety state;

[1655] an excretion prediction means for collecting biodata of the care recipient and predicting the need for excretion;

[1656] a control measure that determines how to respond to the care recipient based on their anxiety state and need for elimination;

[1657] a display means for displaying a friendly avatar to the care recipient to divert the care recipient's attention from excretion;

[1658] a camera monitor means for monitoring the behavior and facial expressions of workers and detecting their stress levels;

[1659] a health prediction means for collecting biodata of workers and predicting their health status;

[1660] A control means for determining a response to a worker based on the worker's anxiety state or stress state;

[1661] A display that provides worker-friendly health management information and reminders;

[1662] A system including:

[1663] (Claim 2)

[1664] 2. The system according to claim 1, further comprising an analysis means for analyzing the video data acquired by the camera monitor means in real time to detect an anxiety state or a stress state.

[1665] (Claim 3)

[1666] 10. The system of claim 1, further comprising a notification means for receiving data from the health prediction means and instructing the worker to take a break. [Explanation of symbols]

[1667] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a camera monitor means for monitoring the behavior and facial expression of the care recipient and detecting an anxiety state; an excretion prediction means for collecting biodata of the care recipient and predicting the need for excretion; A control measure that determines how to respond to the care recipient based on their anxiety state and need for elimination; a display means for displaying a friendly avatar to the care recipient to divert the care recipient's attention from excretion; A system including:

2. 2. The system according to claim 1, further comprising analysis means for analyzing the video data acquired by the camera monitor means in real time to detect an anxiety state.

3. 2. The system according to claim 1, further comprising a notification means for receiving data from the toilet prediction means and notifying a caregiver when it is determined that the care recipient needs to go to the toilet.

Images