System

Abstract

A system is provided.SOLUTION: The system includes a photographing means for acquiring image data for monitoring the physical condition of an aged person, an analysis means for analyzing the acquired image data to generate physical condition information, and a storage / display means for storing and displaying the generated physical condition information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In traditional nursing homes, elderly people face challenges such as managing their physical condition and reducing feelings of loneliness. In addition, there is the risk of them getting lost due to wandering and a lack of convenience within the facility. Specifically, elderly people may feel anxious and lonely because they have no one to talk to, worry about suddenly collapsing indoors, and have difficulty moving around within the facility. These issues are likely to reduce the quality of life and physical and mental health of facility users. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means. Specifically, it provides a system for monitoring the physical condition of elderly people, including an imaging means for acquiring image data, an analysis means for analyzing the acquired image data to generate physical condition information, and a storage / display means for saving and displaying the generated physical condition information. It also includes a voice input means, a voice analysis means, and a voice response generation means for receiving and analyzing voice input from the elderly person and generating a corresponding response. It also provides a location information acquisition means, a location information analysis means, and an abnormality detection means for acquiring the elderly person's location information and detecting abnormal movements based on the location information. These means can improve the physical and mental health and safety of facility users, thereby improving their quality of life.

[0006] "Image data" refers to image information captured to monitor the physical condition of elderly people.

[0007] "Photographing means" refers to cameras and related devices used to photograph the elderly person's face, etc., and obtain image data.

[0008] The "analysis means" refers to algorithms and devices such as software and hardware for analyzing the acquired image data and generating physical condition information of the elderly person.

[0009] "Physical condition information" is a general term for various parameters and data that indicate the health condition of the elderly person, generated by the analysis means.

[0010] The "storage and display means" refers to a storage and display device for storing the generated physical condition information and displaying it as needed.

[0011] "Voice input means" refers to a microphone or related device for accepting voice input from the elderly person and capturing it as voice data.

[0012] The "voice analysis means" refers to algorithms and devices such as software and hardware for analyzing voice data acquired by the voice input means and for understanding and processing the content of speech.

[0013] The "voice response generating means" refers to software or hardware that generates a response based on the analysis results of the voice analysis means and outputs it as voice.

[0014] "Location information acquisition means" refers to a GPS device or other location information measurement device for acquiring the location information of the elderly person in real time.

[0015] The "location information analysis means" refers to an algorithm or device such as software or hardware for analyzing acquired location information and detecting the current state and abnormal movements.

[0016] "Anomaly detection means" refers to software and hardware for identifying abnormal movement patterns detected by the location information analysis means and generating an alert. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention is a system for managing the physical condition of elderly people in nursing homes, reducing feelings of loneliness, and reducing the risk of wandering. This system is realized using smart AI mirrors installed in each room. The specific configuration and program processing of this system are described below.

[0039] System configuration

[0040] This system consists of the following main components:

[0041] 1. Smart AI mirror (device)

[0042] It has a built-in camera to take pictures of the elderly person's face.

[0043] Built-in microphone for voice input.

[0044] Built-in display for showing analysis results.

[0045] 2. Server

[0046] It has the function of receiving and analyzing image data and audio data sent from the terminal.

[0047] It includes a database that generates and stores physical condition information.

[0048] The analysis results are notified to the terminal and the user terminal.

[0049] 3. User terminal (for care staff)

[0050] A tablet or PC that displays health information sent from the server and can be checked by care staff.

[0051] 4. GPS Tracker Insoles

[0052] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[0053] Program processing

[0054] The program for this system is implemented to achieve three main functions: monitoring the physical condition of the elderly, voice interaction, and location information analysis.

[0055] 1. Health monitoring process

[0056] Device:

[0057] When an elderly person stands in front of the mirror, the built-in camera automatically activates and captures an image of their face, which is then sent to a server in real time.

[0058] server:

[0059] The server processes the received image data using an analysis algorithm. As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database, and if necessary, an alert is sent to care staff if an abnormality is detected.

[0060] User device:

[0061] Care staff can check the health information of the elderly person through the user terminal and take necessary measures based on this information.

[0062] Examples:

[0063] Elderly person A stands in front of the smart AI mirror, which takes a picture of their face. The image data is sent to the server, which analyzes the image and determines that "A's health is good." This result is also displayed on the caregiver's tablet.

[0064] 2. Voice Dialogue Processing

[0065] Device:

[0066] When an elderly person speaks to the smart AI mirror, the built-in microphone captures their voice, and the voice data is sent to a server.

[0067] server:

[0068] The server analyzes the audio data and runs natural language processing algorithms to understand what is being said, and generates an appropriate response based on the results.

[0069] Device:

[0070] The response text sent from the server is converted into speech using speech synthesis technology and output to the elderly.

[0071] Examples:

[0072] When Elderly Person B says, "The weather is nice today," the Smart AI Mirror sends the speech to the server. The server analyzes the speech data and generates a response, "The weather is really nice. It would be nice to go for a walk," and outputs this response as voice.

[0073] 3. Location analysis processing

[0074] Location information acquisition method (shoe insoles with GPS tracker):

[0075] The location information of the elderly person is obtained at regular intervals and sent to the server.

[0076] server:

[0077] The server plots the received location information on map data, analyzes movement patterns, and generates a warning message if abnormal movement (e.g., movement into an unauthorized area) is detected.

[0078] User device:

[0079] Care staff will receive a warning message via the user's terminal, enabling them to respond quickly.

[0080] Examples:

[0081] If Elderly Person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his / her location information to the server, which will detect the abnormal location and send a warning to the care staff.

[0082] By combining the above configuration and processing, this system can manage the physical condition of elderly people, reduce feelings of loneliness, and reduce the risk of them wandering. It is an effective means of improving the physical and mental health and quality of life of facility users.

[0083] The processing flow will be explained below.

[0084] Health monitoring process

[0085] Step 1:

[0086] The device detects when an elderly person stands in front of the smart AI mirror, and the built-in camera automatically activates.

[0087] Step 2:

[0088] The device takes a picture of the elderly person's face and captures the image data in real time, then encodes the captured image data.

[0089] Step 3:

[0090] The terminal transmits the encoded image data to the server.

[0091] Step 4:

[0092] The server decodes the received image data and applies it to an analysis algorithm, which estimates blood pressure and heart rate based on facial color, facial expression, and whether the eyes are open or closed.

[0093] Step 5:

[0094] The server stores the physical condition information generated as a result of the analysis in a database, and then transmits this physical condition information to the terminal and the user terminal.

[0095] Step 6:

[0096] The user (care staff) uses the user terminal to check the health information of the elderly person. If any abnormalities are found, they respond promptly.

[0097] Voice interaction processing

[0098] Step 1:

[0099] The device is constantly in listening mode and waiting for voice input from the elderly person.

[0100] Step 2:

[0101] The device captures the voice of the elderly person and records it as audio data.

[0102] Step 3:

[0103] The device converts the captured voice data into text and sends it digitally to a server.

[0104] Step 4:

[0105] The server analyzes the received voice data using a natural language processing algorithm to understand what the elderly person is saying.

[0106] Step 5:

[0107] The server generates an appropriate response text based on the analysis results and sends this response text to the terminal.

[0108] Step 6:

[0109] The response text received by the terminal is converted into voice data using voice synthesis technology and output to the elderly person.

[0110] Processing location analysis

[0111] Step 1:

[0112] The location information acquisition means (shoe insoles with GPS trackers) acquires the elderly person's location information at regular intervals.

[0113] Step 2:

[0114] The location information acquired by the location information acquisition means is encoded and transmitted to a server in real time.

[0115] Step 3:

[0116] The server decodes the received location information and plots it in a map database.

[0117] Step 4:

[0118] The server analyzes movement patterns based on the plotted location information and detects abnormal movement (for example, movement into unauthorized areas).

[0119] Step 5:

[0120] If the server detects an abnormal movement pattern, it generates a warning message and sends this message to the user terminal.

[0121] Step 6:

[0122] The user (care staff) receives and checks the warning message via the user terminal and takes prompt action.

[0123] Example 1

[0124] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] In nursing homes for the elderly, managing the health of the elderly, reducing feelings of loneliness, and reducing the risk of wandering are important issues. Conventional systems do not provide comprehensive solutions to these issues, which increases the burden on care staff and makes it difficult to ensure the safety and health of the elderly. In addition, the high risk of wandering requires real-time monitoring of location information. Therefore, a system that integrates health monitoring, voice dialogue, and location information analysis for the elderly is needed.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0127] In this invention, the server includes a camera means for acquiring image data, an analysis means for analyzing the acquired image data to generate health information, a database means for saving the generated health information, a display means for displaying the generated health information, a notification means for sending an alert when an abnormality is detected, a voice input means for receiving and analyzing voice input and generating a corresponding response, a natural language processing means for analyzing the voice data, a voice response generation means, a voice synthesis means for synthesizing the voice response, and a means for acquiring location information of the elderly person, detecting abnormal movements based on the location information, and sending a warning message as necessary. This makes it possible to appropriately manage the health of the elderly person, reduce feelings of loneliness, and reduce the risk of wandering.

[0128] "Camera means for capturing image data" refers to a photographic device used to capture images of the face and body of the elderly person.

[0129] "Analysis means for analyzing acquired image data to generate physical condition information" refers to algorithms and software for processing acquired image data and calculating physical condition information such as blood pressure, heart rate, and complexion of the elderly person.

[0130] The "database means for storing generated physical condition information" refers to a database system that stores generated physical condition information and can retrieve it as needed.

[0131] The "display means for displaying the generated physical condition information" refers to a display or screen for visually displaying the generated physical condition information.

[0132] "Notification means for sending an alert when an abnormality is detected" refers to a system that sends a warning message to care staff or appropriate personnel when an abnormality is detected in health information.

[0133] "Voice input means" refers to a microphone or voice recognition device that captures the voice spoken by the elderly person and collects the voice data.

[0134] "Natural language processing means for analyzing voice data" refers to natural language processing algorithms and software for analyzing acquired voice data and understanding its content.

[0135] "Voice response generator" refers to software or algorithms for generating appropriate responses based on analyzed voice data.

[0136] The "voice synthesis means for synthesizing a voice response" refers to a voice synthesis technology for outputting the generated response as voice.

[0137] "Location information acquisition means" refers to a location information acquisition device such as a GPS system used to determine the current location of an elderly person.

[0138] "Map data means for plotting received location information on map data" refers to a system for displaying acquired location information on a map.

[0139] "Anomaly detection means for detecting abnormal movements" refers to algorithms or software for analyzing acquired location information and detecting abnormal movements or behaviors.

[0140] "Warning means for sending a warning message when an abnormality is detected" refers to a system that sends a warning message to relevant parties when an abnormality in movement is detected.

[0141] MODE FOR CARRYING OUT THE INVENTION

[0142] This invention is a system aimed at managing the health of elderly people, reducing feelings of loneliness, and reducing the risk of wandering. This system is realized using a smart AI mirror installed in each room. The specific configuration and program processing of this system are described in detail below.

[0143] System configuration

[0144] This system consists of the following main components:

[0145] 1. Smart AI mirror (device)

[0146] It has a built-in camera to take pictures of the elderly person's face.

[0147] Built-in microphone for voice input.

[0148] Built-in display for showing analysis results.

[0149] 2. Server

[0150] It has the function of receiving and analyzing image data and audio data sent from the terminal.

[0151] It includes a database that generates and stores physical condition information.

[0152] The analysis results are notified to the terminal and the user terminal.

[0153] 3. User terminal (for care staff)

[0154] A tablet or PC that displays health information sent from the server and can be checked by care staff.

[0155] 4. GPS Tracker Insoles

[0156] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[0157] Program processing

[0158] The program for this system is implemented to achieve three main functions: monitoring the physical condition of the elderly, voice interaction, and location information analysis.

[0159] Health monitoring process

[0160] Device:

[0161] When an elderly person stands in front of the mirror, the built-in camera automatically activates and captures an image of their face, which is then sent to a server in real time.

[0162] server:

[0163] The server processes the received image data using an analysis algorithm (e.g., OpenCV or TENSORFLOW (registered trademark)). As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database (e.g., MySQL (registered trademark)) and an alert is sent to care staff as necessary.

[0164] User device:

[0165] Care staff can check the health information of the elderly person through the user terminal and take necessary measures based on this information.

[0166] Examples:

[0167] When elderly person A stands in front of the smart AI mirror and the mirror takes a picture of their face, the image data is sent to the server. The server analyzes the image and determines that "A's health is good." This result is also displayed on the care staff's tablet.

[0168] Voice interaction processing

[0169] Device:

[0170] When an elderly person speaks to the smart AI mirror, the built-in microphone captures their voice, and the voice data is sent to a server.

[0171] server:

[0172] The server runs a natural language processing algorithm (e.g., Google®'s Speech-to-Text API or OpenAI®'s GPT-3® model) to analyze the voice data and generates an appropriate response based on the results.

[0173] Device:

[0174] The response text sent from the server is converted into speech using speech synthesis technology (e.g., Google's Text-to-Speech API) and output to the elderly via the mirror.

[0175] Examples:

[0176] When Elderly Person B says, "The weather is nice today," the Smart AI Mirror sends the speech to the server. The server analyzes the speech data and generates a response, "The weather is really nice. It would be nice to go for a walk," and outputs this response as voice.

[0177] Processing location analysis

[0178] Location information acquisition method (shoe insoles with GPS tracker):

[0179] The location information of the elderly person is obtained at regular intervals and sent to the server.

[0180] server:

[0181] The server plots the received location information on map data (e.g., Google Maps API) and analyzes movement patterns. If abnormal movement (e.g., movement into an unauthorized area) is detected, a warning message is generated.

[0182] User device:

[0183] Care staff will receive a warning message via the user's terminal, enabling them to respond quickly.

[0184] Examples:

[0185] If Elderly Person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his / her location information to the server, which will detect the abnormality and send an alert to the care staff.

[0186] Prompt Sentence Examples

[0187] Example prompt 1 (health monitoring):

[0188] "Please extract health information from facial images of elderly people. Specifically, please generate information on three elements: blood pressure, heart rate, and complexion."

[0189] Prompt example 2 (voice dialogue):

[0190] "Generate an appropriate response when an older adult says, 'The weather is nice today.'"

[0191] Example prompt 3 (location analysis):

[0192] "Implement an algorithm that receives location information from seniors, plots it on map data, and detects abnormal movements."

[0193] This system, configured in this way, plays an important role in managing the health of the elderly, and also contributes to reducing feelings of loneliness and the risk of wandering.

[0194] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0195] Health monitoring process

[0196] Step 1: Capture and send a facial image

[0197] Terminal: When an elderly person stands in front of the smart AI mirror, the terminal activates the built-in camera to capture the elderly person's facial image, and the input data is the elderly person's facial image, which is then sent to the server in real time.

[0198] Specific operation: When elderly person A stands in front of the mirror, the device's camera automatically starts up, takes a picture of his face, encrypts the image, and sends it to the server.

[0199] Step 2: Analyzing the image data

[0200] Server: The server uses algorithms (e.g., OpenCV or TensorFlow) to analyze the received image data. The input data is a facial image, which is analyzed to generate physical condition information such as the elderly person's blood pressure, heart rate, and complexion.

[0201] Specific operation: The server uses a GPU to quickly analyze the image data and calculates that the blood pressure is 120 / 80 and the heart rate is 70 BPM.

[0202] Step 3: Save and notify health information

[0203] Server: The generated health information is stored in a database (e.g., MySQL). The input data is the generated health information, and if an abnormality is detected, an alert is sent to the care staff.

[0204] Specific operation: When the server detects an abnormal value (e.g., blood pressure 160 / 100), it immediately generates an alert and notifies the care staff's terminal.

[0205] Step 4: Check your health information

[0206] User: The care staff checks the health information of the elderly person through the user terminal. The input data is the generated health information, and the care staff takes necessary measures based on this.

[0207] Specific actions: Caregiver B uses a tablet device to check A's health information and immediately contacts the doctor.

[0208] Voice interaction processing

[0209] Step 1: Capture and send audio

[0210] Terminal: When the elderly person speaks to the smart AI mirror, the built-in microphone of the terminal captures the voice and transmits the voice data to the server in real time. The input data is an audio file.

[0211] Specific operation: When Elderly Person B says, "The weather is nice today," the device's microphone records the voice and sends it to the server.

[0212] Step 2: Analyzing the audio data

[0213] Server: The server uses natural language processing algorithms (e.g., Google's Speech-to-Text API or OpenAI's GPT-3 model) to analyze the voice data and understand what is being said. The input data is an audio file, which is converted into text and an appropriate response is generated.

[0214] What it does: The server converts the speech "The weather is nice today" into text and generates a response saying "The weather is really nice. It would be nice to go for a walk."

[0215] Step 3: Generate a voice response

[0216] Terminal: The response text sent from the server is converted into speech using speech synthesis technology (e.g., Google's Text-to-Speech API). The input data is the response text, which is then output as speech to the elderly.

[0217] Specific operation: The device outputs a synthesized voice saying, "What beautiful weather. It would be nice to go for a walk."

[0218] Processing location analysis

[0219] Step 1: Obtaining and sending location information

[0220] Location information acquisition means (shoe insoles with GPS trackers): The location information of the elderly is acquired at regular intervals (e.g., every minute) and sent to the server. The input data is location coordinates.

[0221] Specific operation: The GPS built into the insole of elderly person C's shoe captures location information every minute and sends it to the server.

[0222] Step 2: Analyze location information

[0223] Server: The server plots the received location information on map data (e.g., Google Maps API) and analyzes movement patterns. The input data is location information, and abnormal movements are detected.

[0224] Specific operation: The server checks the location of elderly person C, who has left the facility, on a map and detects any abnormalities.

[0225] Step 3: Notification of warning messages

[0226] User terminal: The nursing staff receives the warning message through the user terminal and responds promptly. The input data is the warning message.

[0227] Specific operation: A warning notification stating "Elderly person C has left the facility" appears on the smartphone of caregiver D.

[0228] (Application example 1)

[0229] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0230] In managing the physical and mental health of the elderly, there is a need for systems that allow elderly people at high risk of wandering to live within a safe range, and that allow caregivers and family members to monitor the elderly's location and health information in real time and respond quickly. However, existing systems lack the functionality to manage this information in an integrated manner and send appropriate alerts in real time when an abnormality occurs. It is also important to have a function that can reduce the elderly's sense of loneliness through voice dialogue.

[0231] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0232] In this invention, the server includes an imaging means for acquiring image data, an analysis means for analyzing the acquired image data to generate health information, a storage and display means for storing and displaying the generated health information, a location information acquisition means for acquiring location information, an abnormality detection means for detecting abnormal movements by comparing the acquired location information with a geographical fence, an alert notification means for sending a notification when an abnormality is detected, a voice input means, a voice analysis means, a voice response generation means, a user interface means for integrating the elderly person's health information and location information and enabling real-time confirmation, and a push notification means. This enables health management of the elderly person, monitoring of location information, and prompt response when an abnormality is detected.

[0233] "Photographing means for acquiring image data" refers to a device such as a camera for photographing an image of the face of an elderly person.

[0234] "Analysis means" refers to the algorithms and software that analyze the acquired image data and generate physical condition information such as blood pressure, heart rate, and complexion.

[0235] "Storage and display means" refers to a database that stores the generated physical condition information and a system for displaying that information on a display or the like.

[0236] "Location information acquisition means" refers to devices such as GPS sensors that acquire the current location of elderly people in real time.

[0237] "Anomaly detection means" refers to an algorithm or software that analyzes acquired location information and determines that an anomaly has occurred if the information exceeds a set safety range (geographical fence).

[0238] "Alert notification means" refers to push notification services and communication means for sending notifications to caregivers and family members when an abnormality is detected.

[0239] "Voice input means" refers to a device such as a microphone for capturing the elderly person's voice.

[0240] "Speech analysis means" refers to natural language processing algorithms or software that analyzes captured voice data and understands what is being said.

[0241] The "voice response generation means" refers to a voice synthesis technology for generating an appropriate response based on the analysis results and outputting the response as voice.

[0242] "User interface means" refers to an interface that displays the elderly person's physical condition information and location information in real time, allowing caregivers and family members to easily check them.

[0243] "Push notification means" refers to a system that sends notifications to caregivers' and family members' devices in real time when an abnormality is detected.

[0244] This invention provides a system for managing the physical condition of elderly people, tracking their location information, and reducing the risk of them wandering. The specific configuration and processing of this system will be described below.

[0245] System configuration

[0246] This system consists of the following main components:

[0247] 1. Terminal

[0248] These are mobile devices such as smartphones and smart glasses that are used as belongings for the elderly.

[0249] It has a built-in camera for taking facial images of elderly people.

[0250] It has a built-in microphone for accepting voice input.

[0251] It has a built-in GPS sensor to acquire location information.

[0252] 2. Server

[0253] It has the function of receiving and analyzing image data and audio data sent from the terminal.

[0254] It has a database that generates and stores health information and location information.

[0255] It has a system to generate alerts and notify you if an abnormality is detected.

[0256] 3. User Device

[0257] A tablet or PC that displays health information and location information sent from a server and can be checked by care staff and family members.

[0258] What the program does

[0259] 1. Health monitoring process

[0260] Device: When an elderly person stands in front of the camera, their facial image is automatically captured. This image data is sent to the server in real time.

[0261] Server: Analyzes image data using artificial intelligence analysis tools (e.g., TensorFlow, PyTorch, etc.) to generate health information such as blood pressure and heart rate. The generated health information is stored in a database. If an abnormality is detected, an alert notification is sent to the caregiver's device.

[0262] User device: Care staff can check the health information of elderly people via tablets or PCs and take necessary measures.

[0263] Example: An elderly person stands in front of the device and the camera captures their face. The image data is sent to a server, which analyzes it and determines that the person is in good health. This is then displayed on the caregiver's tablet.

[0264] 2. Voice Dialogue Processing

[0265] Terminal: When an elderly person speaks to the terminal, the microphone captures the voice and the voice data is sent to the server.

[0266] Server: Analyzes the voice data using natural language processing algorithms and generates an appropriate response.

[0267] Terminal: The response sent from the server is output as voice using speech synthesis technology.

[0268] Example: When an elderly person says, "The weather is nice today," the device sends the speech to the server. The server generates a response, "It's really nice weather. It would be nice to go for a walk," and the device outputs it as speech.

[0269] 3. Location analysis processing

[0270] Device: The device's GPS sensor acquires the elderly person's location information and sends it to the server.

[0271] Server: Executes anomaly detection measures that compare location information with geo-fences to detect abnormal movements. If an anomaly is detected, an alert notification measure is used to send a notification to caregivers and family members.

[0272] User device: Care staff and family members can receive notifications through the device, check the elderly person's location, and respond quickly.

[0273] Example: If an elderly person goes outside a set safe area, the GPS sensor sends their location to a server. The server detects the abnormality and sends a notification to the caregiver's smartphone that "the elderly person has gone outside the range."

[0274] Prompt Sentence Examples

[0275] 1. Obtaining location information

[0276] "Get the user's current location. Use the GPS sensor to send the latitude and longitude to the server in real time."

[0277] 2. Setting up geo-fences

[0278] It provides a GUI for configuring safety limits within the app and stores the user's settings in a database.

[0279] 3. Sending emergency alerts

[0280] "If out-of-range movement is detected, a push notification is sent to the registered caregiver's smartphone using Firebase Cloud Messaging."

[0281] 4. Check your health information

[0282] "We will create a UI that displays the user's health data received from the server and updates it in real time."

[0283] This allows the system to manage the health of elderly people and monitor their location information in an integrated manner, protecting their safety and reducing the burden on caregivers and family members.

[0284] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0285] Step 1:

[0286] Terminal: Acquisition of image data

[0287] The device's camera captures the elderly person's facial image. The elderly person's face must be in front of the camera as input, and the captured facial image data is obtained as output.

[0288] Step 2:

[0289] Terminal: Sending image data

[0290] The acquired facial image data is sent to the server in real time. The input is the facial image data captured in step 1, and the output is the data successfully sent to the server.

[0291] Step 3:

[0292] Server: Generates health information

[0293] The server analyzes the received facial image data using an artificial intelligence algorithm (e.g., TensorFlow, PyTorch) to generate physical condition information such as blood pressure, heart rate, and complexion. The input is the facial image data sent from the device, and the output is the generated physical condition information.

[0294] Step 4:

[0295] Server: Storage of health information

[0296] The generated physical condition information is saved in a database. The input is the physical condition information generated in step 3, and the output is the physical condition information saved in the database.

[0297] Step 5:

[0298] Server: Anomaly detection

[0299] The server monitors the stored health information and generates an alert if an abnormality is detected. Pre-set criteria are used for abnormality detection. The input is the health information stored in the database, and the output is an alert notification if an abnormality is detected.

[0300] Step 6:

[0301] Server: Alert Notification

[0302] If an abnormality is detected, the server sends an alert notification to the caregiver or family member's device. The input is the alert generated in step 5, and the output is the notification sent to the caregiver or family member's device.

[0303] Step 7:

[0304] Device: Audio data capture

[0305] The microphone on the device captures the elderly person's voice, and the input is the elderly person's voice, and the output is the captured voice data.

[0306] Step 8:

[0307] Terminal: Sending audio data

[0308] Send the captured audio data to the server in real time. The input is the audio data captured in step 7, and the output is the data successfully sent to the server.

[0309] Step 9:

[0310] Server: Analysis of voice data

[0311] The server analyzes the received voice data, uses natural language processing algorithms (e.g., BERT, GPT-3, etc.) to understand the speech and generate an appropriate response. The input is the voice data sent from the device, and the output is the generated response text.

[0312] Step 10:

[0313] Server: Sending a voice response

[0314] The server sends the generated response text to the terminal. The input is the response text generated in step 9, and the output is the response text sent to the terminal.

[0315] Step 11:

[0316] Terminal: Voice response output

[0317] The device converts the received response text into speech and outputs it to the elderly. It uses speech synthesis technology (e.g., a TTS engine). The input is the response text sent from the server, and the output is the speech output to the elderly.

[0318] Step 12:

[0319] Device: Location information acquisition

[0320] The GPS sensor on the device periodically acquires the elderly person's location information. The input is the current GPS data, and the output is the acquired location information.

[0321] Step 13:

[0322] Device: Sending location information

[0323] Send the acquired location information to the server. The input is the location information acquired in step 12, and the output is the data successfully sent to the server.

[0324] Step 14:

[0325] Server: Location analysis

[0326] The server analyzes the received location information and detects anomalies if the location information exceeds the set geographical fence. The input is the location information sent from the device, and the output is an alert notification if an anomaly is detected.

[0327] Step 15:

[0328] Server: Alert notification of abnormal location

[0329] If an abnormality is detected, the server sends an alert notification to the caregiver or family device. The input is the alert generated in step 14, and the output is the notification sent to the caregiver or family device.

[0330] Step 16:

[0331] User: Check health and location information

[0332] Caregivers and family members can check the health and location information through the user's device. The input is the health and location information sent from the server, and the output is the information displayed on the user's device.

[0333] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0334] This invention is a system for managing the physical condition of elderly people in nursing homes, reducing feelings of loneliness, reducing the risk of wandering, and providing emotional care through emotion recognition. This system is realized using smart AI mirrors installed in each room, and by combining it with an emotion recognition engine, it is also possible to care for the elderly's emotional state. The specific configuration of this system and the program processing are described below.

[0335] System configuration

[0336] This system consists of the following main components:

[0337] 1. Smart AI mirror (device)

[0338] It has a built-in camera to take pictures of the elderly person's face.

[0339] Built-in microphone for voice input.

[0340] Built-in display for showing analysis results.

[0341] It has a built-in emotion recognition engine to recognize emotions.

[0342] 2. Server

[0343] It has the ability to receive and analyze image data, audio data, and emotional data sent from the terminal.

[0344] It includes a database that generates and stores physical condition information, voice analysis results, and emotion analysis results.

[0345] The analysis results are notified to the terminal and the user terminal.

[0346] 3. User terminal (for care staff)

[0347] Tablets and PCs that display various information sent from the server and can be checked by care staff.

[0348] 4. GPS Tracker Insoles

[0349] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[0350] Program processing

[0351] The program of this system is implemented mainly to realize the functions of monitoring the physical condition of the elderly, voice dialogue, location information analysis, and emotion recognition.

[0352] 1. Health monitoring process

[0353] Device:

[0354] When an elderly person stands in front of the mirror, the built-in camera automatically activates to capture an image of their face, which is then sent to a server in real time.

[0355] server:

[0356] The server processes the received image data using an analysis algorithm. As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database, and if necessary, an alert is sent to care staff if an abnormality is detected.

[0357] User device:

[0358] Care staff can check the health information of the elderly person through the user terminal and take necessary measures based on this information.

[0359] Examples:

[0360] Elderly person A stands in front of the smart AI mirror, which takes a picture of their face. The image data is sent to the server, which analyzes the image and determines that "A's health is good." This result is also displayed on the caregiver's tablet.

[0361] 2. Voice Dialogue Processing

[0362] Device:

[0363] When an elderly person speaks to the smart AI mirror, the built-in microphone captures their voice, and the voice data is sent to a server.

[0364] server:

[0365] The server analyzes the audio data and runs natural language processing algorithms to understand what is being said, and generates an appropriate response based on the results.

[0366] Device:

[0367] The response text sent from the server is converted into speech using speech synthesis technology and output to the elderly.

[0368] Examples:

[0369] When Elderly Person B says, "The weather is nice today," the Smart AI Mirror sends the speech to the server. The server analyzes the speech data and generates a response, "The weather is really nice. It would be nice to go for a walk," and outputs this response as voice.

[0370] 3. Location analysis processing

[0371] Location information acquisition method (shoe insoles with GPS tracker):

[0372] The location information of the elderly person is obtained at regular intervals and sent to the server.

[0373] server:

[0374] The server plots the received location information on map data, analyzes movement patterns, and generates a warning message if abnormal movement (e.g., movement into an unauthorized area) is detected.

[0375] User device:

[0376] Care staff will receive a warning message via the user's terminal, enabling them to respond quickly.

[0377] Examples:

[0378] If Elderly Person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his / her location information to the server, which will detect the abnormal location and send a warning to the care staff.

[0379] 4. Emotion Recognition Processing

[0380] Device:

[0381] When an elderly person stands in front of the mirror, the built-in camera and microphone use an emotion recognition engine to capture emotions from the elderly person's facial expressions and voice.

[0382] server:

[0383] The server analyzes the captured emotional data to identify the elderly person's emotional state, and the analysis results are stored in a database.

[0384] Device:

[0385] Based on the emotion analysis results sent from the server, an appropriate response is made. For example, if the emotion is negative, an encouraging voice response is generated.

[0386] User device:

[0387] Care staff can also check the emotional state of the elderly person through the user device and take the necessary measures to provide mental care.

[0388] Examples:

[0389] When elderly person D faces the mirror, the emotion recognition engine determines that D is "sad" from his facial expression. This information is sent to the server and displayed on the care staff's tablet. The staff plays a voice response from the mirror saying, "Are you OK, Mr. D? Shall we talk?"

[0390] By combining the above configuration and processing, this system can manage the physical condition of elderly people, reduce feelings of loneliness, reduce the risk of wandering, and provide psychological care through emotion recognition, thereby significantly improving the physical and mental health and quality of life of facility users.

[0391] The processing flow will be explained below.

[0392] Health monitoring process

[0393] Step 1:

[0394] The device detects when an elderly person stands in front of the smart AI mirror, and the built-in camera automatically activates.

[0395] Step 2:

[0396] The device takes a picture of the elderly person's face and acquires image data in real time.

[0397] Step 3:

[0398] The terminal encodes the acquired image data and transmits it to the server in real time.

[0399] Step 4:

[0400] The server decodes the received image data and applies it to an analysis algorithm, which estimates blood pressure and heart rate based on facial color, facial expression, and whether the eyes are open or closed.

[0401] Step 5:

[0402] The server stores the physical condition information generated as a result of the analysis in a database, and then transmits this physical condition information to the terminal and the user terminal.

[0403] Step 6:

[0404] The user (care staff) uses the user terminal to check the health information of the elderly person. If any abnormalities are found, they respond promptly.

[0405] Voice interaction processing

[0406] Step 1:

[0407] The device is constantly in listening mode and waiting for voice input from the elderly person.

[0408] Step 2:

[0409] The device captures the voice of the elderly person and records it as audio data.

[0410] Step 3:

[0411] The device converts the captured voice data into text and sends it digitally to a server.

[0412] Step 4:

[0413] The server analyzes the received voice data using a natural language processing algorithm to understand what the elderly person is saying.

[0414] Step 5:

[0415] The server generates an appropriate response text based on the analysis results and sends this response text to the terminal.

[0416] Step 6:

[0417] The response text received by the terminal is converted into voice data using voice synthesis technology and output to the elderly person.

[0418] Processing location analysis

[0419] Step 1:

[0420] The location information acquisition means (shoe insoles with GPS trackers) acquires the elderly person's location information at regular intervals.

[0421] Step 2:

[0422] The location information acquired by the location information acquisition means is encoded and transmitted to a server in real time.

[0423] Step 3:

[0424] The server decodes the received location information and plots it in a map database.

[0425] Step 4:

[0426] The server analyzes movement patterns based on the plotted location information and detects abnormal movement (for example, movement into unauthorized areas).

[0427] Step 5:

[0428] If the server detects an abnormal movement pattern, it generates a warning message and sends this message to the user terminal.

[0429] Step 6:

[0430] The user (care staff) receives and checks the warning message via the user terminal and takes prompt action.

[0431] Emotion recognition processing

[0432] Step 1:

[0433] The device activates the built-in camera to capture the elderly person's facial expressions, and also activates the built-in microphone to capture their voice.

[0434] Step 2:

[0435] The facial expression data and voice data captured by the device are encoded in real time and sent to the server.

[0436] Step 3:

[0437] The server decodes the received facial expression and voice data and analyzes it using an emotion recognition engine, which identifies the elderly person's emotional state (joy, anger, sadness, happiness, etc.).

[0438] Step 4:

[0439] The server stores the emotion analysis results in a database and, if necessary, transmits the analysis results to the terminal and the user terminal.

[0440] Step 5:

[0441] The device generates an appropriate response based on the emotion analysis results. For example, if an elderly person is sad, it generates an encouraging or comforting voice response.

[0442] Step 6:

[0443] The response generated by the device is converted into voice data using voice synthesis technology and output to the elderly. This information is also notified to the user (care staff) so that appropriate action can be taken.

[0444] To give a concrete example:

[0445] When elderly person D faces the mirror, the emotion recognition engine determines from D's facial expression that he is "sad." This information is sent to the server and displayed on the care staff's tablet. The staff then plays a voice response from the mirror saying, "Are you OK, Mr. D? Shall we talk?"

[0446] By combining the above configuration and processing, this system can manage the physical condition of elderly people, reduce feelings of loneliness, reduce the risk of wandering, and provide psychological care through emotion recognition, thereby significantly improving the physical and mental health and quality of life of facility users.

[0447] Example 2

[0448] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0449] The purpose of this invention is to manage the physical condition of elderly people, reduce feelings of loneliness, reduce the risk of wandering, and provide psychological care through emotion recognition. Conventional systems have individual functions, but few have been integrated into a single system, making it difficult to efficiently link multiple systems. Furthermore, conventional technology has limitations in meeting the demand for real-time anomaly detection and rapid response.

[0450] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an imaging means for acquiring image data, an analysis means for analyzing the acquired image data and generating physical condition information, a storage and display means for saving and displaying the generated physical condition information, an emotion recognition means for acquiring and analyzing emotion data and identifying the emotional state, and an alert notification means for sending an alert when an abnormality is detected. This makes it possible to perform physical condition management and mental care for the elderly in an integrated and real-time manner.

[0451] The "photographing means for acquiring image data" is a device for photographing the face and physical condition of an elderly person and acquiring the image data.

[0452] "Analysis means for analyzing acquired image data and generating physical condition information" refers to a device or program for analyzing and generating physical condition information such as blood pressure, heart rate, and complexion of an elderly person using acquired image data.

[0453] The "storage and display means for storing and displaying the generated physical condition information" refers to a device or program for storing the physical condition information generated by the analysis in a database and displaying it as needed.

[0454] "Emotion recognition means for acquiring and analyzing emotional data to identify an emotional state" refers to a device or program for capturing emotions from the facial expressions and voice of an elderly person and analyzing the emotional data to identify the emotional state.

[0455] An "alert notification means for sending an alert when an abnormality is detected" is a device or program for sending a notification to care staff and other relevant parties when an abnormality is detected based on the elderly person's physical condition, location information, etc.

[0456] The "voice input means" is a device for capturing the voice of the elderly person and obtaining the voice data.

[0457] The "voice analysis means" is a device or program for analyzing acquired voice data and understanding its content.

[0458] The "voice response generating means" is a device or program for generating an appropriate response based on the analyzed voice data.

[0459] The "location information acquisition means" is a device for acquiring the current location of the elderly person in real time and transmitting that location information.

[0460] The "location information analysis means" is a device or program for analyzing the acquired location information and plotting it on a map to detect abnormal movements.

[0461] The "abnormality detection means" is a device or program for detecting abnormal situations based on the physical condition and location information of the elderly person.

[0462] This invention is a system that aims to manage the physical condition of elderly people, reduce feelings of loneliness, reduce the risk of wandering, and provide mental care through emotion recognition. This system is realized using smart AI mirrors installed in each room, and by combining it with an emotion recognition engine, it is also possible to care for the mental state of the elderly. The specific configuration and program processing of this system are explained below.

[0463] System configuration

[0464] This system consists of the following main components:

[0465] 1. Smart AI mirror (device)

[0466] It has a built-in camera to take pictures of the elderly person's face.

[0467] Built-in microphone for voice input.

[0468] Built-in display for showing analysis results.

[0469] It has a built-in emotion recognition engine to recognize emotions.

[0470] 2. Server

[0471] It has the ability to receive and analyze image data, audio data, and emotional data sent from the terminal.

[0472] It includes a database that generates and stores physical condition information, voice analysis results, and emotion analysis results.

[0473] The analysis results are notified to the terminal and the user terminal.

[0474] 3. User terminal (for care staff)

[0475] Tablets and PCs that display various information sent from the server and can be checked by care staff.

[0476] 4. GPS Tracker Insoles

[0477] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[0478] Specific processing explanation

[0479] Elderly health monitoring

[0480] Device:

[0481] When an elderly person stands in front of the smart AI mirror, the built-in camera captures an image of their face, which is then sent to a server in real time.

[0482] server:

[0483] The server processes the received image data using an analysis algorithm (e.g., OpenCV or TensorFlow). As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database, and if an abnormality is detected, an alert is sent to the care staff.

[0484] User device:

[0485] Care staff can check the health information of the elderly person through the user terminal and take necessary measures.

[0486] Examples:

[0487] When elderly person A stands in front of the smart AI mirror, the built-in camera takes a picture of their face. The image data is analyzed and the mirror determines that "A's health is good." This result can also be viewed on the caregiver's tablet.

[0488] Voice dialogue with the elderly

[0489] Device:

[0490] When an elderly person talks to the smart AI mirror, the built-in microphone captures the voice and sends it to the server.

[0491] server:

[0492] The server analyzes the voice data, runs natural language processing algorithms (e.g., Google Cloud Speech-to-Text), and generates an appropriate response based on the analysis results.

[0493] Device:

[0494] The generated response text is converted into speech using speech synthesis technology (e.g., Google Cloud Text-to-Speech) and output to the elderly.

[0495] Examples:

[0496] When Elderly Person B says, "The weather is nice today," the AI ​​Mirror sends the voice data to the server. The server analyzes the voice and generates a response saying, "The weather is really nice. It would be nice to go for a walk," and outputs it as voice.

[0497] Analysis of elderly people's location information

[0498] Location information acquisition method (shoe insoles with GPS tracker):

[0499] The location information of the elderly person is obtained at regular intervals and sent to the server.

[0500] server:

[0501] The server plots the received location information on map data (e.g., Google Maps API) and analyzes movement patterns. If abnormal movement is detected, a warning message is generated.

[0502] User device:

[0503] Care staff will receive a warning message and be able to respond quickly.

[0504] Examples:

[0505] If elderly person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his location information to the server, and the server will detect the abnormality and send an alert to the care staff.

[0506] Emotion recognition in the elderly

[0507] Device:

[0508] When an elderly person stands in front of the mirror, the built-in camera and microphone capture their emotions using an emotion recognition engine (e.g., Microsoft® Azure® Face API).

[0509] server:

[0510] The captured emotional data is analyzed to identify the emotional state of the elderly person, and the analysis results are stored in a database.

[0511] Device:

[0512] Based on the emotion analysis results sent from the server, an appropriate response is made. For example, if the emotion is negative, an encouraging voice response is generated.

[0513] User device:

[0514] Care staff check the emotional state of the elderly and take the necessary steps to provide mental care.

[0515] Examples:

[0516] When elderly person D looks at the mirror, the emotion recognition engine determines that he is "sad." This information is sent to the server and displayed on the care staff's device. The staff then plays a voice response from the mirror saying, "Mr. D, are you okay? Shall we talk?"

[0517] By combining the above configuration and processing, this system can comprehensively manage the physical condition of the elderly, reduce feelings of loneliness, reduce the risk of wandering, and provide psychological care through emotion recognition, which is expected to significantly improve the physical and mental health and quality of life of the elderly.

[0518] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0519] Step 1:

[0520] Elderly Image Capture

[0521] Terminal: When an elderly person stands in front of the smart AI mirror, the built-in camera automatically captures an image of their face. The image of the elderly person's face reflected in the camera is captured as input, and this is obtained as image data as output. Specifically, when Elderly Person A looks at the mirror with a smile, the display will show "Analyzing your physical condition."

[0522] Step 2:

[0523] Sending image data

[0524] Terminal: Sends the captured image data to the server. The acquired image data is used as input and sent securely using encryption technology (e.g., SSL / TLS). The image data is then transferred to the server as output. Specifically, the image data of elderly person A is sent from the mirror to the server within a few seconds.

[0525] Step 3:

[0526] Image data analysis

[0527] Server: Processes the received image data using an analysis algorithm (e.g., OpenCV or TensorFlow). Using the transmitted image data as input, the server analyzes it to generate health information such as the elderly person's blood pressure, heart rate, and complexion. This health information is obtained as output. Specifically, the server analyzes the image and determines that elderly person A's heart rate is "normal."

[0528] Step 4:

[0529] Storage and notification of health information

[0530] Server: The generated health information is saved in a database. If an abnormality is detected, an alert is sent to the care staff. The health information from the analysis is used as input, and the process of saving it to the database and sending an alert is executed. The output is the saving of the health information and the sending of an alert. Specifically, the server determines that elderly person A's blood pressure is high, and an alert is sent to the care staff's tablet.

[0531] Step 5:

[0532] Displaying health information

[0533] User device: The care staff checks the health information of the elderly person through the user device. The health information sent from the server is used as input and displayed. The health information can be checked by the care staff as output. Specifically, the care staff checks the health information of elderly person A on the tablet device and takes the necessary measures.

[0534] Step 6:

[0535] Capturing voice input

[0536] Terminal: When an elderly person speaks to the smart AI mirror, the built-in microphone captures the voice and sends this voice data to the server. The elderly person's voice is captured as input and obtained as voice data. The voice data is sent to the server as output. In concrete terms, when Elderly Person B speaks to the mirror saying, "How's the weather?", the mirror displays "Analyzing voice."

[0537] Step 7:

[0538] Analysis of audio data

[0539] Server: Analyzes the voice data and runs a natural language processing algorithm (e.g., Google Cloud Speech-to-Text). It uses the transmitted voice data as input and performs the analysis. As output, it generates corresponding text data and obtains an appropriate response text. Specifically, the server analyzes Person B's voice and generates the response "It's sunny today."

[0540] Step 8:

[0541] Response generation and speech output

[0542] Terminal: The response text sent from the server is converted into speech using speech synthesis technology (e.g., Google Cloud Text-to-Speech) and output to the elderly person. The response text is used as input to generate speech. A voice notification is output to the elderly person. Specifically, the mirror notifies Mr. B by voice, saying, "It's sunny today."

[0543] Step 9:

[0544] Obtaining location information

[0545] Location information acquisition means (shoe insoles with GPS trackers): The location information of the elderly person is acquired at regular intervals and sent to the server. GPS data is acquired in real time as input. The acquired location information is sent to the server as output. Specifically, when Mr. C is walking in the facility's garden, the GPS tracker acquires his location information and periodically sends it to the server.

[0546] Step 10:

[0547] Location analysis

[0548] Server: The received location information is plotted on map data (e.g., Google Maps API) and movement patterns are analyzed. The acquired location information is used as input for analysis. Movement patterns and anomaly detection results are obtained as output. Specifically, the server displays Mr. C's location on a map and generates a warning that "he has left the facility."

[0549] Step 11:

[0550] Sending a warning message

[0551] Server: Sends the generated warning message to the user device. As input, it uses the anomaly detection results to create a warning message. As output, it sends the warning message to the user device. Specifically, the server detects an anomaly and sends a warning to the tablet device of the care staff.

[0552] Step 12:

[0553] Capturing Emotional Data

[0554] Device: When an elderly person stands in front of the mirror, the built-in camera and microphone capture their emotions using an emotion recognition engine (e.g., Microsoft Azure Face API). Emotion data is obtained from the elderly person's facial expressions and voice as input. Emotion recognition data is obtained as output. Specifically, when Mr. D looks at the mirror, the mirror displays the message "Analyzing emotions."

[0555] Step 13:

[0556] Sending and analyzing emotional data

[0557] Server: Analyzes the captured emotion data and identifies the emotional state of the elderly person. The analysis is performed using the transmitted emotion data as input. The output is the identified emotional state, and the result is stored in the database. Specifically, the server determines that Mr. D is "sad" based on his facial expression, and stores the result in the database.

[0558] Step 14:

[0559] Responding based on emotional state

[0560] Terminal: Based on the emotion analysis results sent from the server, an appropriate response is made. For example, if the emotion is negative, an encouraging voice response is generated. The emotion analysis results are received as input and an appropriate response is made. The output is a response and care for the elderly person. Specifically, the mirror responds by voice saying, "Mr. D, are you OK? Shall we talk?"

[0561] Step 15:

[0562] Emotional state notification

[0563] User terminal: The care staff checks the emotional state of the elderly person and takes the necessary measures for mental care. As input, it receives and displays the emotion analysis results. As output, the care staff checks the emotional state and is able to provide care. Specifically, the server notifies the staff of Mr. D's emotional state, and the staff quickly checks it.

[0564] (Application example 2)

[0565] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0566] While issues such as managing the health of elderly people, reducing feelings of loneliness, and reducing the risk of wandering are important in nursing homes and other facilities, there is also a need to quickly understand and respond to customers' physical and emotional states in brick-and-mortar stores. However, current systems have difficulty comprehensively resolving these issues, and are unable to adequately address the issues of improving customer satisfaction and employee work efficiency. Furthermore, there is a lack of measures to prevent customers from wandering within brick-and-mortar stores or to detect abnormalities using location information, leaving challenges for store operations.

[0567] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a notification means for analyzing physical condition information and notifying of abnormalities, an emotion analysis means for analyzing emotional states and notifying of the results, and a notification means for monitoring the location of customers in real time and notifying of any abnormalities. This makes it possible to manage the physical condition and emotional care of elderly people and prevent them from getting lost, even in stores, thereby improving customer satisfaction and employee work efficiency.

[0568] "Capture means" refers to a device used to acquire image data of a subject.

[0569] An "analysis means" is a device or system for analyzing acquired data to generate useful information.

[0570] "Storage and display means" refers to a mechanism for storing the generated information and displaying it as needed.

[0571] The "notification means" is a system for notifying a specified terminal of information when a specific condition is met.

[0572] "Emotion analysis means" is a technology for analyzing the emotional state of a subject from their facial expressions and voice.

[0573] "Location information acquisition means" refers to a device or technology for acquiring location information of an object.

[0574] "Voice input means" refers to a device or system for acquiring voice data.

[0575] "Speech analysis means" refers to a technique for analyzing acquired speech data and extracting meaning.

[0576] The "voice response generating means" is a technology for generating an appropriate voice response based on the analysis results.

[0577] The "abnormality detection means" is a system that analyzes the acquired data and detects any abnormal behavior or state.

[0578] This invention provides a system for managing customers' physical condition, recognizing their emotions, and preventing them from getting lost in a brick-and-mortar store. The specific system and its program configuration and processing will be described below.

[0579] System configuration

[0580] This system consists of the following main components:

[0581] 1. Smart AI mirror (device)

[0582] Photography means: Equipped with a built-in camera to photograph the faces of customers while they are being served.

[0583] Voice input means: A built-in microphone is provided to accept voice input from customers.

[0584] Storage and display means: A built-in display is provided to show the analysis results.

[0585] Emotion analysis means: Equipped with an emotion recognition engine to recognize emotions from customers' facial expressions and voices.

[0586] 2. Server

[0587] Analysis means: Has the function of receiving and analyzing image data and audio data sent from the terminal.

[0588] Notification method: Based on the analysis results, the system has the function of notifying staff if an abnormality is detected.

[0589] Location information acquisition method: Receives data from a badge equipped with a GPS tracker that acquires and analyzes customer location information in real time.

[0590] Anomaly detection means: Detects anomalies based on the acquired location information.

[0591] 3. User terminals (for staff)

[0592] Tablets and PCs that display various information sent from the server and can be checked by staff.

[0593] System action

[0594] 1. Health monitoring process

[0595] Terminal: When a customer stands in front of the smart AI mirror, the built-in camera automatically captures an image of the customer's face, and the captured image data is sent to the server in real time.

[0596] Server: The server analyzes the health information using the received image data. The analysis results include the customer's blood pressure, heart rate, and complexion. If an abnormality is detected, a notification method is activated and an alert is sent to staff.

[0597] User terminal: Staff check the customer's health information through the terminal and take necessary measures.

[0598] Examples:

[0599] Customer A stands in front of the smart AI mirror, and the mirror takes a picture of his face. The image data is sent to the server, which analyzes the image and determines that "Customer A is in good health." This result is also displayed on the staff member's tablet.

[0600] 2. Emotion Recognition Processing

[0601] Terminal: When customers talk to the smart AI mirror, the built-in microphone captures the voice, and this voice data is sent to the server.

[0602] Server: The server analyzes the voice data and runs natural language processing algorithms to understand what is being said. Based on this, the customer's emotional state is also analyzed.

[0603] Terminal: Based on the results of the sentiment analysis, necessary actions are taken. If the customer's sentiment is negative, an encouraging voice response is generated.

[0604] User terminal: Staff can check the emotional state of customers through the terminal and take the necessary measures to provide mental care.

[0605] Examples:

[0606] When Customer B says to the smart AI mirror, "I'm tired today," the mirror sends the voice message to the server. The server analyzes the voice data and determines that the customer is tired. This information is sent to the staff member's tablet. The staff member then plays a response such as "Would you like to take a break?" from the mirror.

[0607] 3. Preventing children from getting lost

[0608] Location information acquisition method: Customers are given a badge with a GPS tracker, which sends the customer's location information to the server at regular intervals.

[0609] Server: The server analyzes the received location information and detects any abnormal movements. If an abnormality is detected, it generates a warning message and notifies staff.

[0610] User terminal: If abnormal location movements are detected through the terminal, staff can respond quickly.

[0611] Examples:

[0612] If Customer C gets lost in the store, the GPS tracker will send his location information to the server, which will detect the abnormal location and send an alert to the staff, who will quickly rush to assist the customer.

[0613] Prompt Sentence Examples

[0614] Prompt: Suggest an application for a brick-and-mortar store that monitors customers' physical and emotional states and allows employees to respond quickly. Ideally, the application should have the following characteristics:

[0615] 1. Analyze customer health information in real time and notify staff if there are any abnormalities.

[0616] 2. Recognize the emotional state of the customer and notify staff when necessary.

[0617] 3. It has a function to monitor customer location information to prevent children from getting lost.

[0618] With the above configuration and processing, this system can manage the physical condition and emotional well-being of elderly people and prevent them from getting lost, even in brick-and-mortar stores, thereby improving customer satisfaction and employee work efficiency.

[0619] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0620] Step 1:

[0621] In order for the device to monitor the customer's physical condition, the Smart AI Mirror's camera captures image data of the customer's face. The captured image data is input and sent to the server. Specifically, the camera starts up, captures images at regular intervals, and sends the data to the server.

[0622] Step 2:

[0623] The server generates health information from the transmitted image data using analytical means. The input is the captured image data, and the output is the analyzed health information. The server uses an AI algorithm to extract health information such as blood pressure, heart rate, and complexion from the image data.

[0624] Step 3:

[0625] The server stores the generated health information in a database and sends it to the storage and display means for display. The input is the analyzed health information, and the output is the stored data and displayed health information. The server stores the information in the database and displays the contents on the terminal and the user terminal.

[0626] Step 4:

[0627] The smart AI mirror's microphone captures the customer's speech so that the device can accept the customer's voice input. The input is voice data, which is then sent to the server. The microphone detects the voice, collects data, and sends it to the server.

[0628] Step 5:

[0629] The server analyzes the voice content of the transmitted voice data using an analysis means to understand what the customer is saying. The input is the voice data, and the output is the analyzed voice content. The server uses a natural language processing algorithm to generate text from the voice and analyzes the content.

[0630] Step 6:

[0631] The server generates an appropriate voice response based on the analysis results. The input is the analyzed voice content and the output is the generated voice response. The server uses a response generation algorithm to create a text response and convert it to speech.

[0632] Step 7:

[0633] The terminal outputs the generated voice response to the customer. The input is the generated voice response and the output is the played back audio. The smart AI mirror uses a speaker to communicate the response to the customer.

[0634] Step 8:

[0635] The location information acquisition means provides customers with a badge with a GPS tracker, which transmits the customer's location information to the server at regular intervals. The input is location information, which is sent to the server. The tracker detects the location data and sends it to the server.

[0636] Step 9:

[0637] The server analyzes the received location information and detects anomalous movement. The input is location information and the output is anomalous movement patterns. The server uses a location analysis algorithm to plot the data on a map and identify anomalies.

[0638] Step 10:

[0639] The user terminal receives the anomaly detection notification from the server and notifies the staff that an anomaly has been detected. The input is the anomaly detection notification, and the output is the notified information. The staff terminal receives the alert and displays it.

[0640] Step 11:

[0641] The user's device provides staff with information such as the customer's current situation and location. The input is data from the server, and the output is information provided to the staff. Based on this information, the staff can respond quickly.

[0642] In this way, the system enables customer health management, emotion recognition, and loss prevention in physical stores.

[0643] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0644] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0645] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0646] [Second embodiment]

[0647] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0648] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0649] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0650] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0651] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0652] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0653] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0654] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0655] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0656] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0657] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0658] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0659] This invention is a system for managing the physical condition of elderly people in nursing homes, reducing feelings of loneliness, and reducing the risk of wandering. This system is realized using smart AI mirrors installed in each room. The specific configuration and program processing of this system are described below.

[0660] System configuration

[0661] This system consists of the following main components:

[0662] 1. Smart AI mirror (device)

[0663] It has a built-in camera to take pictures of the elderly person's face.

[0664] Built-in microphone for voice input.

[0665] Built-in display for showing analysis results.

[0666] 2. Server

[0667] It has the function of receiving and analyzing image data and audio data sent from the terminal.

[0668] It includes a database that generates and stores physical condition information.

[0669] The analysis results are notified to the terminal and the user terminal.

[0670] 3. User terminal (for care staff)

[0671] A tablet or PC that displays health information sent from the server and can be checked by care staff.

[0672] 4. GPS Tracker Insoles

[0673] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[0674] Program processing

[0675] The program for this system is implemented to achieve three main functions: monitoring the physical condition of the elderly, voice interaction, and location information analysis.

[0676] 1. Health monitoring process

[0677] Device:

[0678] When an elderly person stands in front of the mirror, the built-in camera automatically activates and captures an image of their face, which is then sent to a server in real time.

[0679] server:

[0680] The server processes the received image data using an analysis algorithm. As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database, and if necessary, an alert is sent to care staff if an abnormality is detected.

[0681] User device:

[0682] Care staff can check the health information of the elderly person through the user terminal and take necessary measures based on this information.

[0683] Examples:

[0684] Elderly person A stands in front of the smart AI mirror, which takes a picture of their face. The image data is sent to the server, which analyzes the image and determines that "A's health is good." This result is also displayed on the caregiver's tablet.

[0685] 2. Voice Dialogue Processing

[0686] Device:

[0687] When an elderly person speaks to the smart AI mirror, the built-in microphone captures their voice, and the voice data is sent to a server.

[0688] server:

[0689] The server analyzes the audio data and runs natural language processing algorithms to understand what is being said, and generates an appropriate response based on the results.

[0690] Device:

[0691] The response text sent from the server is converted into speech using speech synthesis technology and output to the elderly.

[0692] Examples:

[0693] When Elderly Person B says, "The weather is nice today," the Smart AI Mirror sends the speech to the server. The server analyzes the speech data and generates a response, "The weather is really nice. It would be nice to go for a walk," and outputs this response as voice.

[0694] 3. Location analysis processing

[0695] Location information acquisition method (shoe insoles with GPS tracker):

[0696] The location information of the elderly person is obtained at regular intervals and sent to the server.

[0697] server:

[0698] The server plots the received location information on map data, analyzes movement patterns, and generates a warning message if abnormal movement (e.g., movement into an unauthorized area) is detected.

[0699] User device:

[0700] Care staff will receive a warning message via the user's terminal, enabling them to respond quickly.

[0701] Examples:

[0702] If Elderly Person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his / her location information to the server, which will detect the abnormal location and send a warning to the care staff.

[0703] By combining the above configuration and processing, this system can manage the physical condition of elderly people, reduce feelings of loneliness, and reduce the risk of them wandering. It is an effective means of improving the physical and mental health and quality of life of facility users.

[0704] The processing flow will be explained below.

[0705] Health monitoring process

[0706] Step 1:

[0707] The device detects when an elderly person stands in front of the smart AI mirror, and the built-in camera automatically activates.

[0708] Step 2:

[0709] The device takes a picture of the elderly person's face and captures the image data in real time, then encodes the captured image data.

[0710] Step 3:

[0711] The terminal transmits the encoded image data to the server.

[0712] Step 4:

[0713] The server decodes the received image data and applies it to an analysis algorithm, which estimates blood pressure and heart rate based on facial color, facial expression, and whether the eyes are open or closed.

[0714] Step 5:

[0715] The server stores the physical condition information generated as a result of the analysis in a database, and then transmits this physical condition information to the terminal and the user terminal.

[0716] Step 6:

[0717] The user (care staff) uses the user terminal to check the health information of the elderly person. If any abnormalities are found, they respond promptly.

[0718] Voice interaction processing

[0719] Step 1:

[0720] The device is constantly in listening mode and waiting for voice input from the elderly person.

[0721] Step 2:

[0722] The device captures the voice of the elderly person and records it as audio data.

[0723] Step 3:

[0724] The device converts the captured voice data into text and sends it digitally to a server.

[0725] Step 4:

[0726] The server analyzes the received voice data using a natural language processing algorithm to understand what the elderly person is saying.

[0727] Step 5:

[0728] The server generates an appropriate response text based on the analysis results and sends this response text to the terminal.

[0729] Step 6:

[0730] The response text received by the terminal is converted into voice data using voice synthesis technology and output to the elderly person.

[0731] Processing location analysis

[0732] Step 1:

[0733] The location information acquisition means (shoe insoles with GPS trackers) acquires the elderly person's location information at regular intervals.

[0734] Step 2:

[0735] The location information acquired by the location information acquisition means is encoded and transmitted to a server in real time.

[0736] Step 3:

[0737] The server decodes the received location information and plots it in a map database.

[0738] Step 4:

[0739] The server analyzes movement patterns based on the plotted location information and detects abnormal movement (for example, movement into unauthorized areas).

[0740] Step 5:

[0741] If the server detects an abnormal movement pattern, it generates a warning message and sends this message to the user terminal.

[0742] Step 6:

[0743] The user (care staff) receives and checks the warning message via the user terminal and takes prompt action.

[0744] Example 1

[0745] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0746] In nursing homes for the elderly, managing the health of the elderly, reducing feelings of loneliness, and reducing the risk of wandering are important issues. Conventional systems do not provide comprehensive solutions to these issues, which increases the burden on care staff and makes it difficult to ensure the safety and health of the elderly. In addition, the high risk of wandering requires real-time monitoring of location information. Therefore, a system that integrates health monitoring, voice dialogue, and location information analysis for the elderly is needed.

[0747] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0748] In this invention, the server includes a camera means for acquiring image data, an analysis means for analyzing the acquired image data to generate health information, a database means for saving the generated health information, a display means for displaying the generated health information, a notification means for sending an alert when an abnormality is detected, a voice input means for receiving and analyzing voice input and generating a corresponding response, a natural language processing means for analyzing the voice data, a voice response generation means, a voice synthesis means for synthesizing the voice response, and a means for acquiring location information of the elderly person, detecting abnormal movements based on the location information, and sending a warning message as necessary. This makes it possible to appropriately manage the health of the elderly person, reduce feelings of loneliness, and reduce the risk of wandering.

[0749] "Camera means for capturing image data" refers to a photographic device used to capture images of the face and body of the elderly person.

[0750] "Analysis means for analyzing acquired image data to generate physical condition information" refers to algorithms and software for processing acquired image data and calculating physical condition information such as blood pressure, heart rate, and complexion of the elderly person.

[0751] The "database means for storing generated physical condition information" refers to a database system that stores generated physical condition information and can retrieve it as needed.

[0752] The "display means for displaying the generated physical condition information" refers to a display or screen for visually displaying the generated physical condition information.

[0753] "Notification means for sending an alert when an abnormality is detected" refers to a system that sends a warning message to care staff or appropriate personnel when an abnormality is detected in health information.

[0754] "Voice input means" refers to a microphone or voice recognition device that captures the voice spoken by the elderly person and collects the voice data.

[0755] "Natural language processing means for analyzing voice data" refers to natural language processing algorithms and software for analyzing acquired voice data and understanding its content.

[0756] "Voice response generator" refers to software or algorithms for generating appropriate responses based on analyzed voice data.

[0757] The "voice synthesis means for synthesizing a voice response" refers to a voice synthesis technology for outputting the generated response as voice.

[0758] "Location information acquisition means" refers to a location information acquisition device such as a GPS system used to determine the current location of an elderly person.

[0759] "Map data means for plotting received location information on map data" refers to a system for displaying acquired location information on a map.

[0760] "Anomaly detection means for detecting abnormal movements" refers to algorithms or software for analyzing acquired location information and detecting abnormal movements or behaviors.

[0761] "Warning means for sending a warning message when an abnormality is detected" refers to a system that sends a warning message to relevant parties when an abnormality in movement is detected.

[0762] MODE FOR CARRYING OUT THE INVENTION

[0763] This invention is a system aimed at managing the health of elderly people, reducing feelings of loneliness, and reducing the risk of wandering. This system is realized using a smart AI mirror installed in each room. The specific configuration and program processing of this system are described in detail below.

[0764] System configuration

[0765] This system consists of the following main components:

[0766] 1. Smart AI mirror (device)

[0767] It has a built-in camera to take pictures of the elderly person's face.

[0768] Built-in microphone for voice input.

[0769] Built-in display for showing analysis results.

[0770] 2. Server

[0771] It has the function of receiving and analyzing image data and audio data sent from the terminal.

[0772] It includes a database that generates and stores physical condition information.

[0773] The analysis results are notified to the terminal and the user terminal.

[0774] 3. User terminal (for care staff)

[0775] A tablet or PC that displays health information sent from the server and can be checked by care staff.

[0776] 4. GPS Tracker Insoles

[0777] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[0778] Program processing

[0779] The program for this system is implemented to achieve three main functions: monitoring the physical condition of the elderly, voice interaction, and location information analysis.

[0780] Health monitoring process

[0781] Device:

[0782] When an elderly person stands in front of the mirror, the built-in camera automatically activates and captures an image of their face, which is then sent to a server in real time.

[0783] server:

[0784] The server processes the received image data using an analysis algorithm (e.g., OpenCV or TensorFlow). As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database (e.g., MySQL), and alerts are sent to care staff as necessary.

[0785] User device:

[0786] Care staff can check the health information of the elderly person through the user terminal and take necessary measures based on this information.

[0787] Examples:

[0788] When elderly person A stands in front of the smart AI mirror and the mirror takes a picture of their face, the image data is sent to the server. The server analyzes the image and determines that "A's health is good." This result is also displayed on the care staff's tablet.

[0789] Voice interaction processing

[0790] Device:

[0791] When an elderly person speaks to the smart AI mirror, the built-in microphone captures their voice, and the voice data is sent to a server.

[0792] server:

[0793] The server runs a natural language processing algorithm (e.g., Google's Speech-to-Text API or OpenAI's GPT-3 model) to analyze the voice data and generates an appropriate response based on the results.

[0794] Device:

[0795] The response text sent from the server is converted into speech using speech synthesis technology (e.g., Google's Text-to-Speech API) and output to the elderly via the mirror.

[0796] Examples:

[0797] When Elderly Person B says, "The weather is nice today," the Smart AI Mirror sends the speech to the server. The server analyzes the speech data and generates a response, "The weather is really nice. It would be nice to go for a walk," and outputs this response as voice.

[0798] Processing location analysis

[0799] Location information acquisition method (shoe insoles with GPS tracker):

[0800] The location information of the elderly person is obtained at regular intervals and sent to the server.

[0801] server:

[0802] The server plots the received location information on map data (e.g., Google Maps API) and analyzes movement patterns. If abnormal movement (e.g., movement into an unauthorized area) is detected, a warning message is generated.

[0803] User device:

[0804] Care staff will receive a warning message via the user's terminal, enabling them to respond quickly.

[0805] Examples:

[0806] If Elderly Person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his / her location information to the server, which will detect the abnormality and send an alert to the care staff.

[0807] Prompt Sentence Examples

[0808] Example prompt 1 (health monitoring):

[0809] "Please extract health information from facial images of elderly people. Specifically, please generate information on three elements: blood pressure, heart rate, and complexion."

[0810] Prompt example 2 (voice dialogue):

[0811] "Generate an appropriate response when an older adult says, 'The weather is nice today.'"

[0812] Example prompt 3 (location analysis):

[0813] "Implement an algorithm that receives location information from seniors, plots it on map data, and detects abnormal movements."

[0814] This system, configured in this way, plays an important role in managing the health of the elderly, and also contributes to reducing feelings of loneliness and the risk of wandering.

[0815] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0816] Health monitoring process

[0817] Step 1: Capture and send a facial image

[0818] Terminal: When an elderly person stands in front of the smart AI mirror, the terminal activates the built-in camera to capture the elderly person's facial image, and the input data is the elderly person's facial image, which is then sent to the server in real time.

[0819] Specific operation: When elderly person A stands in front of the mirror, the device's camera automatically starts up, takes a picture of his face, encrypts the image, and sends it to the server.

[0820] Step 2: Analyzing the image data

[0821] Server: The server uses algorithms (e.g., OpenCV or TensorFlow) to analyze the received image data. The input data is a facial image, which is analyzed to generate physical condition information such as the elderly person's blood pressure, heart rate, and complexion.

[0822] Specific operation: The server uses a GPU to quickly analyze the image data and calculates that the blood pressure is 120 / 80 and the heart rate is 70 BPM.

[0823] Step 3: Save and notify health information

[0824] Server: The generated health information is stored in a database (e.g., MySQL). The input data is the generated health information, and if an abnormality is detected, an alert is sent to the care staff.

[0825] Specific operation: When the server detects an abnormal value (e.g., blood pressure 160 / 100), it immediately generates an alert and notifies the care staff's terminal.

[0826] Step 4: Check your health information

[0827] User: The care staff checks the health information of the elderly person through the user terminal. The input data is the generated health information, and the care staff takes necessary measures based on this.

[0828] Specific actions: Caregiver B uses a tablet device to check A's health information and immediately contacts the doctor.

[0829] Voice interaction processing

[0830] Step 1: Capture and send audio

[0831] Terminal: When the elderly person speaks to the smart AI mirror, the built-in microphone of the terminal captures the voice and transmits the voice data to the server in real time. The input data is an audio file.

[0832] Specific operation: When Elderly Person B says, "The weather is nice today," the device's microphone records the voice and sends it to the server.

[0833] Step 2: Analyzing the audio data

[0834] Server: The server uses natural language processing algorithms (e.g., Google's Speech-to-Text API or OpenAI's GPT-3 model) to analyze the voice data and understand what is being said. The input data is an audio file, which is converted into text and an appropriate response is generated.

[0835] What it does: The server converts the speech "The weather is nice today" into text and generates a response saying "The weather is really nice. It would be nice to go for a walk."

[0836] Step 3: Generate a voice response

[0837] Terminal: The response text sent from the server is converted into speech using speech synthesis technology (e.g., Google's Text-to-Speech API). The input data is the response text, which is then output as speech to the elderly.

[0838] Specific operation: The device outputs a synthesized voice saying, "What beautiful weather. It would be nice to go for a walk."

[0839] Processing location analysis

[0840] Step 1: Obtaining and sending location information

[0841] Location information acquisition means (shoe insoles with GPS trackers): The location information of the elderly is acquired at regular intervals (e.g., every minute) and sent to the server. The input data is location coordinates.

[0842] Specific operation: The GPS built into the insole of elderly person C's shoe captures location information every minute and sends it to the server.

[0843] Step 2: Analyze location information

[0844] Server: The server plots the received location information on map data (e.g., Google Maps API) and analyzes movement patterns. The input data is location information, and abnormal movements are detected.

[0845] Specific operation: The server checks the location of elderly person C, who has left the facility, on a map and detects any abnormalities.

[0846] Step 3: Notification of warning messages

[0847] User terminal: The nursing staff receives the warning message through the user terminal and responds promptly. The input data is the warning message.

[0848] Specific operation: A warning notification stating "Elderly person C has left the facility" appears on the smartphone of caregiver D.

[0849] (Application example 1)

[0850] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0851] In managing the physical and mental health of the elderly, there is a need for systems that allow elderly people at high risk of wandering to live within a safe range, and that allow caregivers and family members to monitor the elderly's location and health information in real time and respond quickly. However, existing systems lack the functionality to manage this information in an integrated manner and send appropriate alerts in real time when an abnormality occurs. It is also important to have a function that can reduce the elderly's sense of loneliness through voice dialogue.

[0852] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0853] In this invention, the server includes an imaging means for acquiring image data, an analysis means for analyzing the acquired image data to generate health information, a storage and display means for storing and displaying the generated health information, a location information acquisition means for acquiring location information, an abnormality detection means for detecting abnormal movements by comparing the acquired location information with a geographical fence, an alert notification means for sending a notification when an abnormality is detected, a voice input means, a voice analysis means, a voice response generation means, a user interface means for integrating the elderly person's health information and location information and enabling real-time confirmation, and a push notification means. This enables health management of the elderly person, monitoring of location information, and prompt response when an abnormality is detected.

[0854] "Photographing means for acquiring image data" refers to a device such as a camera for photographing an image of the face of an elderly person.

[0855] "Analysis means" refers to the algorithms and software that analyze the acquired image data and generate physical condition information such as blood pressure, heart rate, and complexion.

[0856] "Storage and display means" refers to a database that stores the generated physical condition information and a system for displaying that information on a display or the like.

[0857] "Location information acquisition means" refers to devices such as GPS sensors that acquire the current location of elderly people in real time.

[0858] "Anomaly detection means" refers to an algorithm or software that analyzes acquired location information and determines that an anomaly has occurred if the information exceeds a set safety range (geographical fence).

[0859] "Alert notification means" refers to push notification services and communication means for sending notifications to caregivers and family members when an abnormality is detected.

[0860] "Voice input means" refers to a device such as a microphone for capturing the elderly person's voice.

[0861] "Speech analysis means" refers to natural language processing algorithms or software that analyzes captured voice data and understands what is being said.

[0862] The "voice response generation means" refers to a voice synthesis technology for generating an appropriate response based on the analysis results and outputting the response as voice.

[0863] "User interface means" refers to an interface that displays the elderly person's physical condition information and location information in real time, allowing caregivers and family members to easily check them.

[0864] "Push notification means" refers to a system that sends notifications to caregivers' and family members' devices in real time when an abnormality is detected.

[0865] This invention provides a system for managing the physical condition of elderly people, tracking their location information, and reducing the risk of them wandering. The specific configuration and processing of this system will be described below.

[0866] System configuration

[0867] This system consists of the following main components:

[0868] 1. Terminal

[0869] These are mobile devices such as smartphones and smart glasses that are used as belongings for the elderly.

[0870] It has a built-in camera for taking facial images of elderly people.

[0871] It has a built-in microphone for accepting voice input.

[0872] It has a built-in GPS sensor to acquire location information.

[0873] 2. Server

[0874] It has the function of receiving and analyzing image data and audio data sent from the terminal.

[0875] It has a database that generates and stores health information and location information.

[0876] It has a system to generate alerts and notify you if an abnormality is detected.

[0877] 3. User Device

[0878] A tablet or PC that displays health information and location information sent from a server and can be checked by care staff and family members.

[0879] What the program does

[0880] 1. Health monitoring process

[0881] Device: When an elderly person stands in front of the camera, their facial image is automatically captured. This image data is sent to the server in real time.

[0882] Server: Analyzes image data using artificial intelligence analysis tools (e.g., TensorFlow, PyTorch, etc.) to generate health information such as blood pressure and heart rate. The generated health information is stored in a database. If an abnormality is detected, an alert notification is sent to the caregiver's device.

[0883] User device: Care staff can check the health information of elderly people via tablets or PCs and take necessary measures.

[0884] Example: An elderly person stands in front of the device and the camera captures their face. The image data is sent to a server, which analyzes it and determines that the person is in good health. This is then displayed on the caregiver's tablet.

[0885] 2. Voice Dialogue Processing

[0886] Terminal: When an elderly person speaks to the terminal, the microphone captures the voice and the voice data is sent to the server.

[0887] Server: Analyzes the voice data using natural language processing algorithms and generates an appropriate response.

[0888] Terminal: The response sent from the server is output as voice using speech synthesis technology.

[0889] Example: When an elderly person says, "The weather is nice today," the device sends the speech to the server. The server generates a response, "It's really nice weather. It would be nice to go for a walk," and the device outputs it as speech.

[0890] 3. Location analysis processing

[0891] Device: The device's GPS sensor acquires the elderly person's location information and sends it to the server.

[0892] Server: Executes anomaly detection measures that compare location information with geo-fences to detect abnormal movements. If an anomaly is detected, an alert notification measure is used to send a notification to caregivers and family members.

[0893] User device: Care staff and family members can receive notifications through the device, check the elderly person's location, and respond quickly.

[0894] Example: If an elderly person goes outside a set safe area, the GPS sensor sends their location to a server. The server detects the abnormality and sends a notification to the caregiver's smartphone that "the elderly person has gone outside the range."

[0895] Prompt Sentence Examples

[0896] 1. Obtaining location information

[0897] "Get the user's current location. Use the GPS sensor to send the latitude and longitude to the server in real time."

[0898] 2. Setting up geo-fences

[0899] It provides a GUI for configuring safety limits within the app and stores the user's settings in a database.

[0900] 3. Sending emergency alerts

[0901] "If out-of-range movement is detected, a push notification is sent to the registered caregiver's smartphone using Firebase Cloud Messaging."

[0902] 4. Check your health information

[0903] "We will create a UI that displays the user's health data received from the server and updates it in real time."

[0904] This allows the system to manage the health of elderly people and monitor their location information in an integrated manner, protecting their safety and reducing the burden on caregivers and family members.

[0905] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0906] Step 1:

[0907] Terminal: Acquisition of image data

[0908] The device's camera captures the elderly person's facial image. The elderly person's face must be in front of the camera as input, and the captured facial image data is obtained as output.

[0909] Step 2:

[0910] Terminal: Sending image data

[0911] The acquired facial image data is sent to the server in real time. The input is the facial image data captured in step 1, and the output is the data successfully sent to the server.

[0912] Step 3:

[0913] Server: Generates health information

[0914] The server analyzes the received facial image data using an artificial intelligence algorithm (e.g., TensorFlow, PyTorch) to generate physical condition information such as blood pressure, heart rate, and complexion. The input is the facial image data sent from the device, and the output is the generated physical condition information.

[0915] Step 4:

[0916] Server: Storage of health information

[0917] The generated physical condition information is saved in a database. The input is the physical condition information generated in step 3, and the output is the physical condition information saved in the database.

[0918] Step 5:

[0919] Server: Anomaly detection

[0920] The server monitors the stored health information and generates an alert if an abnormality is detected. Pre-set criteria are used for abnormality detection. The input is the health information stored in the database, and the output is an alert notification if an abnormality is detected.

[0921] Step 6:

[0922] Server: Alert Notification

[0923] If an abnormality is detected, the server sends an alert notification to the caregiver or family member's device. The input is the alert generated in step 5, and the output is the notification sent to the caregiver or family member's device.

[0924] Step 7:

[0925] Device: Audio data capture

[0926] The microphone on the device captures the elderly person's voice, and the input is the elderly person's voice, and the output is the captured voice data.

[0927] Step 8:

[0928] Terminal: Sending audio data

[0929] Send the captured audio data to the server in real time. The input is the audio data captured in step 7, and the output is the data successfully sent to the server.

[0930] Step 9:

[0931] Server: Analysis of voice data

[0932] The server analyzes the received voice data, uses natural language processing algorithms (e.g., BERT, GPT-3, etc.) to understand the speech and generate an appropriate response. The input is the voice data sent from the device, and the output is the generated response text.

[0933] Step 10:

[0934] Server: Sending a voice response

[0935] The server sends the generated response text to the terminal. The input is the response text generated in step 9, and the output is the response text sent to the terminal.

[0936] Step 11:

[0937] Terminal: Voice response output

[0938] The device converts the received response text into speech and outputs it to the elderly. It uses speech synthesis technology (e.g., a TTS engine). The input is the response text sent from the server, and the output is the speech output to the elderly.

[0939] Step 12:

[0940] Device: Location information acquisition

[0941] The GPS sensor on the device periodically acquires the elderly person's location information. The input is the current GPS data, and the output is the acquired location information.

[0942] Step 13:

[0943] Device: Sending location information

[0944] Send the acquired location information to the server. The input is the location information acquired in step 12, and the output is the data successfully sent to the server.

[0945] Step 14:

[0946] Server: Location analysis

[0947] The server analyzes the received location information and detects anomalies if the location information exceeds the set geographical fence. The input is the location information sent from the device, and the output is an alert notification if an anomaly is detected.

[0948] Step 15:

[0949] Server: Alert notification of abnormal location

[0950] If an abnormality is detected, the server sends an alert notification to the caregiver or family device. The input is the alert generated in step 14, and the output is the notification sent to the caregiver or family device.

[0951] Step 16:

[0952] User: Check health and location information

[0953] Caregivers and family members can check the health and location information through the user's device. The input is the health and location information sent from the server, and the output is the information displayed on the user's device.

[0954] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0955] This invention is a system for managing the physical condition of elderly people in nursing homes, reducing feelings of loneliness, reducing the risk of wandering, and providing emotional care through emotion recognition. This system is realized using smart AI mirrors installed in each room, and by combining it with an emotion recognition engine, it is also possible to care for the elderly's emotional state. The specific configuration of this system and the program processing are described below.

[0956] System configuration

[0957] This system consists of the following main components:

[0958] 1. Smart AI mirror (device)

[0959] It has a built-in camera to take pictures of the elderly person's face.

[0960] Built-in microphone for voice input.

[0961] Built-in display for showing analysis results.

[0962] It has a built-in emotion recognition engine to recognize emotions.

[0963] 2. Server

[0964] It has the ability to receive and analyze image data, audio data, and emotional data sent from the terminal.

[0965] It includes a database that generates and stores physical condition information, voice analysis results, and emotion analysis results.

[0966] The analysis results are notified to the terminal and the user terminal.

[0967] 3. User terminal (for care staff)

[0968] Tablets and PCs that display various information sent from the server and can be checked by care staff.

[0969] 4. GPS Tracker Insoles

[0970] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[0971] Program processing

[0972] The program of this system is implemented mainly to realize the functions of monitoring the physical condition of the elderly, voice dialogue, location information analysis, and emotion recognition.

[0973] 1. Health monitoring process

[0974] Device:

[0975] When an elderly person stands in front of the mirror, the built-in camera automatically activates to capture an image of their face, which is then sent to a server in real time.

[0976] server:

[0977] The server processes the received image data using an analysis algorithm. As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database, and if necessary, an alert is sent to care staff if an abnormality is detected.

[0978] User device:

[0979] Care staff can check the health information of the elderly person through the user terminal and take necessary measures based on this information.

[0980] Examples:

[0981] Elderly person A stands in front of the smart AI mirror, which takes a picture of their face. The image data is sent to the server, which analyzes the image and determines that "A's health is good." This result is also displayed on the caregiver's tablet.

[0982] 2. Voice Dialogue Processing

[0983] Device:

[0984] When an elderly person speaks to the smart AI mirror, the built-in microphone captures their voice, and the voice data is sent to a server.

[0985] server:

[0986] The server analyzes the audio data and runs natural language processing algorithms to understand what is being said, and generates an appropriate response based on the results.

[0987] Device:

[0988] The response text sent from the server is converted into speech using speech synthesis technology and output to the elderly.

[0989] Examples:

[0990] When Elderly Person B says, "The weather is nice today," the Smart AI Mirror sends the speech to the server. The server analyzes the speech data and generates a response, "The weather is really nice. It would be nice to go for a walk," and outputs this response as voice.

[0991] 3. Location analysis processing

[0992] Location information acquisition method (shoe insoles with GPS tracker):

[0993] The location information of the elderly person is obtained at regular intervals and sent to the server.

[0994] server:

[0995] The server plots the received location information on map data, analyzes movement patterns, and generates a warning message if abnormal movement (e.g., movement into an unauthorized area) is detected.

[0996] User device:

[0997] Care staff will receive a warning message via the user's terminal, enabling them to respond quickly.

[0998] Examples:

[0999] If Elderly Person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his / her location information to the server, which will detect the abnormal location and send a warning to the care staff.

[1000] 4. Emotion Recognition Processing

[1001] Device:

[1002] When an elderly person stands in front of the mirror, the built-in camera and microphone use an emotion recognition engine to capture emotions from the elderly person's facial expressions and voice.

[1003] server:

[1004] The server analyzes the captured emotional data to identify the elderly person's emotional state, and the analysis results are stored in a database.

[1005] Device:

[1006] Based on the emotion analysis results sent from the server, an appropriate response is made. For example, if the emotion is negative, an encouraging voice response is generated.

[1007] User device:

[1008] Care staff can also check the emotional state of the elderly person through the user device and take the necessary measures to provide mental care.

[1009] Examples:

[1010] When elderly person D faces the mirror, the emotion recognition engine determines that D is "sad" from his facial expression. This information is sent to the server and displayed on the care staff's tablet. The staff plays a voice response from the mirror saying, "Are you OK, Mr. D? Shall we talk?"

[1011] By combining the above configuration and processing, this system can manage the physical condition of elderly people, reduce feelings of loneliness, reduce the risk of wandering, and provide psychological care through emotion recognition, thereby significantly improving the physical and mental health and quality of life of facility users.

[1012] The processing flow will be explained below.

[1013] Health monitoring process

[1014] Step 1:

[1015] The device detects when an elderly person stands in front of the smart AI mirror, and the built-in camera automatically activates.

[1016] Step 2:

[1017] The device takes a picture of the elderly person's face and acquires image data in real time.

[1018] Step 3:

[1019] The terminal encodes the acquired image data and transmits it to the server in real time.

[1020] Step 4:

[1021] The server decodes the received image data and applies it to an analysis algorithm, which estimates blood pressure and heart rate based on facial color, facial expression, and whether the eyes are open or closed.

[1022] Step 5:

[1023] The server stores the physical condition information generated as a result of the analysis in a database, and then transmits this physical condition information to the terminal and the user terminal.

[1024] Step 6:

[1025] The user (care staff) uses the user terminal to check the health information of the elderly person. If any abnormalities are found, they respond promptly.

[1026] Voice interaction processing

[1027] Step 1:

[1028] The device is constantly in listening mode and waiting for voice input from the elderly person.

[1029] Step 2:

[1030] The device captures the voice of the elderly person and records it as audio data.

[1031] Step 3:

[1032] The device converts the captured voice data into text and sends it digitally to a server.

[1033] Step 4:

[1034] The server analyzes the received voice data using a natural language processing algorithm to understand what the elderly person is saying.

[1035] Step 5:

[1036] The server generates an appropriate response text based on the analysis results and sends this response text to the terminal.

[1037] Step 6:

[1038] The response text received by the terminal is converted into voice data using voice synthesis technology and output to the elderly person.

[1039] Processing location analysis

[1040] Step 1:

[1041] The location information acquisition means (shoe insoles with GPS trackers) acquires the elderly person's location information at regular intervals.

[1042] Step 2:

[1043] The location information acquired by the location information acquisition means is encoded and transmitted to a server in real time.

[1044] Step 3:

[1045] The server decodes the received location information and plots it in a map database.

[1046] Step 4:

[1047] The server analyzes movement patterns based on the plotted location information and detects abnormal movement (for example, movement into unauthorized areas).

[1048] Step 5:

[1049] If the server detects an abnormal movement pattern, it generates a warning message and sends this message to the user terminal.

[1050] Step 6:

[1051] The user (care staff) receives and checks the warning message via the user terminal and takes prompt action.

[1052] Emotion recognition processing

[1053] Step 1:

[1054] The device activates the built-in camera to capture the elderly person's facial expressions, and also activates the built-in microphone to capture their voice.

[1055] Step 2:

[1056] The facial expression data and voice data captured by the device are encoded in real time and sent to the server.

[1057] Step 3:

[1058] The server decodes the received facial expression and voice data and analyzes it using an emotion recognition engine, which identifies the elderly person's emotional state (joy, anger, sadness, happiness, etc.).

[1059] Step 4:

[1060] The server stores the emotion analysis results in a database and, if necessary, transmits the analysis results to the terminal and the user terminal.

[1061] Step 5:

[1062] The device generates an appropriate response based on the emotion analysis results. For example, if an elderly person is sad, it generates an encouraging or comforting voice response.

[1063] Step 6:

[1064] The response generated by the device is converted into voice data using voice synthesis technology and output to the elderly. This information is also notified to the user (care staff) so that appropriate action can be taken.

[1065] To give a concrete example:

[1066] When elderly person D faces the mirror, the emotion recognition engine determines from D's facial expression that he is "sad." This information is sent to the server and displayed on the care staff's tablet. The staff then plays a voice response from the mirror saying, "Are you OK, Mr. D? Shall we talk?"

[1067] By combining the above configuration and processing, this system can manage the physical condition of elderly people, reduce feelings of loneliness, reduce the risk of wandering, and provide psychological care through emotion recognition, thereby significantly improving the physical and mental health and quality of life of facility users.

[1068] Example 2

[1069] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1070] The purpose of this invention is to manage the physical condition of elderly people, reduce feelings of loneliness, reduce the risk of wandering, and provide psychological care through emotion recognition. Conventional systems have individual functions, but few have been integrated into a single system, making it difficult to efficiently link multiple systems. Furthermore, conventional technology has limitations in meeting the demand for real-time anomaly detection and rapid response.

[1071] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an imaging means for acquiring image data, an analysis means for analyzing the acquired image data and generating physical condition information, a storage and display means for saving and displaying the generated physical condition information, an emotion recognition means for acquiring and analyzing emotion data and identifying the emotional state, and an alert notification means for sending an alert when an abnormality is detected. This makes it possible to perform physical condition management and mental care for the elderly in an integrated and real-time manner.

[1072] The "photographing means for acquiring image data" is a device for photographing the face and physical condition of an elderly person and acquiring the image data.

[1073] "Analysis means for analyzing acquired image data and generating physical condition information" refers to a device or program for analyzing and generating physical condition information such as blood pressure, heart rate, and complexion of an elderly person using acquired image data.

[1074] The "storage and display means for storing and displaying the generated physical condition information" refers to a device or program for storing the physical condition information generated by the analysis in a database and displaying it as needed.

[1075] "Emotion recognition means for acquiring and analyzing emotional data to identify an emotional state" refers to a device or program for capturing emotions from the facial expressions and voice of an elderly person and analyzing the emotional data to identify the emotional state.

[1076] An "alert notification means for sending an alert when an abnormality is detected" is a device or program for sending a notification to care staff and other relevant parties when an abnormality is detected based on the elderly person's physical condition, location information, etc.

[1077] The "voice input means" is a device for capturing the voice of the elderly person and obtaining the voice data.

[1078] The "voice analysis means" is a device or program for analyzing acquired voice data and understanding its content.

[1079] The "voice response generating means" is a device or program for generating an appropriate response based on the analyzed voice data.

[1080] The "location information acquisition means" is a device for acquiring the current location of the elderly person in real time and transmitting that location information.

[1081] The "location information analysis means" is a device or program for analyzing the acquired location information and plotting it on a map to detect abnormal movements.

[1082] The "abnormality detection means" is a device or program for detecting abnormal situations based on the physical condition and location information of the elderly person.

[1083] This invention is a system that aims to manage the physical condition of elderly people, reduce feelings of loneliness, reduce the risk of wandering, and provide mental care through emotion recognition. This system is realized using smart AI mirrors installed in each room, and by combining it with an emotion recognition engine, it is also possible to care for the mental state of the elderly. The specific configuration and program processing of this system are explained below.

[1084] System configuration

[1085] This system consists of the following main components:

[1086] 1. Smart AI mirror (device)

[1087] It has a built-in camera to take pictures of the elderly person's face.

[1088] Built-in microphone for voice input.

[1089] Built-in display for showing analysis results.

[1090] It has a built-in emotion recognition engine to recognize emotions.

[1091] 2. Server

[1092] It has the ability to receive and analyze image data, audio data, and emotional data sent from the terminal.

[1093] It includes a database that generates and stores physical condition information, voice analysis results, and emotion analysis results.

[1094] The analysis results are notified to the terminal and the user terminal.

[1095] 3. User terminal (for care staff)

[1096] Tablets and PCs that display various information sent from the server and can be checked by care staff.

[1097] 4. GPS Tracker Insoles

[1098] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[1099] Specific processing explanation

[1100] Elderly health monitoring

[1101] Device:

[1102] When an elderly person stands in front of the smart AI mirror, the built-in camera captures an image of their face, which is then sent to a server in real time.

[1103] server:

[1104] The server processes the received image data using an analysis algorithm (e.g., OpenCV or TensorFlow). As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database, and if an abnormality is detected, an alert is sent to the care staff.

[1105] User device:

[1106] Care staff can check the health information of the elderly person through the user terminal and take necessary measures.

[1107] Examples:

[1108] When elderly person A stands in front of the smart AI mirror, the built-in camera takes a picture of their face. The image data is analyzed and the mirror determines that "A's health is good." This result can also be viewed on the caregiver's tablet.

[1109] Voice dialogue with the elderly

[1110] Device:

[1111] When an elderly person talks to the smart AI mirror, the built-in microphone captures the voice and sends it to the server.

[1112] server:

[1113] The server analyzes the voice data, runs natural language processing algorithms (e.g., Google Cloud Speech-to-Text), and generates an appropriate response based on the analysis results.

[1114] Device:

[1115] The generated response text is converted into speech using speech synthesis technology (e.g., Google Cloud Text-to-Speech) and output to the elderly.

[1116] Examples:

[1117] When Elderly Person B says, "The weather is nice today," the AI ​​Mirror sends the voice data to the server. The server analyzes the voice and generates a response saying, "The weather is really nice. It would be nice to go for a walk," and outputs it as voice.

[1118] Analysis of elderly people's location information

[1119] Location information acquisition method (shoe insoles with GPS tracker):

[1120] The location information of the elderly person is obtained at regular intervals and sent to the server.

[1121] server:

[1122] The server plots the received location information on map data (e.g., Google Maps API) and analyzes movement patterns. If abnormal movement is detected, a warning message is generated.

[1123] User device:

[1124] Care staff will receive a warning message and be able to respond quickly.

[1125] Examples:

[1126] If elderly person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his location information to the server, and the server will detect the abnormality and send an alert to the care staff.

[1127] Emotion recognition in the elderly

[1128] Device:

[1129] When an elderly person stands in front of the mirror, the built-in camera and microphone capture their emotions using an emotion recognition engine (e.g., Microsoft Azure Face API).

[1130] server:

[1131] The captured emotional data is analyzed to identify the emotional state of the elderly person, and the analysis results are stored in a database.

[1132] Device:

[1133] Based on the emotion analysis results sent from the server, an appropriate response is made. For example, if the emotion is negative, an encouraging voice response is generated.

[1134] User device:

[1135] Care staff check the emotional state of the elderly and take the necessary steps to provide mental care.

[1136] Examples:

[1137] When elderly person D looks at the mirror, the emotion recognition engine determines that he is "sad." This information is sent to the server and displayed on the care staff's device. The staff then plays a voice response from the mirror saying, "Mr. D, are you okay? Shall we talk?"

[1138] By combining the above configuration and processing, this system can comprehensively manage the physical condition of the elderly, reduce feelings of loneliness, reduce the risk of wandering, and provide psychological care through emotion recognition, which is expected to significantly improve the physical and mental health and quality of life of the elderly.

[1139] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1140] Step 1:

[1141] Elderly Image Capture

[1142] Terminal: When an elderly person stands in front of the smart AI mirror, the built-in camera automatically captures an image of their face. The image of the elderly person's face reflected in the camera is captured as input, and this is obtained as image data as output. Specifically, when Elderly Person A looks at the mirror with a smile, the display will show "Analyzing your physical condition."

[1143] Step 2:

[1144] Sending image data

[1145] Terminal: Sends the captured image data to the server. The acquired image data is used as input and sent securely using encryption technology (e.g., SSL / TLS). The image data is then transferred to the server as output. Specifically, the image data of elderly person A is sent from the mirror to the server within a few seconds.

[1146] Step 3:

[1147] Image data analysis

[1148] Server: Processes the received image data using an analysis algorithm (e.g., OpenCV or TensorFlow). Using the transmitted image data as input, the server analyzes it to generate health information such as the elderly person's blood pressure, heart rate, and complexion. This health information is obtained as output. Specifically, the server analyzes the image and determines that elderly person A's heart rate is "normal."

[1149] Step 4:

[1150] Storage and notification of health information

[1151] Server: The generated health information is saved in a database. If an abnormality is detected, an alert is sent to the care staff. The health information from the analysis is used as input, and the process of saving it to the database and sending an alert is executed. The output is the saving of the health information and the sending of an alert. Specifically, the server determines that elderly person A's blood pressure is high, and an alert is sent to the care staff's tablet.

[1152] Step 5:

[1153] Displaying health information

[1154] User device: The care staff checks the health information of the elderly person through the user device. The health information sent from the server is used as input and displayed. The health information can be checked by the care staff as output. Specifically, the care staff checks the health information of elderly person A on the tablet device and takes the necessary measures.

[1155] Step 6:

[1156] Capturing voice input

[1157] Terminal: When an elderly person speaks to the smart AI mirror, the built-in microphone captures the voice and sends this voice data to the server. The elderly person's voice is captured as input and obtained as voice data. The voice data is sent to the server as output. In concrete terms, when Elderly Person B speaks to the mirror saying, "How's the weather?", the mirror displays "Analyzing voice."

[1158] Step 7:

[1159] Analysis of audio data

[1160] Server: Analyzes the voice data and runs a natural language processing algorithm (e.g., Google Cloud Speech-to-Text). It uses the transmitted voice data as input and performs the analysis. As output, it generates corresponding text data and obtains an appropriate response text. Specifically, the server analyzes Person B's voice and generates the response "It's sunny today."

[1161] Step 8:

[1162] Response generation and speech output

[1163] Terminal: The response text sent from the server is converted into speech using speech synthesis technology (e.g., Google Cloud Text-to-Speech) and output to the elderly person. The response text is used as input to generate speech. A voice notification is output to the elderly person. Specifically, the mirror notifies Mr. B by voice, saying, "It's sunny today."

[1164] Step 9:

[1165] Obtaining location information

[1166] Location information acquisition means (shoe insoles with GPS trackers): The location information of the elderly person is acquired at regular intervals and sent to the server. GPS data is acquired in real time as input. The acquired location information is sent to the server as output. Specifically, when Mr. C is walking in the facility's garden, the GPS tracker acquires his location information and periodically sends it to the server.

[1167] Step 10:

[1168] Location analysis

[1169] Server: The received location information is plotted on map data (e.g., Google Maps API) and movement patterns are analyzed. The acquired location information is used as input for analysis. Movement patterns and anomaly detection results are obtained as output. Specifically, the server displays Mr. C's location on a map and generates a warning that "he has left the facility."

[1170] Step 11:

[1171] Sending a warning message

[1172] Server: Sends the generated warning message to the user device. As input, it uses the anomaly detection results to create a warning message. As output, it sends the warning message to the user device. Specifically, the server detects an anomaly and sends a warning to the tablet device of the care staff.

[1173] Step 12:

[1174] Capturing Emotional Data

[1175] Device: When an elderly person stands in front of the mirror, the built-in camera and microphone capture their emotions using an emotion recognition engine (e.g., Microsoft Azure Face API). Emotion data is obtained from the elderly person's facial expressions and voice as input. Emotion recognition data is obtained as output. Specifically, when Mr. D looks at the mirror, the mirror displays the message "Analyzing emotions."

[1176] Step 13:

[1177] Sending and analyzing emotional data

[1178] Server: Analyzes the captured emotion data and identifies the emotional state of the elderly person. The analysis is performed using the transmitted emotion data as input. The output is the identified emotional state, and the result is stored in the database. Specifically, the server determines that Mr. D is "sad" based on his facial expression, and stores the result in the database.

[1179] Step 14:

[1180] Responding based on emotional state

[1181] Terminal: Based on the emotion analysis results sent from the server, an appropriate response is made. For example, if the emotion is negative, an encouraging voice response is generated. The emotion analysis results are received as input and an appropriate response is made. The output is a response and care for the elderly person. Specifically, the mirror responds by voice saying, "Mr. D, are you OK? Shall we talk?"

[1182] Step 15:

[1183] Emotional state notification

[1184] User terminal: The care staff checks the emotional state of the elderly person and takes the necessary measures for mental care. As input, it receives and displays the emotion analysis results. As output, the care staff checks the emotional state and is able to provide care. Specifically, the server notifies the staff of Mr. D's emotional state, and the staff quickly checks it.

[1185] (Application example 2)

[1186] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1187] While issues such as managing the health of elderly people, reducing feelings of loneliness, and reducing the risk of wandering are important in nursing homes and other facilities, there is also a need to quickly understand and respond to customers' physical and emotional states in brick-and-mortar stores. However, current systems have difficulty comprehensively resolving these issues, and are unable to adequately address the issues of improving customer satisfaction and employee work efficiency. Furthermore, there is a lack of measures to prevent customers from wandering within brick-and-mortar stores or to detect abnormalities using location information, leaving challenges for store operations.

[1188] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a notification means for analyzing physical condition information and notifying of abnormalities, an emotion analysis means for analyzing emotional states and notifying of the results, and a notification means for monitoring the location of customers in real time and notifying of any abnormalities. This makes it possible to manage the physical condition and emotional care of elderly people and prevent them from getting lost, even in stores, thereby improving customer satisfaction and employee work efficiency.

[1189] "Capture means" refers to a device used to acquire image data of a subject.

[1190] An "analysis means" is a device or system for analyzing acquired data to generate useful information.

[1191] "Storage and display means" refers to a mechanism for storing the generated information and displaying it as needed.

[1192] The "notification means" is a system for notifying a specified terminal of information when a specific condition is met.

[1193] "Emotion analysis means" is a technology for analyzing the emotional state of a subject from their facial expressions and voice.

[1194] "Location information acquisition means" refers to a device or technology for acquiring location information of an object.

[1195] "Voice input means" refers to a device or system for acquiring voice data.

[1196] "Speech analysis means" refers to a technique for analyzing acquired speech data and extracting meaning.

[1197] The "voice response generating means" is a technology for generating an appropriate voice response based on the analysis results.

[1198] The "abnormality detection means" is a system that analyzes the acquired data and detects any abnormal behavior or state.

[1199] This invention provides a system for managing customers' physical condition, recognizing their emotions, and preventing them from getting lost in a brick-and-mortar store. The specific system and its program configuration and processing will be described below.

[1200] System configuration

[1201] This system consists of the following main components:

[1202] 1. Smart AI mirror (device)

[1203] Photography means: Equipped with a built-in camera to photograph the faces of customers while they are being served.

[1204] Voice input means: A built-in microphone is provided to accept voice input from customers.

[1205] Storage and display means: A built-in display is provided to show the analysis results.

[1206] Emotion analysis means: Equipped with an emotion recognition engine to recognize emotions from customers' facial expressions and voices.

[1207] 2. Server

[1208] Analysis means: Has the function of receiving and analyzing image data and audio data sent from the terminal.

[1209] Notification method: Based on the analysis results, the system has the function of notifying staff if an abnormality is detected.

[1210] Location information acquisition method: Receives data from a badge equipped with a GPS tracker that acquires and analyzes customer location information in real time.

[1211] Anomaly detection means: Detects anomalies based on the acquired location information.

[1212] 3. User terminals (for staff)

[1213] Tablets and PCs that display various information sent from the server and can be checked by staff.

[1214] System action

[1215] 1. Health monitoring process

[1216] Terminal: When a customer stands in front of the smart AI mirror, the built-in camera automatically captures an image of the customer's face, and the captured image data is sent to the server in real time.

[1217] Server: The server analyzes the health information using the received image data. The analysis results include the customer's blood pressure, heart rate, and complexion. If an abnormality is detected, a notification method is activated and an alert is sent to staff.

[1218] User terminal: Staff check the customer's health information through the terminal and take necessary measures.

[1219] Examples:

[1220] Customer A stands in front of the smart AI mirror, and the mirror takes a picture of his face. The image data is sent to the server, which analyzes the image and determines that "Customer A is in good health." This result is also displayed on the staff member's tablet.

[1221] 2. Emotion Recognition Processing

[1222] Terminal: When customers talk to the smart AI mirror, the built-in microphone captures the voice, and this voice data is sent to the server.

[1223] Server: The server analyzes the voice data and runs natural language processing algorithms to understand what is being said. Based on this, the customer's emotional state is also analyzed.

[1224] Terminal: Based on the results of the sentiment analysis, necessary actions are taken. If the customer's sentiment is negative, an encouraging voice response is generated.

[1225] User terminal: Staff can check the emotional state of customers through the terminal and take the necessary measures to provide mental care.

[1226] Examples:

[1227] When Customer B says to the smart AI mirror, "I'm tired today," the mirror sends the voice message to the server. The server analyzes the voice data and determines that the customer is tired. This information is sent to the staff member's tablet. The staff member then plays a response such as "Would you like to take a break?" from the mirror.

[1228] 3. Preventing children from getting lost

[1229] Location information acquisition method: Customers are given a badge with a GPS tracker, which sends the customer's location information to the server at regular intervals.

[1230] Server: The server analyzes the received location information and detects any abnormal movements. If an abnormality is detected, it generates a warning message and notifies staff.

[1231] User terminal: If abnormal location movements are detected through the terminal, staff can respond quickly.

[1232] Examples:

[1233] If Customer C gets lost in the store, the GPS tracker will send his location information to the server, which will detect the abnormal location and send an alert to the staff, who will quickly rush to assist the customer.

[1234] Prompt Sentence Examples

[1235] Prompt: Suggest an application for a brick-and-mortar store that monitors customers' physical and emotional states and allows employees to respond quickly. Ideally, the application should have the following characteristics:

[1236] 1. Analyze customer health information in real time and notify staff if there are any abnormalities.

[1237] 2. Recognize the emotional state of the customer and notify staff when necessary.

[1238] 3. It has a function to monitor customer location information to prevent children from getting lost.

[1239] With the above configuration and processing, this system can manage the physical condition and emotional well-being of elderly people and prevent them from getting lost, even in brick-and-mortar stores, thereby improving customer satisfaction and employee work efficiency.

[1240] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1241] Step 1:

[1242] In order for the device to monitor the customer's physical condition, the Smart AI Mirror's camera captures image data of the customer's face. The captured image data is input and sent to the server. Specifically, the camera starts up, captures images at regular intervals, and sends the data to the server.

[1243] Step 2:

[1244] The server generates health information from the transmitted image data using analytical means. The input is the captured image data, and the output is the analyzed health information. The server uses an AI algorithm to extract health information such as blood pressure, heart rate, and complexion from the image data.

[1245] Step 3:

[1246] The server stores the generated health information in a database and sends it to the storage and display means for display. The input is the analyzed health information, and the output is the stored data and displayed health information. The server stores the information in the database and displays the contents on the terminal and the user terminal.

[1247] Step 4:

[1248] The smart AI mirror's microphone captures the customer's speech so that the device can accept the customer's voice input. The input is voice data, which is then sent to the server. The microphone detects the voice, collects data, and sends it to the server.

[1249] Step 5:

[1250] The server analyzes the voice content of the transmitted voice data using an analysis means to understand what the customer is saying. The input is the voice data, and the output is the analyzed voice content. The server uses a natural language processing algorithm to generate text from the voice and analyzes the content.

[1251] Step 6:

[1252] The server generates an appropriate voice response based on the analysis results. The input is the analyzed voice content and the output is the generated voice response. The server uses a response generation algorithm to create a text response and convert it to speech.

[1253] Step 7:

[1254] The terminal outputs the generated voice response to the customer. The input is the generated voice response and the output is the played back audio. The smart AI mirror uses a speaker to communicate the response to the customer.

[1255] Step 8:

[1256] The location information acquisition means provides customers with a badge with a GPS tracker, which transmits the customer's location information to the server at regular intervals. The input is location information, which is sent to the server. The tracker detects the location data and sends it to the server.

[1257] Step 9:

[1258] The server analyzes the received location information and detects anomalous movement. The input is location information and the output is anomalous movement patterns. The server uses a location analysis algorithm to plot the data on a map and identify anomalies.

[1259] Step 10:

[1260] The user terminal receives the anomaly detection notification from the server and notifies the staff that an anomaly has been detected. The input is the anomaly detection notification, and the output is the notified information. The staff terminal receives the alert and displays it.

[1261] Step 11:

[1262] The user's device provides staff with information such as the customer's current situation and location. The input is data from the server, and the output is information provided to the staff. Based on this information, the staff can respond quickly.

[1263] In this way, the system enables customer health management, emotion recognition, and loss prevention in physical stores.

[1264] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1265] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1266] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1267] [Third embodiment]

[1268] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1269] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1270] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1271] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1272] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1273] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1274] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1275] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1276] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1277] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1278] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1279] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1280] This invention is a system for managing the physical condition of elderly people in nursing homes, reducing feelings of loneliness, and reducing the risk of wandering. This system is realized using smart AI mirrors installed in each room. The specific configuration and program processing of this system are described below.

[1281] System configuration

[1282] This system consists of the following main components:

[1283] 1. Smart AI mirror (device)

[1284] It has a built-in camera to take pictures of the elderly person's face.

[1285] Built-in microphone for voice input.

[1286] Built-in display for showing analysis results.

[1287] 2. Server

[1288] It has the function of receiving and analyzing image data and audio data sent from the terminal.

[1289] It includes a database that generates and stores physical condition information.

[1290] The analysis results are notified to the terminal and the user terminal.

[1291] 3. User terminal (for care staff)

[1292] A tablet or PC that displays health information sent from the server and can be checked by care staff.

[1293] 4. GPS Tracker Insoles

[1294] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[1295] Program processing

[1296] The program for this system is implemented to achieve three main functions: monitoring the physical condition of the elderly, voice interaction, and location information analysis.

[1297] 1. Health monitoring process

[1298] Device:

[1299] When an elderly person stands in front of the mirror, the built-in camera automatically activates and captures an image of their face, which is then sent to a server in real time.

[1300] server:

[1301] The server processes the received image data using an analysis algorithm. As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database, and if necessary, an alert is sent to care staff if an abnormality is detected.

[1302] User device:

[1303] Care staff can check the health information of the elderly person through the user terminal and take necessary measures based on this information.

[1304] Examples:

[1305] Elderly person A stands in front of the smart AI mirror, which takes a picture of their face. The image data is sent to the server, which analyzes the image and determines that "A's health is good." This result is also displayed on the caregiver's tablet.

[1306] 2. Voice Dialogue Processing

[1307] Device:

[1308] When an elderly person speaks to the smart AI mirror, the built-in microphone captures their voice, and the voice data is sent to a server.

[1309] server:

[1310] The server analyzes the audio data and runs natural language processing algorithms to understand what is being said, and generates an appropriate response based on the results.

[1311] Device:

[1312] The response text sent from the server is converted into speech using speech synthesis technology and output to the elderly.

[1313] Examples:

[1314] When Elderly Person B says, "The weather is nice today," the Smart AI Mirror sends the speech to the server. The server analyzes the speech data and generates a response, "The weather is really nice. It would be nice to go for a walk," and outputs this response as voice.

[1315] 3. Location analysis processing

[1316] Location information acquisition method (shoe insoles with GPS tracker):

[1317] The location information of the elderly person is obtained at regular intervals and sent to the server.

[1318] server:

[1319] The server plots the received location information on map data, analyzes movement patterns, and generates a warning message if abnormal movement (e.g., movement into an unauthorized area) is detected.

[1320] User device:

[1321] Care staff will receive a warning message via the user's terminal, enabling them to respond quickly.

[1322] Examples:

[1323] If Elderly Person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his / her location information to the server, which will detect the abnormal location and send a warning to the care staff.

[1324] By combining the above configuration and processing, this system can manage the physical condition of elderly people, reduce feelings of loneliness, and reduce the risk of them wandering. It is an effective means of improving the physical and mental health and quality of life of facility users.

[1325] The processing flow will be explained below.

[1326] Health monitoring process

[1327] Step 1:

[1328] The device detects when an elderly person stands in front of the smart AI mirror, and the built-in camera automatically activates.

[1329] Step 2:

[1330] The device takes a picture of the elderly person's face and captures the image data in real time, then encodes the captured image data.

[1331] Step 3:

[1332] The terminal transmits the encoded image data to the server.

[1333] Step 4:

[1334] The server decodes the received image data and applies it to an analysis algorithm, which estimates blood pressure and heart rate based on facial color, facial expression, and whether the eyes are open or closed.

[1335] Step 5:

[1336] The server stores the physical condition information generated as a result of the analysis in a database, and then transmits this physical condition information to the terminal and the user terminal.

[1337] Step 6:

[1338] The user (care staff) uses the user terminal to check the health information of the elderly person. If any abnormalities are found, they respond promptly.

[1339] Voice interaction processing

[1340] Step 1:

[1341] The device is constantly in listening mode and waiting for voice input from the elderly person.

[1342] Step 2:

[1343] The device captures the voice of the elderly person and records it as audio data.

[1344] Step 3:

[1345] The device converts the captured voice data into text and sends it digitally to a server.

[1346] Step 4:

[1347] The server analyzes the received voice data using a natural language processing algorithm to understand what the elderly person is saying.

[1348] Step 5:

[1349] The server generates an appropriate response text based on the analysis results and sends this response text to the terminal.

[1350] Step 6:

[1351] The response text received by the terminal is converted into voice data using voice synthesis technology and output to the elderly person.

[1352] Processing location analysis

[1353] Step 1:

[1354] The location information acquisition means (shoe insoles with GPS trackers) acquires the elderly person's location information at regular intervals.

[1355] Step 2:

[1356] The location information acquired by the location information acquisition means is encoded and transmitted to a server in real time.

[1357] Step 3:

[1358] The server decodes the received location information and plots it in a map database.

[1359] Step 4:

[1360] The server analyzes movement patterns based on the plotted location information and detects abnormal movement (for example, movement into unauthorized areas).

[1361] Step 5:

[1362] If the server detects an abnormal movement pattern, it generates a warning message and sends this message to the user terminal.

[1363] Step 6:

[1364] The user (care staff) receives and checks the warning message via the user terminal and takes prompt action.

[1365] Example 1

[1366] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1367] In nursing homes for the elderly, managing the health of the elderly, reducing feelings of loneliness, and reducing the risk of wandering are important issues. Conventional systems do not provide comprehensive solutions to these issues, which increases the burden on care staff and makes it difficult to ensure the safety and health of the elderly. In addition, the high risk of wandering requires real-time monitoring of location information. Therefore, a system that integrates health monitoring, voice dialogue, and location information analysis for the elderly is needed.

[1368] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1369] In this invention, the server includes a camera means for acquiring image data, an analysis means for analyzing the acquired image data to generate health information, a database means for saving the generated health information, a display means for displaying the generated health information, a notification means for sending an alert when an abnormality is detected, a voice input means for receiving and analyzing voice input and generating a corresponding response, a natural language processing means for analyzing the voice data, a voice response generation means, a voice synthesis means for synthesizing the voice response, and a means for acquiring location information of the elderly person, detecting abnormal movements based on the location information, and sending a warning message as necessary. This makes it possible to appropriately manage the health of the elderly person, reduce feelings of loneliness, and reduce the risk of wandering.

[1370] "Camera means for capturing image data" refers to a photographic device used to capture images of the face and body of the elderly person.

[1371] "Analysis means for analyzing acquired image data to generate physical condition information" refers to algorithms and software for processing acquired image data and calculating physical condition information such as blood pressure, heart rate, and complexion of the elderly person.

[1372] The "database means for storing generated physical condition information" refers to a database system that stores generated physical condition information and can retrieve it as needed.

[1373] The "display means for displaying the generated physical condition information" refers to a display or screen for visually displaying the generated physical condition information.

[1374] "Notification means for sending an alert when an abnormality is detected" refers to a system that sends a warning message to care staff or appropriate personnel when an abnormality is detected in health information.

[1375] "Voice input means" refers to a microphone or voice recognition device that captures the voice spoken by the elderly person and collects the voice data.

[1376] "Natural language processing means for analyzing voice data" refers to natural language processing algorithms and software for analyzing acquired voice data and understanding its content.

[1377] "Voice response generator" refers to software or algorithms for generating appropriate responses based on analyzed voice data.

[1378] The "voice synthesis means for synthesizing a voice response" refers to a voice synthesis technology for outputting the generated response as voice.

[1379] "Location information acquisition means" refers to a location information acquisition device such as a GPS system used to determine the current location of an elderly person.

[1380] "Map data means for plotting received location information on map data" refers to a system for displaying acquired location information on a map.

[1381] "Anomaly detection means for detecting abnormal movements" refers to algorithms or software for analyzing acquired location information and detecting abnormal movements or behaviors.

[1382] "Warning means for sending a warning message when an abnormality is detected" refers to a system that sends a warning message to relevant parties when an abnormality in movement is detected.

[1383] MODE FOR CARRYING OUT THE INVENTION

[1384] This invention is a system aimed at managing the health of elderly people, reducing feelings of loneliness, and reducing the risk of wandering. This system is realized using a smart AI mirror installed in each room. The specific configuration and program processing of this system are described in detail below.

[1385] System configuration

[1386] This system consists of the following main components:

[1387] 1. Smart AI mirror (device)

[1388] It has a built-in camera to take pictures of the elderly person's face.

[1389] Built-in microphone for voice input.

[1390] Built-in display for showing analysis results.

[1391] 2. Server

[1392] It has the function of receiving and analyzing image data and audio data sent from the terminal.

[1393] It includes a database that generates and stores physical condition information.

[1394] The analysis results are notified to the terminal and the user terminal.

[1395] 3. User terminal (for care staff)

[1396] A tablet or PC that displays health information sent from the server and can be checked by care staff.

[1397] 4. GPS Tracker Insoles

[1398] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[1399] Program processing

[1400] The program for this system is implemented to achieve three main functions: monitoring the physical condition of the elderly, voice interaction, and location information analysis.

[1401] Health monitoring process

[1402] Device:

[1403] When an elderly person stands in front of the mirror, the built-in camera automatically activates and captures an image of their face, which is then sent to a server in real time.

[1404] server:

[1405] The server processes the received image data using an analysis algorithm (e.g., OpenCV or TensorFlow). As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database (e.g., MySQL), and alerts are sent to care staff as necessary.

[1406] User device:

[1407] Care staff can check the health information of the elderly person through the user terminal and take necessary measures based on this information.

[1408] Examples:

[1409] When elderly person A stands in front of the smart AI mirror and the mirror takes a picture of their face, the image data is sent to the server. The server analyzes the image and determines that "A's health is good." This result is also displayed on the care staff's tablet.

[1410] Voice interaction processing

[1411] Device:

[1412] When an elderly person speaks to the smart AI mirror, the built-in microphone captures their voice, and the voice data is sent to a server.

[1413] server:

[1414] The server runs a natural language processing algorithm (e.g., Google's Speech-to-Text API or OpenAI's GPT-3 model) to analyze the voice data and generates an appropriate response based on the results.

[1415] Device:

[1416] The response text sent from the server is converted into speech using speech synthesis technology (e.g., Google's Text-to-Speech API) and output to the elderly via the mirror.

[1417] Examples:

[1418] When Elderly Person B says, "The weather is nice today," the Smart AI Mirror sends the speech to the server. The server analyzes the speech data and generates a response, "The weather is really nice. It would be nice to go for a walk," and outputs this response as voice.

[1419] Processing location analysis

[1420] Location information acquisition method (shoe insoles with GPS tracker):

[1421] The location information of the elderly person is obtained at regular intervals and sent to the server.

[1422] server:

[1423] The server plots the received location information on map data (e.g., Google Maps API) and analyzes movement patterns. If abnormal movement (e.g., movement into an unauthorized area) is detected, a warning message is generated.

[1424] User device:

[1425] Care staff will receive a warning message via the user's terminal, enabling them to respond quickly.

[1426] Examples:

[1427] If Elderly Person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his / her location information to the server, which will detect the abnormality and send an alert to the care staff.

[1428] Prompt Sentence Examples

[1429] Example prompt 1 (health monitoring):

[1430] "Please extract health information from facial images of elderly people. Specifically, please generate information on three elements: blood pressure, heart rate, and complexion."

[1431] Prompt example 2 (voice dialogue):

[1432] "Generate an appropriate response when an older adult says, 'The weather is nice today.'"

[1433] Example prompt 3 (location analysis):

[1434] "Implement an algorithm that receives location information from seniors, plots it on map data, and detects abnormal movements."

[1435] This system, configured in this way, plays an important role in managing the health of the elderly, and also contributes to reducing feelings of loneliness and the risk of wandering.

[1436] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1437] Health monitoring process

[1438] Step 1: Capture and send a facial image

[1439] Terminal: When an elderly person stands in front of the smart AI mirror, the terminal activates the built-in camera to capture the elderly person's facial image, and the input data is the elderly person's facial image, which is then sent to the server in real time.

[1440] Specific operation: When elderly person A stands in front of the mirror, the device's camera automatically starts up, takes a picture of his face, encrypts the image, and sends it to the server.

[1441] Step 2: Analyzing the image data

[1442] Server: The server uses algorithms (e.g., OpenCV or TensorFlow) to analyze the received image data. The input data is a facial image, which is analyzed to generate physical condition information such as the elderly person's blood pressure, heart rate, and complexion.

[1443] Specific operation: The server uses a GPU to quickly analyze the image data and calculates that the blood pressure is 120 / 80 and the heart rate is 70 BPM.

[1444] Step 3: Save and notify health information

[1445] Server: The generated health information is stored in a database (e.g., MySQL). The input data is the generated health information, and if an abnormality is detected, an alert is sent to the care staff.

[1446] Specific operation: When the server detects an abnormal value (e.g., blood pressure 160 / 100), it immediately generates an alert and notifies the care staff's terminal.

[1447] Step 4: Check your health information

[1448] User: The care staff checks the health information of the elderly person through the user terminal. The input data is the generated health information, and the care staff takes necessary measures based on this.

[1449] Specific actions: Caregiver B uses a tablet device to check A's health information and immediately contacts the doctor.

[1450] Voice interaction processing

[1451] Step 1: Capture and send audio

[1452] Terminal: When the elderly person speaks to the smart AI mirror, the built-in microphone of the terminal captures the voice and transmits the voice data to the server in real time. The input data is an audio file.

[1453] Specific operation: When Elderly Person B says, "The weather is nice today," the device's microphone records the voice and sends it to the server.

[1454] Step 2: Analyzing the audio data

[1455] Server: The server uses natural language processing algorithms (e.g., Google's Speech-to-Text API or OpenAI's GPT-3 model) to analyze the voice data and understand what is being said. The input data is an audio file, which is converted into text and an appropriate response is generated.

[1456] What it does: The server converts the speech "The weather is nice today" into text and generates a response saying "The weather is really nice. It would be nice to go for a walk."

[1457] Step 3: Generate a voice response

[1458] Terminal: The response text sent from the server is converted into speech using speech synthesis technology (e.g., Google's Text-to-Speech API). The input data is the response text, which is then output as speech to the elderly.

[1459] Specific operation: The device outputs a synthesized voice saying, "What beautiful weather. It would be nice to go for a walk."

[1460] Processing location analysis

[1461] Step 1: Obtaining and sending location information

[1462] Location information acquisition means (shoe insoles with GPS trackers): The location information of the elderly is acquired at regular intervals (e.g., every minute) and sent to the server. The input data is location coordinates.

[1463] Specific operation: The GPS built into the insole of elderly person C's shoe captures location information every minute and sends it to the server.

[1464] Step 2: Analyze location information

[1465] Server: The server plots the received location information on map data (e.g., Google Maps API) and analyzes movement patterns. The input data is location information, and abnormal movements are detected.

[1466] Specific operation: The server checks the location of elderly person C, who has left the facility, on a map and detects any abnormalities.

[1467] Step 3: Notification of warning messages

[1468] User terminal: The nursing staff receives the warning message through the user terminal and responds promptly. The input data is the warning message.

[1469] Specific operation: A warning notification stating "Elderly person C has left the facility" appears on the smartphone of caregiver D.

[1470] (Application example 1)

[1471] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1472] In managing the physical and mental health of the elderly, there is a need for systems that allow elderly people at high risk of wandering to live within a safe range, and that allow caregivers and family members to monitor the elderly's location and health information in real time and respond quickly. However, existing systems lack the functionality to manage this information in an integrated manner and send appropriate alerts in real time when an abnormality occurs. It is also important to have a function that can reduce the elderly's sense of loneliness through voice dialogue.

[1473] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1474] In this invention, the server includes an imaging means for acquiring image data, an analysis means for analyzing the acquired image data to generate health information, a storage and display means for storing and displaying the generated health information, a location information acquisition means for acquiring location information, an abnormality detection means for detecting abnormal movements by comparing the acquired location information with a geographical fence, an alert notification means for sending a notification when an abnormality is detected, a voice input means, a voice analysis means, a voice response generation means, a user interface means for integrating the elderly person's health information and location information and enabling real-time confirmation, and a push notification means. This enables health management of the elderly person, monitoring of location information, and prompt response when an abnormality is detected.

[1475] "Photographing means for acquiring image data" refers to a device such as a camera for photographing an image of the face of an elderly person.

[1476] "Analysis means" refers to the algorithms and software that analyze the acquired image data and generate physical condition information such as blood pressure, heart rate, and complexion.

[1477] "Storage and display means" refers to a database that stores the generated physical condition information and a system for displaying that information on a display or the like.

[1478] "Location information acquisition means" refers to devices such as GPS sensors that acquire the current location of elderly people in real time.

[1479] "Anomaly detection means" refers to an algorithm or software that analyzes acquired location information and determines that an anomaly has occurred if the information exceeds a set safety range (geographical fence).

[1480] "Alert notification means" refers to push notification services and communication means for sending notifications to caregivers and family members when an abnormality is detected.

[1481] "Voice input means" refers to a device such as a microphone for capturing the elderly person's voice.

[1482] "Speech analysis means" refers to natural language processing algorithms or software that analyzes captured voice data and understands what is being said.

[1483] The "voice response generation means" refers to a voice synthesis technology for generating an appropriate response based on the analysis results and outputting the response as voice.

[1484] "User interface means" refers to an interface that displays the elderly person's physical condition information and location information in real time, allowing caregivers and family members to easily check them.

[1485] "Push notification means" refers to a system that sends notifications to caregivers' and family members' devices in real time when an abnormality is detected.

[1486] This invention provides a system for managing the physical condition of elderly people, tracking their location information, and reducing the risk of them wandering. The specific configuration and processing of this system will be described below.

[1487] System configuration

[1488] This system consists of the following main components:

[1489] 1. Terminal

[1490] These are mobile devices such as smartphones and smart glasses that are used as belongings for the elderly.

[1491] It has a built-in camera for taking facial images of elderly people.

[1492] It has a built-in microphone for accepting voice input.

[1493] It has a built-in GPS sensor to acquire location information.

[1494] 2. Server

[1495] It has the function of receiving and analyzing image data and audio data sent from the terminal.

[1496] It has a database that generates and stores health information and location information.

[1497] It has a system to generate alerts and notify you if an abnormality is detected.

[1498] 3. User Device

[1499] A tablet or PC that displays health information and location information sent from a server and can be checked by care staff and family members.

[1500] What the program does

[1501] 1. Health monitoring process

[1502] Device: When an elderly person stands in front of the camera, their facial image is automatically captured. This image data is sent to the server in real time.

[1503] Server: Analyzes image data using artificial intelligence analysis tools (e.g., TensorFlow, PyTorch, etc.) to generate health information such as blood pressure and heart rate. The generated health information is stored in a database. If an abnormality is detected, an alert notification is sent to the caregiver's device.

[1504] User device: Care staff can check the health information of elderly people via tablets or PCs and take necessary measures.

[1505] Example: An elderly person stands in front of the device and the camera captures their face. The image data is sent to a server, which analyzes it and determines that the person is in good health. This is then displayed on the caregiver's tablet.

[1506] 2. Voice Dialogue Processing

[1507] Terminal: When an elderly person speaks to the terminal, the microphone captures the voice and the voice data is sent to the server.

[1508] Server: Analyzes the voice data using natural language processing algorithms and generates an appropriate response.

[1509] Terminal: The response sent from the server is output as voice using speech synthesis technology.

[1510] Example: When an elderly person says, "The weather is nice today," the device sends the speech to the server. The server generates a response, "It's really nice weather. It would be nice to go for a walk," and the device outputs it as speech.

[1511] 3. Location analysis processing

[1512] Device: The device's GPS sensor acquires the elderly person's location information and sends it to the server.

[1513] Server: Executes anomaly detection measures that compare location information with geo-fences to detect abnormal movements. If an anomaly is detected, an alert notification measure is used to send a notification to caregivers and family members.

[1514] User device: Care staff and family members can receive notifications through the device, check the elderly person's location, and respond quickly.

[1515] Example: If an elderly person goes outside a set safe area, the GPS sensor sends their location to a server. The server detects the abnormality and sends a notification to the caregiver's smartphone that "the elderly person has gone outside the range."

[1516] Prompt Sentence Examples

[1517] 1. Obtaining location information

[1518] "Get the user's current location. Use the GPS sensor to send the latitude and longitude to the server in real time."

[1519] 2. Setting up geo-fences

[1520] It provides a GUI for configuring safety limits within the app and stores the user's settings in a database.

[1521] 3. Sending emergency alerts

[1522] "If out-of-range movement is detected, a push notification is sent to the registered caregiver's smartphone using Firebase Cloud Messaging."

[1523] 4. Check your health information

[1524] "We will create a UI that displays the user's health data received from the server and updates it in real time."

[1525] This allows the system to manage the health of elderly people and monitor their location information in an integrated manner, protecting their safety and reducing the burden on caregivers and family members.

[1526] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1527] Step 1:

[1528] Terminal: Acquisition of image data

[1529] The device's camera captures the elderly person's facial image. The elderly person's face must be in front of the camera as input, and the captured facial image data is obtained as output.

[1530] Step 2:

[1531] Terminal: Sending image data

[1532] The acquired facial image data is sent to the server in real time. The input is the facial image data captured in step 1, and the output is the data successfully sent to the server.

[1533] Step 3:

[1534] Server: Generates health information

[1535] The server analyzes the received facial image data using an artificial intelligence algorithm (e.g., TensorFlow, PyTorch) to generate physical condition information such as blood pressure, heart rate, and complexion. The input is the facial image data sent from the device, and the output is the generated physical condition information.

[1536] Step 4:

[1537] Server: Storage of health information

[1538] The generated physical condition information is saved in a database. The input is the physical condition information generated in step 3, and the output is the physical condition information saved in the database.

[1539] Step 5:

[1540] Server: Anomaly detection

[1541] The server monitors the stored health information and generates an alert if an abnormality is detected. Pre-set criteria are used for abnormality detection. The input is the health information stored in the database, and the output is an alert notification if an abnormality is detected.

[1542] Step 6:

[1543] Server: Alert Notification

[1544] If an abnormality is detected, the server sends an alert notification to the caregiver or family member's device. The input is the alert generated in step 5, and the output is the notification sent to the caregiver or family member's device.

[1545] Step 7:

[1546] Device: Audio data capture

[1547] The microphone on the device captures the elderly person's voice, and the input is the elderly person's voice, and the output is the captured voice data.

[1548] Step 8:

[1549] Terminal: Sending audio data

[1550] Send the captured audio data to the server in real time. The input is the audio data captured in step 7, and the output is the data successfully sent to the server.

[1551] Step 9:

[1552] Server: Analysis of voice data

[1553] The server analyzes the received voice data, uses natural language processing algorithms (e.g., BERT, GPT-3, etc.) to understand the speech and generate an appropriate response. The input is the voice data sent from the device, and the output is the generated response text.

[1554] Step 10:

[1555] Server: Sending a voice response

[1556] The server sends the generated response text to the terminal. The input is the response text generated in step 9, and the output is the response text sent to the terminal.

[1557] Step 11:

[1558] Terminal: Voice response output

[1559] The device converts the received response text into speech and outputs it to the elderly. It uses speech synthesis technology (e.g., a TTS engine). The input is the response text sent from the server, and the output is the speech output to the elderly.

[1560] Step 12:

[1561] Device: Location information acquisition

[1562] The GPS sensor on the device periodically acquires the elderly person's location information. The input is the current GPS data, and the output is the acquired location information.

[1563] Step 13:

[1564] Device: Sending location information

[1565] Send the acquired location information to the server. The input is the location information acquired in step 12, and the output is the data successfully sent to the server.

[1566] Step 14:

[1567] Server: Location analysis

[1568] The server analyzes the received location information and detects anomalies if the location information exceeds the set geographical fence. The input is the location information sent from the device, and the output is an alert notification if an anomaly is detected.

[1569] Step 15:

[1570] Server: Alert notification of abnormal location

[1571] If an abnormality is detected, the server sends an alert notification to the caregiver or family device. The input is the alert generated in step 14, and the output is the notification sent to the caregiver or family device.

[1572] Step 16:

[1573] User: Check health and location information

[1574] Caregivers and family members can check the health and location information through the user's device. The input is the health and location information sent from the server, and the output is the information displayed on the user's device.

[1575] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1576] This invention is a system for managing the physical condition of elderly people in nursing homes, reducing feelings of loneliness, reducing the risk of wandering, and providing emotional care through emotion recognition. This system is realized using smart AI mirrors installed in each room, and by combining it with an emotion recognition engine, it is also possible to care for the elderly's emotional state. The specific configuration of this system and the program processing are described below.

[1577] System configuration

[1578] This system consists of the following main components:

[1579] 1. Smart AI mirror (device)

[1580] It has a built-in camera to take pictures of the elderly person's face.

[1581] Built-in microphone for voice input.

[1582] Built-in display for showing analysis results.

[1583] It has a built-in emotion recognition engine to recognize emotions.

[1584] 2. Server

[1585] It has the ability to receive and analyze image data, audio data, and emotional data sent from the terminal.

[1586] It includes a database that generates and stores physical condition information, voice analysis results, and emotion analysis results.

[1587] The analysis results are notified to the terminal and the user terminal.

[1588] 3. User terminal (for care staff)

[1589] Tablets and PCs that display various information sent from the server and can be checked by care staff.

[1590] 4. GPS Tracker Insoles

[1591] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[1592] Program processing

[1593] The program of this system is implemented mainly to realize the functions of monitoring the physical condition of the elderly, voice dialogue, location information analysis, and emotion recognition.

[1594] 1. Health monitoring process

[1595] Device:

[1596] When an elderly person stands in front of the mirror, the built-in camera automatically activates to capture an image of their face, which is then sent to a server in real time.

[1597] server:

[1598] The server processes the received image data using an analysis algorithm. As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database, and if necessary, an alert is sent to care staff if an abnormality is detected.

[1599] User device:

[1600] Care staff can check the health information of the elderly person through the user terminal and take necessary measures based on this information.

[1601] Examples:

[1602] Elderly person A stands in front of the smart AI mirror, which takes a picture of their face. The image data is sent to the server, which analyzes the image and determines that "A's health is good." This result is also displayed on the caregiver's tablet.

[1603] 2. Voice Dialogue Processing

[1604] Device:

[1605] When an elderly person speaks to the smart AI mirror, the built-in microphone captures their voice, and the voice data is sent to a server.

[1606] server:

[1607] The server analyzes the audio data and runs natural language processing algorithms to understand what is being said, and generates an appropriate response based on the results.

[1608] Device:

[1609] The response text sent from the server is converted into speech using speech synthesis technology and output to the elderly.

[1610] Examples:

[1611] When Elderly Person B says, "The weather is nice today," the Smart AI Mirror sends the speech to the server. The server analyzes the speech data and generates a response, "The weather is really nice. It would be nice to go for a walk," and outputs this response as voice.

[1612] 3. Location analysis processing

[1613] Location information acquisition method (shoe insoles with GPS tracker):

[1614] The location information of the elderly person is obtained at regular intervals and sent to the server.

[1615] server:

[1616] The server plots the received location information on map data, analyzes movement patterns, and generates a warning message if abnormal movement (e.g., movement into an unauthorized area) is detected.

[1617] User device:

[1618] Care staff will receive a warning message via the user's terminal, enabling them to respond quickly.

[1619] Examples:

[1620] If Elderly Person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his / her location information to the server, which will detect the abnormal location and send a warning to the care staff.

[1621] 4. Emotion Recognition Processing

[1622] Device:

[1623] When an elderly person stands in front of the mirror, the built-in camera and microphone use an emotion recognition engine to capture emotions from the elderly person's facial expressions and voice.

[1624] server:

[1625] The server analyzes the captured emotional data to identify the elderly person's emotional state, and the analysis results are stored in a database.

[1626] Device:

[1627] Based on the emotion analysis results sent from the server, an appropriate response is made. For example, if the emotion is negative, an encouraging voice response is generated.

[1628] User device:

[1629] Care staff can also check the emotional state of the elderly person through the user device and take the necessary measures to provide mental care.

[1630] Examples:

[1631] When elderly person D faces the mirror, the emotion recognition engine determines that D is "sad" from his facial expression. This information is sent to the server and displayed on the care staff's tablet. The staff plays a voice response from the mirror saying, "Are you OK, Mr. D? Shall we talk?"

[1632] By combining the above configuration and processing, this system can manage the physical condition of elderly people, reduce feelings of loneliness, reduce the risk of wandering, and provide psychological care through emotion recognition, thereby significantly improving the physical and mental health and quality of life of facility users.

[1633] The processing flow will be explained below.

[1634] Health monitoring process

[1635] Step 1:

[1636] The device detects when an elderly person stands in front of the smart AI mirror, and the built-in camera automatically activates.

[1637] Step 2:

[1638] The device takes a picture of the elderly person's face and acquires image data in real time.

[1639] Step 3:

[1640] The terminal encodes the acquired image data and transmits it to the server in real time.

[1641] Step 4:

[1642] The server decodes the received image data and applies it to an analysis algorithm, which estimates blood pressure and heart rate based on facial color, facial expression, and whether the eyes are open or closed.

[1643] Step 5:

[1644] The server stores the physical condition information generated as a result of the analysis in a database, and then transmits this physical condition information to the terminal and the user terminal.

[1645] Step 6:

[1646] The user (care staff) uses the user terminal to check the health information of the elderly person. If any abnormalities are found, they respond promptly.

[1647] Voice interaction processing

[1648] Step 1:

[1649] The device is constantly in listening mode and waiting for voice input from the elderly person.

[1650] Step 2:

[1651] The device captures the voice of the elderly person and records it as audio data.

[1652] Step 3:

[1653] The device converts the captured voice data into text and sends it digitally to a server.

[1654] Step 4:

[1655] The server analyzes the received voice data using a natural language processing algorithm to understand what the elderly person is saying.

[1656] Step 5:

[1657] The server generates an appropriate response text based on the analysis results and sends this response text to the terminal.

[1658] Step 6:

[1659] The response text received by the terminal is converted into voice data using voice synthesis technology and output to the elderly person.

[1660] Processing location analysis

[1661] Step 1:

[1662] The location information acquisition means (shoe insoles with GPS trackers) acquires the elderly person's location information at regular intervals.

[1663] Step 2:

[1664] The location information acquired by the location information acquisition means is encoded and transmitted to a server in real time.

[1665] Step 3:

[1666] The server decodes the received location information and plots it in a map database.

[1667] Step 4:

[1668] The server analyzes movement patterns based on the plotted location information and detects abnormal movement (for example, movement into unauthorized areas).

[1669] Step 5:

[1670] If the server detects an abnormal movement pattern, it generates a warning message and sends this message to the user terminal.

[1671] Step 6:

[1672] The user (care staff) receives and checks the warning message via the user terminal and takes prompt action.

[1673] Emotion recognition processing

[1674] Step 1:

[1675] The device activates the built-in camera to capture the elderly person's facial expressions, and also activates the built-in microphone to capture their voice.

[1676] Step 2:

[1677] The facial expression data and voice data captured by the device are encoded in real time and sent to the server.

[1678] Step 3:

[1679] The server decodes the received facial expression and voice data and analyzes it using an emotion recognition engine, which identifies the elderly person's emotional state (joy, anger, sadness, happiness, etc.).

[1680] Step 4:

[1681] The server stores the emotion analysis results in a database and, if necessary, transmits the analysis results to the terminal and the user terminal.

[1682] Step 5:

[1683] The device generates an appropriate response based on the emotion analysis results. For example, if an elderly person is sad, it generates an encouraging or comforting voice response.

[1684] Step 6:

[1685] The response generated by the device is converted into voice data using voice synthesis technology and output to the elderly. This information is also notified to the user (care staff) so that appropriate action can be taken.

[1686] To give a concrete example:

[1687] When elderly person D faces the mirror, the emotion recognition engine determines from D's facial expression that he is "sad." This information is sent to the server and displayed on the care staff's tablet. The staff then plays a voice response from the mirror saying, "Are you OK, Mr. D? Shall we talk?"

[1688] By combining the above configuration and processing, this system can manage the physical condition of elderly people, reduce feelings of loneliness, reduce the risk of wandering, and provide psychological care through emotion recognition, thereby significantly improving the physical and mental health and quality of life of facility users.

[1689] Example 2

[1690] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1691] The purpose of this invention is to manage the physical condition of elderly people, reduce feelings of loneliness, reduce the risk of wandering, and provide psychological care through emotion recognition. Conventional systems have individual functions, but few have been integrated into a single system, making it difficult to efficiently link multiple systems. Furthermore, conventional technology has limitations in meeting the demand for real-time anomaly detection and rapid response.

[1692] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes an imaging means for acquiring image data, an analysis means for analyzing the acquired image data and generating physical condition information, a storage and display means for saving and displaying the generated physical condition information, an emotion recognition means for acquiring and analyzing emotion data and identifying the emotional state, and an alert notification means for sending an alert when an abnormality is detected. This makes it possible to perform physical condition management and mental care for the elderly in an integrated and real-time manner.

[1693] The "photographing means for acquiring image data" is a device for photographing the face and physical condition of an elderly person and acquiring the image data.

[1694] "Analysis means for analyzing acquired image data and generating physical condition information" refers to a device or program for analyzing and generating physical condition information such as blood pressure, heart rate, and complexion of an elderly person using acquired image data.

[1695] The "storage and display means for storing and displaying the generated physical condition information" refers to a device or program for storing the physical condition information generated by the analysis in a database and displaying it as needed.

[1696] "Emotion recognition means for acquiring and analyzing emotional data to identify an emotional state" refers to a device or program for capturing emotions from the facial expressions and voice of an elderly person and analyzing the emotional data to identify the emotional state.

[1697] An "alert notification means for sending an alert when an abnormality is detected" is a device or program for sending a notification to care staff and other relevant parties when an abnormality is detected based on the elderly person's physical condition, location information, etc.

[1698] The "voice input means" is a device for capturing the voice of the elderly person and obtaining the voice data.

[1699] The "voice analysis means" is a device or program for analyzing acquired voice data and understanding its content.

[1700] The "voice response generating means" is a device or program for generating an appropriate response based on the analyzed voice data.

[1701] The "location information acquisition means" is a device for acquiring the current location of the elderly person in real time and transmitting that location information.

[1702] The "location information analysis means" is a device or program for analyzing the acquired location information and plotting it on a map to detect abnormal movements.

[1703] The "abnormality detection means" is a device or program for detecting abnormal situations based on the physical condition and location information of the elderly person.

[1704] This invention is a system that aims to manage the physical condition of elderly people, reduce feelings of loneliness, reduce the risk of wandering, and provide mental care through emotion recognition. This system is realized using smart AI mirrors installed in each room, and by combining it with an emotion recognition engine, it is also possible to care for the mental state of the elderly. The specific configuration and program processing of this system are explained below.

[1705] System configuration

[1706] This system consists of the following main components:

[1707] 1. Smart AI mirror (device)

[1708] It has a built-in camera to take pictures of the elderly person's face.

[1709] Built-in microphone for voice input.

[1710] Built-in display for showing analysis results.

[1711] It has a built-in emotion recognition engine to recognize emotions.

[1712] 2. Server

[1713] It has the ability to receive and analyze image data, audio data, and emotional data sent from the terminal.

[1714] It includes a database that generates and stores physical condition information, voice analysis results, and emotion analysis results.

[1715] The analysis results are notified to the terminal and the user terminal.

[1716] 3. User terminal (for care staff)

[1717] Tablets and PCs that display various information sent from the server and can be checked by care staff.

[1718] 4. GPS Tracker Insoles

[1719] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[1720] Specific processing explanation

[1721] Elderly health monitoring

[1722] Device:

[1723] When an elderly person stands in front of the smart AI mirror, the built-in camera captures an image of their face, which is then sent to a server in real time.

[1724] server:

[1725] The server processes the received image data using an analysis algorithm (e.g., OpenCV or TensorFlow). As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database, and if an abnormality is detected, an alert is sent to the care staff.

[1726] User device:

[1727] Care staff can check the health information of the elderly person through the user terminal and take necessary measures.

[1728] Examples:

[1729] When elderly person A stands in front of the smart AI mirror, the built-in camera takes a picture of their face. The image data is analyzed and the mirror determines that "A's health is good." This result can also be viewed on the caregiver's tablet.

[1730] Voice dialogue with the elderly

[1731] Device:

[1732] When an elderly person talks to the smart AI mirror, the built-in microphone captures the voice and sends it to the server.

[1733] server:

[1734] The server analyzes the voice data, runs natural language processing algorithms (e.g., Google Cloud Speech-to-Text), and generates an appropriate response based on the analysis results.

[1735] Device:

[1736] The generated response text is converted into speech using speech synthesis technology (e.g., Google Cloud Text-to-Speech) and output to the elderly.

[1737] Examples:

[1738] When Elderly Person B says, "The weather is nice today," the AI ​​Mirror sends the voice data to the server. The server analyzes the voice and generates a response saying, "The weather is really nice. It would be nice to go for a walk," and outputs it as voice.

[1739] Analysis of elderly people's location information

[1740] Location information acquisition method (shoe insoles with GPS tracker):

[1741] The location information of the elderly person is obtained at regular intervals and sent to the server.

[1742] server:

[1743] The server plots the received location information on map data (e.g., Google Maps API) and analyzes movement patterns. If abnormal movement is detected, a warning message is generated.

[1744] User device:

[1745] Care staff will receive a warning message and be able to respond quickly.

[1746] Examples:

[1747] If elderly person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his location information to the server, and the server will detect the abnormality and send an alert to the care staff.

[1748] Emotion recognition in the elderly

[1749] Device:

[1750] When an elderly person stands in front of the mirror, the built-in camera and microphone capture their emotions using an emotion recognition engine (e.g., Microsoft Azure Face API).

[1751] server:

[1752] The captured emotional data is analyzed to identify the emotional state of the elderly person, and the analysis results are stored in a database.

[1753] Device:

[1754] Based on the emotion analysis results sent from the server, an appropriate response is made. For example, if the emotion is negative, an encouraging voice response is generated.

[1755] User device:

[1756] Care staff check the emotional state of the elderly and take the necessary steps to provide mental care.

[1757] Examples:

[1758] When elderly person D looks at the mirror, the emotion recognition engine determines that he is "sad." This information is sent to the server and displayed on the care staff's device. The staff then plays a voice response from the mirror saying, "Mr. D, are you okay? Shall we talk?"

[1759] By combining the above configuration and processing, this system can comprehensively manage the physical condition of the elderly, reduce feelings of loneliness, reduce the risk of wandering, and provide psychological care through emotion recognition, which is expected to significantly improve the physical and mental health and quality of life of the elderly.

[1760] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1761] Step 1:

[1762] Elderly Image Capture

[1763] Terminal: When an elderly person stands in front of the smart AI mirror, the built-in camera automatically captures an image of their face. The image of the elderly person's face reflected in the camera is captured as input, and this is obtained as image data as output. Specifically, when Elderly Person A looks at the mirror with a smile, the display will show "Analyzing your physical condition."

[1764] Step 2:

[1765] Sending image data

[1766] Terminal: Sends the captured image data to the server. The acquired image data is used as input and sent securely using encryption technology (e.g., SSL / TLS). The image data is then transferred to the server as output. Specifically, the image data of elderly person A is sent from the mirror to the server within a few seconds.

[1767] Step 3:

[1768] Image data analysis

[1769] Server: Processes the received image data using an analysis algorithm (e.g., OpenCV or TensorFlow). Using the transmitted image data as input, the server analyzes it to generate health information such as the elderly person's blood pressure, heart rate, and complexion. This health information is obtained as output. Specifically, the server analyzes the image and determines that elderly person A's heart rate is "normal."

[1770] Step 4:

[1771] Storage and notification of health information

[1772] Server: The generated health information is saved in a database. If an abnormality is detected, an alert is sent to the care staff. The health information from the analysis is used as input, and the process of saving it to the database and sending an alert is executed. The output is the saving of the health information and the sending of an alert. Specifically, the server determines that elderly person A's blood pressure is high, and an alert is sent to the care staff's tablet.

[1773] Step 5:

[1774] Displaying health information

[1775] User device: The care staff checks the health information of the elderly person through the user device. The health information sent from the server is used as input and displayed. The health information can be checked by the care staff as output. Specifically, the care staff checks the health information of elderly person A on the tablet device and takes the necessary measures.

[1776] Step 6:

[1777] Capturing voice input

[1778] Terminal: When an elderly person speaks to the smart AI mirror, the built-in microphone captures the voice and sends this voice data to the server. The elderly person's voice is captured as input and obtained as voice data. The voice data is sent to the server as output. In concrete terms, when Elderly Person B speaks to the mirror saying, "How's the weather?", the mirror displays "Analyzing voice."

[1779] Step 7:

[1780] Analysis of audio data

[1781] Server: Analyzes the voice data and runs a natural language processing algorithm (e.g., Google Cloud Speech-to-Text). It uses the transmitted voice data as input and performs the analysis. As output, it generates corresponding text data and obtains an appropriate response text. Specifically, the server analyzes Person B's voice and generates the response "It's sunny today."

[1782] Step 8:

[1783] Response generation and speech output

[1784] Terminal: The response text sent from the server is converted into speech using speech synthesis technology (e.g., Google Cloud Text-to-Speech) and output to the elderly person. The response text is used as input to generate speech. A voice notification is output to the elderly person. Specifically, the mirror notifies Mr. B by voice, saying, "It's sunny today."

[1785] Step 9:

[1786] Obtaining location information

[1787] Location information acquisition means (shoe insoles with GPS trackers): The location information of the elderly person is acquired at regular intervals and sent to the server. GPS data is acquired in real time as input. The acquired location information is sent to the server as output. Specifically, when Mr. C is walking in the facility's garden, the GPS tracker acquires his location information and periodically sends it to the server.

[1788] Step 10:

[1789] Location analysis

[1790] Server: The received location information is plotted on map data (e.g., Google Maps API) and movement patterns are analyzed. The acquired location information is used as input for analysis. Movement patterns and anomaly detection results are obtained as output. Specifically, the server displays Mr. C's location on a map and generates a warning that "he has left the facility."

[1791] Step 11:

[1792] Sending a warning message

[1793] Server: Sends the generated warning message to the user device. As input, it uses the anomaly detection results to create a warning message. As output, it sends the warning message to the user device. Specifically, the server detects an anomaly and sends a warning to the tablet device of the care staff.

[1794] Step 12:

[1795] Capturing Emotional Data

[1796] Device: When an elderly person stands in front of the mirror, the built-in camera and microphone capture their emotions using an emotion recognition engine (e.g., Microsoft Azure Face API). Emotion data is obtained from the elderly person's facial expressions and voice as input. Emotion recognition data is obtained as output. Specifically, when Mr. D looks at the mirror, the mirror displays the message "Analyzing emotions."

[1797] Step 13:

[1798] Sending and analyzing emotional data

[1799] Server: Analyzes the captured emotion data and identifies the emotional state of the elderly person. The analysis is performed using the transmitted emotion data as input. The output is the identified emotional state, and the result is stored in the database. Specifically, the server determines that Mr. D is "sad" based on his facial expression, and stores the result in the database.

[1800] Step 14:

[1801] Responding based on emotional state

[1802] Terminal: Based on the emotion analysis results sent from the server, an appropriate response is made. For example, if the emotion is negative, an encouraging voice response is generated. The emotion analysis results are received as input and an appropriate response is made. The output is a response and care for the elderly person. Specifically, the mirror responds by voice saying, "Mr. D, are you OK? Shall we talk?"

[1803] Step 15:

[1804] Emotional state notification

[1805] User terminal: The care staff checks the emotional state of the elderly person and takes the necessary measures for mental care. As input, it receives and displays the emotion analysis results. As output, the care staff checks the emotional state and is able to provide care. Specifically, the server notifies the staff of Mr. D's emotional state, and the staff quickly checks it.

[1806] (Application example 2)

[1807] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1808] While issues such as managing the health of elderly people, reducing feelings of loneliness, and reducing the risk of wandering are important in nursing homes and other facilities, there is also a need to quickly understand and respond to customers' physical and emotional states in brick-and-mortar stores. However, current systems have difficulty comprehensively resolving these issues, and are unable to adequately address the issues of improving customer satisfaction and employee work efficiency. Furthermore, there is a lack of measures to prevent customers from wandering within brick-and-mortar stores or to detect abnormalities using location information, leaving challenges for store operations.

[1809] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a notification means for analyzing physical condition information and notifying of abnormalities, an emotion analysis means for analyzing emotional states and notifying of the results, and a notification means for monitoring the location of customers in real time and notifying of any abnormalities. This makes it possible to manage the physical condition and emotional care of elderly people and prevent them from getting lost, even in stores, thereby improving customer satisfaction and employee work efficiency.

[1810] "Capture means" refers to a device used to acquire image data of a subject.

[1811] An "analysis means" is a device or system for analyzing acquired data to generate useful information.

[1812] "Storage and display means" refers to a mechanism for storing the generated information and displaying it as needed.

[1813] The "notification means" is a system for notifying a specified terminal of information when a specific condition is met.

[1814] "Emotion analysis means" is a technology for analyzing the emotional state of a subject from their facial expressions and voice.

[1815] "Location information acquisition means" refers to a device or technology for acquiring location information of an object.

[1816] "Voice input means" refers to a device or system for acquiring voice data.

[1817] "Speech analysis means" refers to a technique for analyzing acquired speech data and extracting meaning.

[1818] The "voice response generating means" is a technology for generating an appropriate voice response based on the analysis results.

[1819] The "abnormality detection means" is a system that analyzes the acquired data and detects any abnormal behavior or state.

[1820] This invention provides a system for managing customers' physical condition, recognizing their emotions, and preventing them from getting lost in a brick-and-mortar store. The specific system and its program configuration and processing will be described below.

[1821] System configuration

[1822] This system consists of the following main components:

[1823] 1. Smart AI mirror (device)

[1824] Photography means: Equipped with a built-in camera to photograph the faces of customers while they are being served.

[1825] Voice input means: A built-in microphone is provided to accept voice input from customers.

[1826] Storage and display means: A built-in display is provided to show the analysis results.

[1827] Emotion analysis means: Equipped with an emotion recognition engine to recognize emotions from customers' facial expressions and voices.

[1828] 2. Server

[1829] Analysis means: Has the function of receiving and analyzing image data and audio data sent from the terminal.

[1830] Notification method: Based on the analysis results, the system has the function of notifying staff if an abnormality is detected.

[1831] Location information acquisition method: Receives data from a badge equipped with a GPS tracker that acquires and analyzes customer location information in real time.

[1832] Anomaly detection means: Detects anomalies based on the acquired location information.

[1833] 3. User terminals (for staff)

[1834] Tablets and PCs that display various information sent from the server and can be checked by staff.

[1835] System action

[1836] 1. Health monitoring process

[1837] Terminal: When a customer stands in front of the smart AI mirror, the built-in camera automatically captures an image of the customer's face, and the captured image data is sent to the server in real time.

[1838] Server: The server analyzes the health information using the received image data. The analysis results include the customer's blood pressure, heart rate, and complexion. If an abnormality is detected, a notification method is activated and an alert is sent to staff.

[1839] User terminal: Staff check the customer's health information through the terminal and take necessary measures.

[1840] Examples:

[1841] Customer A stands in front of the smart AI mirror, and the mirror takes a picture of his face. The image data is sent to the server, which analyzes the image and determines that "Customer A is in good health." This result is also displayed on the staff member's tablet.

[1842] 2. Emotion Recognition Processing

[1843] Terminal: When customers talk to the smart AI mirror, the built-in microphone captures the voice, and this voice data is sent to the server.

[1844] Server: The server analyzes the voice data and runs natural language processing algorithms to understand what is being said. Based on this, the customer's emotional state is also analyzed.

[1845] Terminal: Based on the results of the sentiment analysis, necessary actions are taken. If the customer's sentiment is negative, an encouraging voice response is generated.

[1846] User terminal: Staff can check the emotional state of customers through the terminal and take the necessary measures to provide mental care.

[1847] Examples:

[1848] When Customer B says to the smart AI mirror, "I'm tired today," the mirror sends the voice message to the server. The server analyzes the voice data and determines that the customer is tired. This information is sent to the staff member's tablet. The staff member then plays a response such as "Would you like to take a break?" from the mirror.

[1849] 3. Preventing children from getting lost

[1850] Location information acquisition method: Customers are given a badge with a GPS tracker, which sends the customer's location information to the server at regular intervals.

[1851] Server: The server analyzes the received location information and detects any abnormal movements. If an abnormality is detected, it generates a warning message and notifies staff.

[1852] User terminal: If abnormal location movements are detected through the terminal, staff can respond quickly.

[1853] Examples:

[1854] If Customer C gets lost in the store, the GPS tracker will send his location information to the server, which will detect the abnormal location and send an alert to the staff, who will quickly rush to assist the customer.

[1855] Prompt Sentence Examples

[1856] Prompt: Suggest an application for a brick-and-mortar store that monitors customers' physical and emotional states and allows employees to respond quickly. Ideally, the application should have the following characteristics:

[1857] 1. Analyze customer health information in real time and notify staff if there are any abnormalities.

[1858] 2. Recognize the emotional state of the customer and notify staff when necessary.

[1859] 3. It has a function to monitor customer location information to prevent children from getting lost.

[1860] With the above configuration and processing, this system can manage the physical condition and emotional well-being of elderly people and prevent them from getting lost, even in brick-and-mortar stores, thereby improving customer satisfaction and employee work efficiency.

[1861] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1862] Step 1:

[1863] In order for the device to monitor the customer's physical condition, the Smart AI Mirror's camera captures image data of the customer's face. The captured image data is input and sent to the server. Specifically, the camera starts up, captures images at regular intervals, and sends the data to the server.

[1864] Step 2:

[1865] The server generates health information from the transmitted image data using analytical means. The input is the captured image data, and the output is the analyzed health information. The server uses an AI algorithm to extract health information such as blood pressure, heart rate, and complexion from the image data.

[1866] Step 3:

[1867] The server stores the generated health information in a database and sends it to the storage and display means for display. The input is the analyzed health information, and the output is the stored data and displayed health information. The server stores the information in the database and displays the contents on the terminal and the user terminal.

[1868] Step 4:

[1869] The smart AI mirror's microphone captures the customer's speech so that the device can accept the customer's voice input. The input is voice data, which is then sent to the server. The microphone detects the voice, collects data, and sends it to the server.

[1870] Step 5:

[1871] The server analyzes the voice content of the transmitted voice data using an analysis means to understand what the customer is saying. The input is the voice data, and the output is the analyzed voice content. The server uses a natural language processing algorithm to generate text from the voice and analyzes the content.

[1872] Step 6:

[1873] The server generates an appropriate voice response based on the analysis results. The input is the analyzed voice content and the output is the generated voice response. The server uses a response generation algorithm to create a text response and convert it to speech.

[1874] Step 7:

[1875] The terminal outputs the generated voice response to the customer. The input is the generated voice response and the output is the played back audio. The smart AI mirror uses a speaker to communicate the response to the customer.

[1876] Step 8:

[1877] The location information acquisition means provides customers with a badge with a GPS tracker, which transmits the customer's location information to the server at regular intervals. The input is location information, which is sent to the server. The tracker detects the location data and sends it to the server.

[1878] Step 9:

[1879] The server analyzes the received location information and detects anomalous movement. The input is location information and the output is anomalous movement patterns. The server uses a location analysis algorithm to plot the data on a map and identify anomalies.

[1880] Step 10:

[1881] The user terminal receives the anomaly detection notification from the server and notifies the staff that an anomaly has been detected. The input is the anomaly detection notification, and the output is the notified information. The staff terminal receives the alert and displays it.

[1882] Step 11:

[1883] The user's device provides staff with information such as the customer's current situation and location. The input is data from the server, and the output is information provided to the staff. Based on this information, the staff can respond quickly.

[1884] In this way, the system enables customer health management, emotion recognition, and loss prevention in physical stores.

[1885] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1886] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1887] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1888] [Fourth embodiment]

[1889] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1890] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1891] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1892] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1893] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1894] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1895] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1896] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1897] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1898] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1899] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1900] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1901] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1902] This invention is a system for managing the physical condition of elderly people in nursing homes, reducing feelings of loneliness, and reducing the risk of wandering. This system is realized using smart AI mirrors installed in each room. The specific configuration and program processing of this system are described below.

[1903] System configuration

[1904] This system consists of the following main components:

[1905] 1. Smart AI mirror (device)

[1906] It has a built-in camera to take pictures of the elderly person's face.

[1907] Built-in microphone for voice input.

[1908] Built-in display for showing analysis results.

[1909] 2. Server

[1910] It has the function of receiving and analyzing image data and audio data sent from the terminal.

[1911] It includes a database that generates and stores physical condition information.

[1912] The analysis results are notified to the terminal and the user terminal.

[1913] 3. User terminal (for care staff)

[1914] A tablet or PC that displays health information sent from the server and can be checked by care staff.

[1915] 4. GPS Tracker Insoles

[1916] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[1917] Program processing

[1918] The program for this system is implemented to achieve three main functions: monitoring the physical condition of the elderly, voice interaction, and location information analysis.

[1919] 1. Health monitoring process

[1920] Device:

[1921] When an elderly person stands in front of the mirror, the built-in camera automatically activates and captures an image of their face, which is then sent to a server in real time.

[1922] server:

[1923] The server processes the received image data using an analysis algorithm. As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database, and if necessary, an alert is sent to care staff if an abnormality is detected.

[1924] User device:

[1925] Care staff can check the health information of the elderly person through the user terminal and take necessary measures based on this information.

[1926] Examples:

[1927] Elderly person A stands in front of the smart AI mirror, which takes a picture of their face. The image data is sent to the server, which analyzes the image and determines that "A's health is good." This result is also displayed on the caregiver's tablet.

[1928] 2. Voice Dialogue Processing

[1929] Device:

[1930] When an elderly person speaks to the smart AI mirror, the built-in microphone captures their voice, and the voice data is sent to a server.

[1931] server:

[1932] The server analyzes the audio data and runs natural language processing algorithms to understand what is being said, and generates an appropriate response based on the results.

[1933] Device:

[1934] The response text sent from the server is converted into speech using speech synthesis technology and output to the elderly.

[1935] Examples:

[1936] When Elderly Person B says, "The weather is nice today," the Smart AI Mirror sends the speech to the server. The server analyzes the speech data and generates a response, "The weather is really nice. It would be nice to go for a walk," and outputs this response as voice.

[1937] 3. Location analysis processing

[1938] Location information acquisition method (shoe insoles with GPS tracker):

[1939] The location information of the elderly person is obtained at regular intervals and sent to the server.

[1940] server:

[1941] The server plots the received location information on map data, analyzes movement patterns, and generates a warning message if abnormal movement (e.g., movement into an unauthorized area) is detected.

[1942] User device:

[1943] Care staff will receive a warning message via the user's terminal, enabling them to respond quickly.

[1944] Examples:

[1945] If Elderly Person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his / her location information to the server, which will detect the abnormal location and send a warning to the care staff.

[1946] By combining the above configuration and processing, this system can manage the physical condition of elderly people, reduce feelings of loneliness, and reduce the risk of them wandering. It is an effective means of improving the physical and mental health and quality of life of facility users.

[1947] The processing flow will be explained below.

[1948] Health monitoring process

[1949] Step 1:

[1950] The device detects when an elderly person stands in front of the smart AI mirror, and the built-in camera automatically activates.

[1951] Step 2:

[1952] The device takes a picture of the elderly person's face and captures the image data in real time, then encodes the captured image data.

[1953] Step 3:

[1954] The terminal transmits the encoded image data to the server.

[1955] Step 4:

[1956] The server decodes the received image data and applies it to an analysis algorithm, which estimates blood pressure and heart rate based on facial color, facial expression, and whether the eyes are open or closed.

[1957] Step 5:

[1958] The server stores the physical condition information generated as a result of the analysis in a database, and then transmits this physical condition information to the terminal and the user terminal.

[1959] Step 6:

[1960] The user (care staff) uses the user terminal to check the health information of the elderly person. If any abnormalities are found, they respond promptly.

[1961] Voice interaction processing

[1962] Step 1:

[1963] The device is constantly in listening mode and waiting for voice input from the elderly person.

[1964] Step 2:

[1965] The device captures the voice of the elderly person and records it as audio data.

[1966] Step 3:

[1967] The device converts the captured voice data into text and sends it digitally to a server.

[1968] Step 4:

[1969] The server analyzes the received voice data using a natural language processing algorithm to understand what the elderly person is saying.

[1970] Step 5:

[1971] The server generates an appropriate response text based on the analysis results and sends this response text to the terminal.

[1972] Step 6:

[1973] The response text received by the terminal is converted into voice data using voice synthesis technology and output to the elderly person.

[1974] Processing location analysis

[1975] Step 1:

[1976] The location information acquisition means (shoe insoles with GPS trackers) acquires the elderly person's location information at regular intervals.

[1977] Step 2:

[1978] The location information acquired by the location information acquisition means is encoded and transmitted to a server in real time.

[1979] Step 3:

[1980] The server decodes the received location information and plots it in a map database.

[1981] Step 4:

[1982] The server analyzes movement patterns based on the plotted location information and detects abnormal movement (for example, movement into unauthorized areas).

[1983] Step 5:

[1984] If the server detects an abnormal movement pattern, it generates a warning message and sends this message to the user terminal.

[1985] Step 6:

[1986] The user (care staff) receives and checks the warning message via the user terminal and takes prompt action.

[1987] Example 1

[1988] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1989] In nursing homes for the elderly, managing the health of the elderly, reducing feelings of loneliness, and reducing the risk of wandering are important issues. Conventional systems do not provide comprehensive solutions to these issues, which increases the burden on care staff and makes it difficult to ensure the safety and health of the elderly. In addition, the high risk of wandering requires real-time monitoring of location information. Therefore, a system that integrates health monitoring, voice dialogue, and location information analysis for the elderly is needed.

[1990] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1991] In this invention, the server includes a camera means for acquiring image data, an analysis means for analyzing the acquired image data to generate health information, a database means for saving the generated health information, a display means for displaying the generated health information, a notification means for sending an alert when an abnormality is detected, a voice input means for receiving and analyzing voice input and generating a corresponding response, a natural language processing means for analyzing the voice data, a voice response generation means, a voice synthesis means for synthesizing the voice response, and a means for acquiring location information of the elderly person, detecting abnormal movements based on the location information, and sending a warning message as necessary. This makes it possible to appropriately manage the health of the elderly person, reduce feelings of loneliness, and reduce the risk of wandering.

[1992] "Camera means for capturing image data" refers to a photographic device used to capture images of the face and body of the elderly person.

[1993] "Analysis means for analyzing acquired image data to generate physical condition information" refers to algorithms and software for processing acquired image data and calculating physical condition information such as blood pressure, heart rate, and complexion of the elderly person.

[1994] The "database means for storing generated physical condition information" refers to a database system that stores generated physical condition information and can retrieve it as needed.

[1995] The "display means for displaying the generated physical condition information" refers to a display or screen for visually displaying the generated physical condition information.

[1996] "Notification means for sending an alert when an abnormality is detected" refers to a system that sends a warning message to care staff or appropriate personnel when an abnormality is detected in health information.

[1997] "Voice input means" refers to a microphone or voice recognition device that captures the voice spoken by the elderly person and collects the voice data.

[1998] "Natural language processing means for analyzing voice data" refers to natural language processing algorithms and software for analyzing acquired voice data and understanding its content.

[1999] "Voice response generator" refers to software or algorithms for generating appropriate responses based on analyzed voice data.

[2000] The "voice synthesis means for synthesizing a voice response" refers to a voice synthesis technology for outputting the generated response as voice.

[2001] "Location information acquisition means" refers to a location information acquisition device such as a GPS system used to determine the current location of an elderly person.

[2002] "Map data means for plotting received location information on map data" refers to a system for displaying acquired location information on a map.

[2003] "Anomaly detection means for detecting abnormal movements" refers to algorithms or software for analyzing acquired location information and detecting abnormal movements or behaviors.

[2004] "Warning means for sending a warning message when an abnormality is detected" refers to a system that sends a warning message to relevant parties when an abnormality in movement is detected.

[2005] MODE FOR CARRYING OUT THE INVENTION

[2006] This invention is a system aimed at managing the health of elderly people, reducing feelings of loneliness, and reducing the risk of wandering. This system is realized using a smart AI mirror installed in each room. The specific configuration and program processing of this system are described in detail below.

[2007] System configuration

[2008] This system consists of the following main components:

[2009] 1. Smart AI mirror (device)

[2010] It has a built-in camera to take pictures of the elderly person's face.

[2011] Built-in microphone for voice input.

[2012] Built-in display for showing analysis results.

[2013] 2. Server

[2014] It has the function of receiving and analyzing image data and audio data sent from the terminal.

[2015] It includes a database that generates and stores physical condition information.

[2016] The analysis results are notified to the terminal and the user terminal.

[2017] 3. User terminal (for care staff)

[2018] A tablet or PC that displays health information sent from the server and can be checked by care staff.

[2019] 4. GPS Tracker Insoles

[2020] It has a built-in GPS function that acquires and transmits the location information of elderly people in real time.

[2021] Program processing

[2022] The program for this system is implemented to achieve three main functions: monitoring the physical condition of the elderly, voice interaction, and location information analysis.

[2023] Health monitoring process

[2024] Device:

[2025] When an elderly person stands in front of the mirror, the built-in camera automatically activates and captures an image of their face, which is then sent to a server in real time.

[2026] server:

[2027] The server processes the received image data using an analysis algorithm (e.g., OpenCV or TensorFlow). As a result of the analysis, physical condition information such as the elderly person's blood pressure, heart rate, and complexion is generated. The generated physical condition information is stored in a database (e.g., MySQL), and alerts are sent to care staff as necessary.

[2028] User device:

[2029] Care staff can check the health information of the elderly person through the user terminal and take necessary measures based on this information.

[2030] Examples:

[2031] When elderly person A stands in front of the smart AI mirror and the mirror takes a picture of their face, the image data is sent to the server. The server analyzes the image and determines that "A's health is good." This result is also displayed on the care staff's tablet.

[2032] Voice interaction processing

[2033] Device:

[2034] When an elderly person speaks to the smart AI mirror, the built-in microphone captures their voice, and the voice data is sent to a server.

[2035] server:

[2036] The server runs a natural language processing algorithm (e.g., Google's Speech-to-Text API or OpenAI's GPT-3 model) to analyze the voice data and generates an appropriate response based on the results.

[2037] Device:

[2038] The response text sent from the server is converted into speech using speech synthesis technology (e.g., Google's Text-to-Speech API) and output to the elderly via the mirror.

[2039] Examples:

[2040] When Elderly Person B says, "The weather is nice today," the Smart AI Mirror sends the speech to the server. The server analyzes the speech data and generates a response, "The weather is really nice. It would be nice to go for a walk," and outputs this response as voice.

[2041] Processing location analysis

[2042] Location information acquisition method (shoe insoles with GPS tracker):

[2043] The location information of the elderly person is obtained at regular intervals and sent to the server.

[2044] server:

[2045] The server plots the received location information on map data (e.g., Google Maps API) and analyzes movement patterns. If abnormal movement (e.g., movement into an unauthorized area) is detected, a warning message is generated.

[2046] User device:

[2047] Care staff will receive a warning message via the user's terminal, enabling them to respond quickly.

[2048] Examples:

[2049] If Elderly Person C goes outside the facility and is in danger of getting lost, the GPS tracker will send his / her location information to the server, which will detect the abnormality and send an alert to the care staff.

[2050] Prompt Sentence Examples

[2051] Example prompt 1 (health monitoring):

[2052] "Please extract health information from facial images of elderly people. Specifically, please generate information on three elements: blood pressure, heart rate, and complexion."

[2053] Prompt example 2 (voice dialogue):

[2054] "Generate an appropriate response when an older adult says, 'The weather is nice today.'"

[2055] Example prompt 3 (location analysis):

[2056] "Implement an algorithm that receives location information from seniors, plots it on map data, and detects abnormal movements."

[2057] This system, configured in this way, plays an important role in managing the health of the elderly, and also contributes to reducing feelings of loneliness and the risk of wandering.

[2058] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2059] Health monitoring process

[2060] Step 1: Capture and send a facial image

[2061] Terminal: When an elderly person stands in front of the smart AI mirror, the terminal activates the built-in camera to capture the elderly person's facial image, and the input data is the elderly person's facial image, which is then sent to the server in real time.

[2062] Specific operation: When elderly person A stands in front of the mirror, the device's camera automatically starts up, takes a picture of his face, encrypts the image, and sends it to the server.

[2063] Step 2: Analyzing the image data

[2064] Server: The server uses algorithms (e.g., OpenCV or TensorFlow) to analyze the received image data. The input data is a facial image, which is analyzed to generate physical condition information such as the elderly person's blood pressure, heart rate, and complexion.

[2065] Specific operation: The server uses a GPU to quickly analyze the image data and calculates that the blood pressure is 120 / 80 and the heart rate is 70 BPM.

[2066] Step 3: Save and notify health information

[2067] Server: The generated health information is stored in a database (e.g., MySQL). The input data is the generated health information, and if an abnormality is detected, an alert is sent to the care staff.

[2068] Specific operation: When the server detects an abnormal value (e.g., blood pressure 160 / 100), it immediately generates an alert and notifies the care staff's terminal.

[2069] Step 4: Check your health information

[2070] User: The care staff checks the health information of the elderly person through the user terminal. The input data is the generated health information, and the care staff takes necessary measures based on this.

[2071] Specific actions: Caregiver B uses a tablet device to check A's health information and immediately contacts the doctor.

[2072] Voice interaction processing

[2073] Step 1: Capture and send audio

[2074] Terminal: When the elderly person speaks to the smart AI mirror, the built-in microphone of the terminal captures the voice and transmits the voice data to the server in real time. The input data is an audio file.

[2075] Specific operation: When Elderly Person B says, "The weather is nice today," the device's microphone records the voice and sends it to the server.

[2076] Step 2: Analyzing the audio data

[2077] Server: The server uses natural language processing algorithms (e.g., Google's Speech-to-Text API or OpenAI's GPT-3 model) to analyze the voice data and understand what is being said. The input data is an audio file, which is converted into text and an appropriate response is generated.

[2078] What it does: The server converts the speech "The weather is nice today" into text and generates a response saying "The weather is really nice. It would be nice to go for a walk."

[2079] Step 3: Generate a voice response

[2080] Terminal: The response text sent from the server is converted into speech using speech synthesis technology (e.g., Google's Text-to-Speech API). The input data is the response text, which is then output as speech to the elderly.

[2081] Specific operation: The device outputs a synthesized voice saying, "What beautiful weather. It would be nice to go for a walk."

[2082] Processing location analysis

[2083] Step 1: Obtaining and sending location information

[2084] Location information acquisition means (shoe insoles with GPS trackers): The location information of the elderly is acquired at regular intervals (e.g., every minute) and sent to the server. The input data is location coordinates.

[2085] Specific operation: The GPS built into the insole of elderly person C's shoe captures location information every minute and sends it to the server.

[2086] Step 2: Analyze location information

[2087] Server: The server plots the received location information on map data (e.g., Google Maps API) and analyzes movement patterns. The input data is location information, and abnormal movements are detected.

[2088] Specific operation: The server checks the location of elderly person C, who has left the facility, on a map and detects any abnormalities.

[2089] Step 3: Notification of warning messages

[2090] User terminal: The nursing staff receives the warning message through the user terminal and responds promptly. The input data is the warning message.

[2091] Specific operation: A warning notification stating "Elderly person C has left the facility" appears on the smartphone of caregiver D.

[2092] (Application example 1)

[2093] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2094] In managing the physical and mental health of the elderly, there is a need for systems that allow elderly people at high risk of wandering to live within a safe range, and that allow caregivers and family members to monitor the elderly's location and health information in real time and respond quickly. However, existing systems lack the functionality to manage this information in an integrated manner and send appropriate alerts in real time when an abnormality occurs. It is also important to have a function that can reduce the elderly's sense of loneliness through voice dialogue.

[2095] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2096] In this invention, the server includes an imaging means for acquiring image data, an analysis means for analyzing the acquired image data to generate health information, a storage and display means for storing and displaying the generated health information, a location information acquisition means for acquiring location information, an abnormality detection means for detecting abnormal movements by comparing the acquired location information with a geographical fence, an alert notification means for sending a notification when an abnormality is detected, a voice input means, a voice analysis means, a voice response generation means, a user interface means for integrating the elderly person's health information and location information and enabling real-time confirmation, and a push notification means. This enables health management of the elderly person, monitoring of location information, and prompt response when an abnormality is detected.

[2097] "Photographing means for acquiring image data" refers to a device such as a camera for photographing an image of the face of an elderly person.

[2098] "Analysis means" refers to the algorithms and software that analyze the acquired image data and generate physical condition information such as blood pressure, heart rate, and complexion.

[2099] "Storage and display means" refers to a database that stores the generated physical condition information and a system for displaying that information on a display or the like.

[2100] "Location information acquisition means" refers to devices such as GPS sensors that acquire the current location of elderly people in real time.

[2101] "Anomaly detection means" refers to an algorithm or software that analyzes acquired location information and determines that an anomaly has occurred if the information exceeds a set safety range (geographical fence).

[2102] "Alert notification means" refers to push notification services and communication means for sending notifications to caregivers and family members when an abnormality is detected.

[2103] "Voice input means" refers to a device such as a microphone for capturing the elderly person's voice.

[2104] "Speech analysis means" refers to natural language processing algorithms or software that analyzes captured voice data and understands what is being said.

[2105] The "voice response generation means" refers to a voice synthesis technology for generating an appropriate response based on the analysis results and outputting the response as voice.

[2106] "User interface means" refers to an interface that displays the elderly person's physical condition information and location information in real time, allowing caregivers and family members to easily check them.

[2107] "Push notification means" refers to a system that sends notifications to caregivers' and family members' devices in real time when an abnormality is detected.

[2108] This invention provides a system for managing the physical condition of elderly people, tracking their location information, and reducing the risk of them wandering. The specific configuration and processing of this system will be described below.

[2109] System configuration

[2110] This system consists of the following main components:

[2111] 1. Terminal

[2112] These are mobile devices such as smartphones and smart glasses that are used as belongings for the elderly.

[2113] It has a built-in camera for taking facial images of elderly people.

[2114] It has a built-in microphone for accepting voice input.

[2115] It has a built-in GPS sensor to acquire location information.

[2116] 2. Server

[2117] It has the function of receiving and analyzing image data and audio data sent from the terminal.

[2118] It has a database that generates and stores health information and location information.

[2119] It has a system to generate alerts and notify you if an abnormality is detected.

[2120] 3. User Device

[2121] A tablet or PC that displays health information and location information sent from a server and can be checked by care staff and family members.

[2122] What the program does

[2123] 1. Health monitoring process

[2124] Device: When an elderly person stands in front of the camera, their facial image is automatically captured. This image data is sent to the server in real time.

[2125] Server: Analyzes image data using artificial intelligence analysis tools (e.g., TensorFlow, PyTorch, etc.) to generate health information such as blood pressure and heart rate. The generated health information is stored in a database. If an abnormality is detected, an alert notification is sent to the caregiver's device.

[2126] User device: Care staff can check the health information of elderly people via tablets or PCs and take necessary measures.

[2127] Example: An elderly person stands in front of the device and the camera captures their face. The image data is sent to a server, which analyzes it and determines that the person is in good health. This is then displayed on the caregiver's tablet.

[2128] 2. Voice Dialogue Processing

[2129] Terminal: When an elderly person speaks to the terminal, the microphone captures the voice and the voice data is sent to the server.

[2130] Server: Analyzes the voice data using natural language processing algorithms and generates an appropriate response.

[2131] Terminal: The response sent from the server is output as voice using speech synthesis technology.

[2132] Example: When an elderly person says, "The weather is nice today," the device sends the speech to the server. The server generates a response, "It's really nice weather. It would be nice to go for a walk," and the device outputs it as speech.

[2133] 3. Location analysis processing

[2134] Device: The device's GPS sensor acquires the elderly person's location information and sends it to the server.

[2135] Server: Executes anomaly detection measures that compare location information with geo-fences to detect abnormal movements. If an anomaly is detected, an alert notification measure is used to send a notification to caregivers and family members.

[2136] User device: Care staff and family members can receive notifications through the device, check the elderly person's location, and respond quickly.

[2137] Example: If an elderly person goes outside a set safe area, the GPS sensor sends their location to a server. The server detects the abnormality and sends a notification to the caregiver's smartphone that "the elderly person has gone outside the range."

[2138] Prompt Sentence Examples

[2139] 1. Obtaining location information

[2140] "Get the user's current location. Use the GPS sensor to send the latitude and longitude to the server in real time."

[2141] 2. Setting up geo-fences

[2142] It provides a GUI for configuring safety limits within the app and stores the user's settings in a database.

[2143] 3. Sending emergency alerts

[2144] "If out-of-range movement is detected, a push notification is sent to the registered caregiver's smartphone using Firebase Cloud Messaging."

[2145] 4. Check your health information

[2146] "We will create a UI that displays the user's health data received from the server and updates it in real time."

[2147] This allows the system to manage the health of elderly people and monitor their location information in an integrated manner, protecting their safety and reducing the burden on caregivers and family members.

[2148] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2149] Step 1:

[2150] Terminal: Acquisition of image data

[2151] The device's camera captures the elderly person's facial image. The elderly person's face must be in front of the camera as input, and the captured facial image data is obtained as output.

[2152] Step 2:

[2153] Terminal: Sending image data

[2154] The acquired facial image data is sent to the server in real time. The input is the facial image data captured in step 1, and the output is the data successfully sent to the server.

[2155] Step 3:

[2156] Server: Generates health information

[2157] The server analyzes the received facial image data using an artificial intelligence algorithm (e.g., TensorFlow, PyTorch) to generate physical condition information such as blood pressure, heart rate, and complexion. The input is the facial image data sent from the device, and the output is the generated physical condition information.

[2158] Step 4:

[2159] Server: Storage of health information

[2160] The generated physical condition information is saved in a database. The input is the physical condition information generated in step 3, and the output is the physical condition information saved in the database.

[2161] Step 5:

[2162] Server: Anomaly detection

[2163] The server monitors the stored health information and generates an alert if an abnormality is detected. Pre-set criteria are used for abnormality detection. The input is the health information stored in the database, and the output is an alert notification if an abnormality is detected.

[2164] Step 6:

[2165] Server: Alert Notification

[2166] If an abnormality is detected, the server sends an alert notification to the caregiver or family member's device. The input is the alert generated in step 5, and the output is the notification sent to the caregiver or family member's device.

[2167] Step 7:

[2168] Device: Audio data capture

[2169] The microphone on the device captures the elderly person's voice, and the input is the elderly person's voice, and the output is the captured voice data.

[2170] Step 8:

[2171] Terminal: Sending audio data

[2172] Send the captured audio data to the server in real time. The input is the audio data captured in step 7, and the output is the data successfully sent to the server.

[2173] Step 9:

[2174] Server: Analysis of voice data

[2175] The server analyzes the received voice data, uses natural language processing algorithms (e.g., BERT, GPT-3, etc.) to understand the speech and generate an appropriate response. The input is the voice data sent from the device, and the output is the generated response text.

[2176] Ste...

Claims

1. To monitor the health of the elderly, an imaging means for acquiring image data; an analysis means for analyzing the acquired image data to generate physical condition information; a storage and display means for storing and displaying the generated physical condition information; A system including:

2. to accept and analyze the elderly person's voice input and generate corresponding responses; A voice input means; A voice analysis means; A voice response generating means; The system of claim 1 further comprising:

3. To acquire location information of elderly people and detect abnormal movements based on that location information, location information acquisition means; location information analysis means; Anomaly detection means; The system of claim 1 further comprising:

Images